Driving the User-Centric Uber Experience with Renato Beninatto

Renato: We were looking at the quality of language. It is a theme that in the language industry, everybody wants to talk about quality, as if quality was an entity, a measurable thing that you have a clear definition of what it is, and quality is the same for all content and for all products. And that is not the case.

Brian: Today we welcome back Renato Beninatto, Chairman at Nimdzi Insights. He is the chairman and co-founder of Nimdzi Insights, a market research and consulting company specialised in the language industry. In addition to his work at Nimdzi, Renato is the co-author of The General Theory of the Translation Company, co-owner of Multilingual Media, and adjunct professor at the Middlebury Institute of International Studies at Monterey. Renato, welcome back to The Native Experience.

Renato: Thank you, Brian. It is a pleasure to be called back. I must have said something interesting last time.

Brian: Tell us what you are nerding out on today.

Renato: My name is Renato Beninatto. I am the chairman and co-founder of Nimdzi Insights. We are based in Seattle. We are a market research and consulting company specialised in the language industry. We work with the biggest technology companies, medium-sized exporters, translation companies, and technology companies in the language space. What I am nerding out on is Ethiopian food. I found it maybe three months ago and it is so good. I had something called Doro Wat, which is a spicy chicken. You eat on a big piece of bread. Here in Seattle there is a big Ethiopian community very close to my house. An Italian friend introduced us to the restaurant. The owner was very welcoming, teaching us how to eat and what to choose. It was an amazing experience.

Brian: When you were on the show last time, you touched on your experience with Uber. You did a customer perception study that saved Uber $4 million per year. Walk us through the experience from start to finish.

Renato: The main aspect of the story is that we were looking at the quality of language. In the language industry, everybody wants to talk about quality as if it was an entity, a measurable thing with a clear definition. And quality is the same for all content and products. That is not the case.

Renato: Uber has been a member of Nimdzi. We sell memberships that include access to all our research and ad hoc consulting with our team. We have monthly sync-ups where we bring our consultants and they bring their internal staff. They told us they were having issues with their vendors for translation and LQA, linguistic quality assessment. They asked us to do an audit of their suppliers to figure out what was wrong.

Renato: After a series of interviews with the vendors and internal stakeholders, we found out that the problem was not the vendor. The problem was Uber. They had too much quality. They had too many steps in the quality process. For 40 languages, they had four individuals touching every translation. The translator, the LQA reviewer from their LQA vendor, the Uber language lead, and the in-country reviewer.

Renato: Frequently the translator would write one thing, the LQA reviewer would correct that, the language lead at Uber would change it again, and then the in-country reviewer would revert to what the translator had done originally, correctly. It was very frustrating. The program manager for one vendor had 17 meetings every week with the buyer. There were six or seven communication channels: email, Slack, chat, the translation management system, and in-person meetings. Information was diluted, scattered, and forgotten.

Renato: There was a change in management. A new leader read our report and asked for our recommendation. I told him I am from Brazil and had just come back from a trip. My mother never saw Uber as a translated app. She only sees it in Portuguese. She does not speak English. She does not think it was translated. Maybe we were overcomplicating the concept. Four people were trying to be more compliant than the others, but nobody was asking the client what they wanted.

Renato: I did a pilot. I asked three questions to random people in Portuguese and French. Things like, if Uber was a person, where would they come from? More than two thirds said they would come from another city in Brazil or France, from a neighbourhood, from another state. Only 30% thought the app was foreign. Some even said it was German because Uber is a German word. It is an American company.

Renato: The challenge was to understand what is important to the local user in each market so we can adjust quality to the purpose. The client liked the idea and we expanded. We did a more structured approach to ask users what they think of the app. The challenge was how to ask about quality of translation without using the word translation and without creating bias. If you use the word translation, people will find mistakes.

Renato: We needed a user research strategy that was scalable and repeatable to collect unbiased data. We created a series of 10 questions addressing style, formality, whether it was approachable or distant, friendly or authoritative. We wanted to understand if the language Uber used was appropriate for the market and if it felt local.

Renato: The feedback was extremely positive. We have done this research in 40 markets. They have three products: Uber Rides, Uber Eats, and the driver app. Drivers are very important. Frequently they work in the market but use the app in a different language. In San Francisco, my Uber driver was Chinese and using the app in Chinese. Uber is not in the country of Georgia, but they translate the driver interface into Georgian because a large number of drivers in Poland are from Georgia and they wanted the app in their own language.

Renato: We looked at tone, English contamination. One product is Uber Eats, another is Uber Green. Should they change that to Uber Come in Spanish or Verde for green? Some languages are more tolerant to English than others. We have done this work for the fourth year now. The demographics change, the perception changes, the app changes.

Renato: By removing a lot of quality steps and using well-trained machine translation with light post-editing, they were able to translate more, faster, and with the same level of satisfaction as their elaborate quality control process. The satisfaction rate has improved visibly over four years. We also look at demographics: male, female, whether there is any difference in how the app addresses people, formality levels.

Renato: Different markets require different levels of quality control. Some markets like Brazil are very tolerant to mistakes and language contamination. Other markets like Japan and France require stricter quality control. Instead of sometimes having 100 quality people, today they have about 12. They save $4 million a year, deliver a better experience, and can adjust the process to the market.

Renato: Another anecdote: Uber is present in the UAE and Egypt. They used to have two versions of Arabic. Modern Standard Arabic is a language nobody speaks except TV announcers. It is grammatically correct Arabic used across the Arab world. There are close to 20 countries with Arabic as an official language and about nine major variants. A person from Tunisia speaking to a person from Egypt has very different accents and often different words for common things.

Renato: When they hired a quality control person in Egypt, that person asked, why are we doing two versions? We should have only one Arabic. Nimdzi tested it and they now do one version in Modern Standard Arabic that works for both countries. They use different words for car, tokenised so that one country gets one word and the other gets the other. Everything else is the same. That has informed the development of the app.

Renato: We also benchmark against local competitors. We asked the same questionnaire to users of competitor apps, usually local ones. In Japan and Brazil, the rate of people who thought the competitor app was foreign was the same as Uber: about 70% thinking local, 30% foreign. The perception of language had nothing to do with whether it was translated or not.

Brian: How do you communicate without creating bias in the answers?

Renato: You need to write questions from the perception of local individuals, not from an American-centric approach. We design surveys together with in-country consultants and in-country people from Uber. There are generic questions about tone, formality, and style, and others specific to the market. We avoid yes or no questions. Instead of asking if the app is good or bad, we ask about feelings and experiences. We use a Likert scale from 1 to 7 because it is more nuanced and avoids people always choosing the extremes. The bias we want to avoid is confirmation bias.

Renato: My role in this was to look at my mum and realise she has no clue she is using a translated app. That led us into another experience with Airbnb. Until a couple of years ago, if you were in Japan and wanted to rent a property in the Dominican Republic, your interface would be in Japanese but the property description would be in Spanish or English. You would click a button to translate it.

Renato: Airbnb wanted to change that. They wanted every user to have the experience entirely in their preferred language. So they went with a machine translation first approach. The property description for a Japanese user appears first in Japanese. If the translation seems weird, they can click to see the original. Instead of see the translation, you see the original. This is the first time a company has gone machine translation first for user-generated content. We have tested 26 markets. The feedback is very positive. The best feedback is when someone says they did not even realise it was a translation.

Brian: What frameworks do you recommend?

Renato: Design thinking is user-centric design. You have business thinking about money and growth. Engineering thinking about how things connect. Systems thinking about processes. Design thinking puts the user at the centre of the experience. We at Nimdzi trained our team in this concept of reframing the problem. When the client tells you they have a problem, they are usually describing a symptom. It is like going to the doctor and saying you have a fever. The problem is not the fever, it is the infection causing it.

Renato: When Uber told us they had a problem with their vendors, that was not the problem. We reframed it. The problem was the way they were addressing and talking to their clients. They were not listening. Once you start listening, your product becomes better.

Renato: There is a very simple technique called the five whys. It is used in root cause analysis. Like children, small children ask why. They are not happy with the first answer. Ask why five times. Sometimes it is three, sometimes four, but five is the rule of thumb. It takes you to the root cause of the problem instead of staying at the surface with symptoms.

Brian: If you did that where you asked why five times, the person might even answer their own question. The complaining is over. They were trying to find an answer and nobody was asking the right questions.

Renato: It is the way to go.

Brian: Renato, thank you so much. This was really great digging into the Uber study and also Airbnb.

Renato: Anytime, Brian. It is always a pleasure.

Brian: Renato Beninatto, Chairman at Nimdzi Insights. Remember, always strive for authenticity and embrace the power of native experiences.