Posted:
Today we announced that the Google Translate app now does real-time visual translation of 20 more languages. So the next time you’re in Prague and can’t read a menu, we’ve got your back. But how are we able to recognize these new languages?

In short: deep neural nets. When the Word Lens team joined Google, we were excited for the opportunity to work with some of the leading researchers in deep learning. Neural nets have gotten a lot of attention in the last few years because they’ve set all kinds of records in image recognition. Five years ago, if you gave a computer an image of a cat or a dog, it had trouble telling which was which. Thanks to convolutional neural networks, not only can computers tell the difference between cats and dogs, they can even recognize different breeds of dogs. Yes, they’re good for more than just trippy art—if you're translating a foreign menu or sign with the latest version of Google's Translate app, you're now using a deep neural net. And the amazing part is it can all work on your phone, without an Internet connection. Here’s how.

Step by step
First, when a camera image comes in, the Google Translate app has to find the letters in the picture. It needs to weed out background objects like trees or cars, and pick up on the words we want translated. It looks at blobs of pixels that have similar color to each other that are also near other similar blobs of pixels. Those are possibly letters, and if they’re near each other, that makes a continuous line we should read.
Second, Translate has to recognize what each letter actually is. This is where deep learning comes in. We use a convolutional neural network, training it on letters and non-letters so it can learn what different letters look like.

But interestingly, if we train just on very “clean”-looking letters, we risk not understanding what real-life letters look like. Letters out in the real world are marred by reflections, dirt, smudges, and all kinds of weirdness. So we built our letter generator to create all kinds of fake “dirt” to convincingly mimic the noisiness of the real world—fake reflections, fake smudges, fake weirdness all around.

Why not just train on real-life photos of letters? Well, it’s tough to find enough examples in all the languages we need, and it’s harder to maintain the fine control over what examples we use when we’re aiming to train a really efficient, compact neural network. So it’s more effective to simulate the dirt.
Some of the “dirty” letters we use for training. Dirt, highlights, and rotation, but not too much because we don’t want to confuse our neural net.

The third step is to take those recognized letters, and look them up in a dictionary to get translations. Since every previous step could have failed in some way, the dictionary lookup needs to be approximate. That way, if we read an ‘S’ as a ‘5’, we’ll still be able to find the word ‘5uper’.

Finally, we render the translation on top of the original words in the same style as the original. We can do this because we’ve already found and read the letters in the image, so we know exactly where they are. We can look at the colors surrounding the letters and use that to erase the original letters. And then we can draw the translation on top using the original foreground color.

Crunching it down for mobile
Now, if we could do this visual translation in our data centers, it wouldn’t be too hard. But a lot of our users, especially those getting online for the very first time, have slow or intermittent network connections and smartphones starved for computing power. These low-end phones can be about 50 times slower than a good laptop—and a good laptop is already much slower than the data centers that typically run our image recognition systems. So how do we get visual translation on these phones, with no connection to the cloud, translating in real-time as the camera moves around?

We needed to develop a very small neural net, and put severe limits on how much we tried to teach it—in essence, put an upper bound on the density of information it handles. The challenge here was in creating the most effective training data. Since we’re generating our own training data, we put a lot of effort into including just the right data and nothing more. For instance, we want to be able to recognize a letter with a small amount of rotation, but not too much. If we overdo the rotation, the neural network will use too much of its information density on unimportant things. So we put effort into making tools that would give us a fast iteration time and good visualizations. Inside of a few minutes, we can change the algorithms for generating training data, generate it, retrain, and visualize. From there we can look at what kind of letters are failing and why. At one point, we were warping our training data too much, and ‘$’ started to be recognized as ‘S’. We were able to quickly identify that and adjust the warping parameters to fix the problem. It was like trying to paint a picture of letters that you’d see in real life with all their imperfections painted just perfectly.

To achieve real-time, we also heavily optimized and hand-tuned the math operations. That meant using the mobile processor’s SIMD instructions and tuning things like matrix multiplies to fit processing into all levels of cache memory.

In the end, we were able to get our networks to give us significantly better results while running about as fast as our old system—great for translating what you see around you on the fly. Sometimes new technology can seem very abstract, and it's not always obvious what the applications for things like convolutional neural nets could be. We think breaking down language barriers is one great use.

Posted:
The Google Translate app already lets you instantly visually translate printed text in seven languages. Just open the app, click on the camera, and point it at the text you need to translate—a street sign, ingredient list, instruction manual, dials on a washing machine. You'll see the text transform live on your screen into the other language. No Internet connection or cell phone data needed.

Today, we’re updating the Google Translate app again—expanding instant visual translation to 20 more languages (for a total of 27!), and making real-time voice translations a lot faster and smoother—so even more people can experience the world in their language.
Instantly translate printed text in 27 languages
We started out with seven languages—English, French, German, Italian, Portuguese, Russian and Spanish—and today we're adding 20 more. You can now translate to and from English and Bulgarian, Catalan, Croatian, Czech, Danish, Dutch, Filipino, Finnish, Hungarian, Indonesian, Lithuanian, Norwegian, Polish, Romanian, Slovak, Swedish, Turkish and Ukrainian. You can also do one-way translations from English to Hindi and Thai. (Or, try snapping a pic of the text you’d like translated—we have a total of 37 languages in camera mode.)

To try out the new languages, go to the Google Translate app, set “English” along with the language you’d like to translate, and click the camera button; you'll be prompted to download a small (~2 MB) language pack for each.

Ready to see all of these languages in action?
And how exactly did we get so many new languages running on a device with no data connection? It’s all about convolutional neural networks (whew)—geek out on that over on our Research blog.

Have a natural, smoother conversation—even with a slower mobile network
In many emerging markets, slow mobile networks can make it challenging to access many online tools - so if you live in an area with unreliable mobile networks, our other update today is for you. In addition to instant visual translation, we’ve also improved our voice conversation mode (enabling real-time translation of conversations across 32 languages), so it’s even faster and more natural on slow networks.
These updates are coming to both Android and iOS, rolling out over the next few days.

Translate Community helps us get better every day
On top of today’s updates, we’re also continuously working to improve the quality of the translations themselves and to add new languages. A year ago this week, we launched Translate Community, a place for multilingual people from anywhere in the world to provide and correct translations. Thanks to the millions of language lovers who have already pitched in—more than 100 million words so far!—we've been updating our translations for over 90 language pairs, and plan to update many more as our community grows.

We’ve still got lots of work to do: more than half of the content on the Internet is in English, but only around 20% of the world’s population speaks English. Today’s updates knock down a few more language barriers, helping you communicate better and get the information you need.

Posted:
People use Google Translate a whole lot—we translate over 100 billion words a day! However, in the past, our translation systems have generally been better at making sense of government and business documents than in helping people casually communicate.

But that’s all changing thanks to people like you and a recent update we rolled out. So the next time you translate informal speech in Google Translate, you might just find a better translation. Here's an example of how it’s improved:
So how exactly are people like you impacting Google Translate? Well, with Translate Community hundreds of thousands of people have generously donated their time in service of cross-language communication. It’s fun and really easy: tell us what languages you speak; choose to either see a phrase and translate it on your own or correct current translations already in the system. Based on translations from the community, we will incorporate corrections and over time learn the language a little better.

There’s a whole lot more work to do, but with more help from everyday people through Translate Community, we can continue to improve the 90 languages we already speak and keep adding more.

Posted:
During the Translate Community: Google I/O Challenge nearly 2.5 million phrases or about 12 million words were translated and validated by participants during Google I/O and beyond. From Afrikaans to Zulu, we saw approximately 75,000 people representing all of the 117 languages available in Google Translate Community take the lead and improve Google Translate for the languages they speak.
Some of the top contributing languages were what you might expect based on the number of speakers of each language; Spanish, Russian, French and Portuguese led the way through the challenge and languages like Bengali and Vietnamese notably moved their way up the rankings.
Of the languages that are not yet in Google Translate, Kyrgyz speakers shined the brightest with nearly 40,000 phrases translated and validated during the challenge. The Kyrgyz community has continued to plan events and rally to help add their language to Google Translate.

Thank you to the I/O Extended event organizers, Google Developer Groups and everyone who contributed to improving their language in Google Translate throughout this 10-day-challenge. We're excited to continue to work together in improving Google Translate.

Posted:
For the last 10 months, multilingual users around the world have flocked to the Google Translate Community to help improve their language(s) on Google Translate through translating and validating common phrases.

Since launch, we've seen some amazing contributions, from Kyrgyz speakers who are getting us closer to adding their language to Google Translate, to Bengali speakers who organized 80 translate-a-thon events, significantly increasing translation quality for their language.
Translate Community: Google I/O Challenge (May 26 - June 5, 2015)
Now, we're challenging all Google I/O attendees (onsite and offsite!) to represent your language(s) during the Translate Community: I/O Challenge running from May 26 to June 5, 2015. Our goal is to reach over 5 million total contributions during the challenge.

You can make meaningful contributions in just a few minutes, and remember that all contributions matter—we encourage you to spread the word in your local community and amongst your friends and family to increase the contributions for your language(s). More contributions mean higher quality translations for your language(s), or helping your language(s) become supported on Google Translate, if they aren’t yet.

To get started:
  • Sign up in the new version of Translate Community at g.co/translate/io
  • Set your language(s) and contribute with as many high-quality translations / validations you have time for
  • Invite others to join the challenge and show support for your language on social with the official #io15 & #loveyourlanguage hashtags
You can follow which languages are getting the most contributions on our Google+ page, where we’ll post updates on who's leading the way throughout the challenge. Besides helping your language rise to the top of our leaderboard, if you’re one of the top high-quality contributors, you’ll get a Google Translate certificate for your linguistic legerdemain and might even get a shoutout on our social channels.

Built with Polymer
In the spirit of Google I/O, we recently released a new version of the Translate Community using Polymer. In addition to supporting your language, be one of the first to try out the new look of Translate Community.
This new version takes advantage of Web Components in Polymer. We're one of the first teams at Google to use Polymer this way—it’s now much easier to add new features like badges, upgrade our design, and ensure it works great on smartphones and tablets, in addition to desktop. We're looking forward to leading the way by offering our community a fun and engaging place to make a positive impact.

Posted:
Have you ever slashed the wind with your friends? Attached a gold leaf to the back of a Buddha statue? Or caught the moon in your hands?

You probably have, but if you don’t speak Vietnamese, Thai or Bengali you probably called it something else. These are some phrases Google Translate learned to understand a little better during the first of a series of translate-a-thons held over the last month in Bangladesh, Myanmar, Vietnam and Thailand, part of a “Love your Language” project to get languages better represented on the Web. (For those playing at home, the Vietnamese call it slashing the wind when they gossip, the Thais say they attach a gold leaf to the back of a Buddha statue when they do something selfless, and Bengalis say they’ve caught the moon in their hand when they receive something rare).

Google Translate provides free translation in 90 languages, but for those that don’t have much presence on the Web - like Myanmar, Bengali, Vietnamese and Thai - it could use a little help.

This is where the Translate Community tool and passionate language speakers can make a difference. By letting people validate, match, rate and supply translations, it can boost the translation of these languages online for millions of people.

Try it out yourself at translate.google.com/community
Since kicking off on International Mother Tongue Day (21 Feb), more than 50,000 people have come together, online and off, to use this tool to improve translations for Bengali, Myanmar, Vietnamese and Thai. They gathered at startup hubs in Yangon and university campuses in Vietnam, Bangladesh and Thailand.
This gentleman showed up at Phandeeyar Innovation Hub in Yangon with a handwritten list of phrases he wanted Google to get right.

So far, more than 100 translate-a-thons have been held and more than 10 million words have been added. That's 17 times more words than Tolstoy used for War and Peace, 12 times the number of words in the English version of the Bible.

It's made a huge difference. The quality of Bengali translations are now twice as good as they were before human review. While in Thailand, Google Translate learned more Thai in seven days with the help of volunteers than in all of 2014.

The following graph show the spike in Translate Community inputs over the last month, March 26 saw a major spike for Bangladesh Independence day, setting a new record for the largest volume of translations contributed in 24 hours.
Google Translate Community surfaces a random selection of popular words and phrases that users are asking Google Translate to explain in their language — from music lyrics, to local recipes, to human rights.
Teaching Google Vietnamese at the University of Technology in Ho Chi Minh City
Teaching Google Vietnamese at the University of Technology in Ho Chi Minh City We sometimes think the offline world and online world are separate. They're not. A huge thank you to all the people that joined us for the Love your Language series of events. Your efforts have made it easier for people from downtown Dhaka to upcountry Thailand to access the web in a language they understand.
Teaching Thai at Siam University International College
And the improvements don’t need to stop. By joining the Translate Community you can join us in making the web work better for everyone — no matter what language you speak.

Posted:
February 21 marks the 15th anniversary of the UNESCO declaration of International Mother Language Day. Since then each Mother Language Day has promoted the preservation and protection of the approximately 7,000 languages that are spoken throughout the world, half of which are estimated to become extinct in a few generations.

In honor of Mother Language Day 2015, we've decorated the Google Translate homepage with an illustration that celebrates this year's theme of "inclusive education through and with language." Language education helps people connect with others both within and outside their local community.


Click on our illustration on the homepage to visit the Google Translate Community where you can help add new languages to Google Translate and improve those that are currently supported. We've already seen Cantonese, Kyrgyz and Pashto speakers contribute a lot, and we hope to continue our collaboration with these communities so we can eventually add these languages.


We hope you join us for Mother Language Day to improve translation for everyone and show pride for your language. We'll be highlighting the top languages with the most contributions to Translate Community over the next 48 hours on our Google+ page. Show some love for your language and help it get to the top of the list by contributing today!

Posted by Aaron Babst, Community/Program Manager, Google Translate

(Cross-posted on the Inside Search Blog)