Accuracy of YouTube's new translate audio function

A few weeks ago Google added a new function to YouTube that automatically creates captions in English to the best of its ability, resulting in instantaneous captions that are quite accurate (but far from perfect) as long as the sound quality is good enough. When the sound quality is bad it's a complete mess, but that's to be expected from a new technology.

More exciting than this though is the possibility to then take these captions and translate them into any of the languages that Google Translate supports, which means that just a short time after a new video in English is uploaded, people from 51 linguistic backgrounds can watch these subtitles and get a fairly good idea of what the subject is about. The accuracy here naturally also depends on the language, as languages like Norwegian and Spanish are generally quite good (a lot of content and/or grammar resembling English) while others like Korean still aren't.

In addition to this, if the party that has uploaded the video then adds accurate subtitles in English, these can then be used instead of audio guesswork to make a better translation.

So let's take a look at what this means in practice. Just a few days ago the United States announced a new treaty with Russia on a reduction of their nuclear stockpiles (about a third). This made news worldwide, and that means we have a number of professional translations to refer to. Here's a video with BBC Spanish containing a part of the press briefing, which can be seen in its entirety here.



First of all, let's look at the English script present in the BBC video. In it President Obama says the following:

I just concluded a productive phone call with President Medvedev, and I'm pleased to announce that after a year of intense negotiations, the United States and Russia have agreed to the most comprehensive arms control agreement in nearly two decades...in many ways, nuclear weapons represent both the darkest days of the Cold War, and the troubling threats of our time. Today we've taken another step forward in leaving behind the legacy of the 20th century, while building a more secure future for our children...broadly speaking, the new START treaty makes progress in several areas. It cuts by about a third the weapons that the United States and Russia will deploy. It significantly reduces missiles and launchers. It puts in place a strong and effective verification regime...with this agreement, the United States and Russia, the two largest nuclear powers in the world, also send a clear signal that we intend to lead, by upholding our own commitments under the Nuclear Non-proliferation Treaty, we strengthen our global efforts to stop the spread of these weapons, and to ensure that other nations meet their own responsibilities.

From the professionally done subtitles we then get an accurate translation of the above.

Acabo de terminar una productiva charla telefónica con el presidente Medvedev, y me complace anunciar que trás un año de intensas negociaciones EE.UU. y Rusia han acordado el más completo tratado sobre armos nucleares en dos décadas...en muchos sentidos las armas nucleares representan los peores días de la guerra fría y la más inquietante amenaza de nuestros días. Hoy damos otro paso para dejar atrás el legado del siglo XX y lograr un futuro más seguro para los niños...hablando de forma general, el nuevo tratado logra progresos en varias áreas. Corta en un tercio el armamento del que disponen EE.UU. y Rusia, reduce significativamente los mísiles y pone en práctice un sistema de verificación más severo...con este acuerdo Estados Unidos y Rusia, las dos principales potencias nucleares del mundo, también transmiten una señal clara sobre sus pretenciones de liderazgo, respetando nuestros compromisos en el Tratado de No proliferación Nuclear, reforzando nuestros esfuerzos globales para evitar que se propaguen estas armas y asegurando que otras naciones cumplan con sus responsabilidades.

When the video was first uploaded there were no subtitles, so I used the audio function to generate them and translate them into Spanish, so this is now the absolute minimum quality (errors in audio recognition compounded by errors in translation) one can expect from a video with as clear an orator as Obama.

Pero ha concluido que una productiva llamada telefónica con el Presidente de que hasta el y me complace anunciar que para el año que de intensas negociaciones, los EE.UU. y Rusia han acordado con el control más completo de armas acuerdo en casi dos décadas...en muchos aspectos, representan las armas nucleares, tanto los días más oscuros de la guerra fría en la mayoría de los amenaza inquietantes de nuestro tiempo. Hoy damos otro paso adelante por al dejar atrás el legado del siglo XX, mientras que la construcción de un future más seguro para nuestros hijos...en términos generales, el tratado de empezar de nuevo hace que el progreso en varias áreas a la policía cerca de un tercio, las armas nucleares que los EE.UU. y Rusia que los pobres, que reduce significativamente los misiles y lanzadores, que pone en marcha un fuerte y eficaz verificación de la fusión de...con este acuerdo los Estados Unidos y Rusia, las dos mayores potencias nucleares en el mundo, también enviar una señal clara de que en diez pero manteniendo nuestros proprios compromisos en virtud de la no proliferación nuclear en virtud de tratados fortalezcamos nuestros esfuerzos globales para detener la propagación de estas armas y para garantizar que otras naciones alli mí proprias responsabilidades.

Not bad. There are certainly some glaring errors (cuts is translated as policía, as it seems to have been confused with cops) but the general message comes through, as can be seen when we now translate that text back into English (now this includes audio errors, compounded by two automatic translations).

But production has concluded that a phone call with the President that I am pleased to announce that for the year of intense negotiations, the U.S. and Russia have agreed with the most comprehensive arms control agreement in nearly two decades ... in many aspects of nuclear arms, both the darkest days of the Cold War most disturbing threat of our time. Today we take another step forward to overcoming the legacy of the twentieth century, while building a safer future for our children ... in general terms, the treaty of starting over makes progress in several areas the police about a third, the U.S. nuclear weapons Russia and the poor, which greatly reduces the missiles and launchers, which sets up a strong and effective verification of merging ... with this agreement the United States and Russia, the two largest nuclear powers in the world, also send a clear signal that in ten while maintaining our own set of commitments under the nuclear nonproliferation treaty strengthen our global efforts to halt the spread of these weapons and other nations to ensure that my own responsibility there.

Just an hour or so ago the White House got around to uploading the complete transcript, so now we're able to compare the translation produced by this with that by audio alone. Now there is no reference to the "cops":

Me acaba de concluir una llamada telefónica productiva con Presidente Medvedev. Y me complace anunciar que después de un año de intensa negociaciones, los EE.UU. y Rusia han acordado la más completo del acuerdo de control de armas en casi dos décadas...en muchos sentidos, las armas nucleares representan tanto a los días más oscuros de la Guerra Fría, y las amenazas más preocupantes de nuestro tiempo. Hoy en día, hemos dado otro paso adelante -- al dejar atrás el legado del siglo 20, mientras que la construcción un futuro más seguro para nuestros hijos...en términos generales, el tratado START nuevo avance en varios ámbitos. Corta - alrededor de un tercio - las armas nucleares que el EE.UU. y Rusia desplegará. Reduce significativamente los misiles y lanzadores. Se pone en marcha un régimen de verificación fuerte y eficaz...con este acuerdo, los Estados Unidos y Rusia -- las dos mayores potencias nucleares del mundo -- también enviar una señal clara de que tenemos la intención de plomo. Por la defensa de nuestros proprios compromisos en el marco del Nuclear Tratado de No Proliferación, estamos fortaleciendo nuestros esfuerzos mundiales para detener la propagación de estas armas, y asegurarse de que otras naciones a satisfacer sus proprias responsabilidades.

And pasting this back into Google Translate gives us an idea of what it looks like to a Spanish speaker.

I just concluded a telephone call with President Medvedev productive. And I'm pleased to announce that after a year of intense negotiations, the U.S. and Russia have agreed on the most comprehensive arms control agreement in nearly two decades ... in many ways, nuclear weapons represent both the darkest days of the Cold War, and most troubling threats of our time. Today, we have taken another step forward - to leave behind the legacy of the 20th century, while building a safer future for our children ... in broad terms, the START treaty further progress in several areas. Short - about a third - nuclear weapons that the U.S. and Russia deployed. Significantly reduces the missiles and launchers. It sets up a strong verification regime and effective ... with this agreement, the United States and Russia - the two largest nuclear powers in the world - also send a clear signal that we intend to lead. By defending our own set commitments under the Nuclear Nonproliferation Treaty, we are strengthening our global efforts to halt the spread of these weapons, and ensure that other nations meet their own responsibilities.

One interesting error that has slipped through though is the part saying "we intend to lead". It looks like an improvement over the original "we intend to ten" (not sure where that came from), but the Spanish version actually says "tenemos la intención de plomo", meaning "we have the intention to lead", as in the element lead that melts when the temperature reaches 327.46 °C.

The end result of this technology is clear: using the internet in languages other than English (and for those who are hard of hearing) has now become that much easier. In order to get a general idea about a video in English, just click four times and you have a general approximation of what is being said. The next big step needs to be more accurate audio recognition, because at the moment only speeches from the clearest of speakers result in a legible transcript. But so far so good.