Microsoft plans to use AI to translate audio into foreign languages in real-time within Microsoft Edge to add subtitles and dub videos produced on major video sites.
At its developer conference Microsoft Build, Microsoft named several sites that will benefit from Edge's new real-time translation capabilities, including Reuters, CNBC News, Bloomberg, Coursera, and even Microsoft's own LinkedIn.
Interestingly, Microsoft also cites Google's YouTube as a beneficiary of the translation feature. (This is a little strange because YouTube offers the AI closed captions shown above, which can be automatically translated by adjusting the video settings. YouTube doesn't offer dubbed audio. )
Microsoft plans to translate the video from Spanish to English, and from English to German, Hindi, Italian, Russian, and Spanish. Microsoft says it plans to add more languages and video platforms in the future.
AI translation has been a staple of Microsoft apps for some time. For example, Microsoft Teams introduced inline message translation in his 2018. In 2022, Microsoft announced live translation of captions for Teams Premium subscribers. Microsoft Edge itself has supported translation of web pages or selected text since 2020. Live Captions, added as part of the Windows 11 2022 Update, adds AI-generated captions to streamed videos and pre-recorded videos like family reunions. However, no translation is performed.
Mark Hackman / IDG
Mark Hackman / IDG
Mark Hackman / IDG
But what Teams and other Microsoft apps don't do is perform voice-overs. This involves replacing spoken audio with AI-generated audio translated into the user's native language.
And doing so can be expensive. There are many companies that perform dubbing using AI. synthesys.io's “Individual” plan provides just 15 minutes of audio/video generation per month for $20. (The $41/month “Creator Unlimited” plan gives you unlimited minutes.) Celebrities.io does this for free, provided you provide the service with the URL of your YouTube video. But it seemed like it took him over 30 minutes to translate a 3 minute video, and then he gave up.
Automatic translation of web pages is one of the most useful features of web browsers. Will translated and dubbed videos become just as popular? No matter what language you speak, most people know where to go to get news in their local language. But this is definitely a differentiator for Edge, and a useful tool if you absolutely need it.
