Does AI easily dub the language for film and television?

Finding international films that may appeal to the US market is an important part of XYZ film work.

Maxime Cottray is the Chief Operating Officer of an independent studio based in Los Angeles.

He says the US market has always been difficult for foreign language films.

“It's limited to viewers along the New York coast through arthouse film,” he says.

It's partly a language problem.

“America grew up with dubbing, not subtitles, but rather dubbing like Europe,” he points out.

However, the language's hurdles may be easy to clear with the new AI-driven dubbing system.

The audio and video of the recent Swedish sci-fi film, Watch The Skies, was fed to a digital tool called Deepeditor.

You can manipulate the video so that the actors look like they are really speaking the language in which the film is being made.

“When I first saw the results of high tech two years ago, I thought it was good, but I saw the latest cut, so it's amazing. I'm sure if the average person saw it, they wouldn't notice it – I think they speak any language, no matter what language they are,” says Cottray.

The English version of Watch The Skies was released in May at 110 AMC Theatres across the United States.

“If the film wasn't dubbed into English to contextualize this result, the film would never have made it in our cinemas in the first place,” Cottray said.

“American audiences could see independent Swedish films that would otherwise have only seen very niche audiences.”

He says AMC is planning to run more releases like this.

The DeepDator was developed by Flawless, headquartered in Soho, London.

Writer and Director Scott Mann founded the company in 2020 and has worked on films such as robbery, tournaments and final scores.

He felt that the traditional dubbing techniques of international versions of his film were not in line with the emotional influence of the original.

“When I worked on Heist in 2014 with a great cast including Robert De Niro, and I saw the film being translated into another language. That's no wonder films and television don't travel well.

“It's all out of sync, it's run differently, and from a pure film production perspective, we see very low grade products in other parts of the world.”

Flawless developed a unique technique for identifying and modifying faces based on the method originally presented in a research paper in 2018.

“Deep Editors use a combination of face detection, face recognition and landmark detection [such as facial features] “3D face tracking to understand the actor's appearance, physical behavior, and emotional performance in every shot,” Mann said.

The technology can save the original performance of actors between languages, reducing costs and time, he says, without re-shooting or re-recording.

According to him, Watch the Skies was the world's first fully visually subsided feature film.

In addition to giving the actor a different language-speaking appearance, deep editors can also transfer better performances from one take to another, or keep the original performance intact with emotional content while exchanging lines from new lines.

Thanks to the explosion of streaming platforms like Netflix and Apple, Global Film Dubbing Market is set to increase from USD 4 billion (£3 billion) in 2024 to $7.6 billion by 2033, according to a report by Business Research Insights.

Mann doesn't say how much the technology costs, but he says it varies from project to project. “I think it works out at about a tenth of the cost of filming it or changing it in other ways.”

His clients include “almost all really big streamers.”

Mann believes the technology will allow a wider audience to see the film.

“There's a huge amount of incredible films and television that English-speaking people don't see because a lot of people don't want to see it in dubbed or subtitles,” Mann says.

This technology doesn't replace actors, Mann said. He says that voice actors are used rather than replacing them with composite voices.

“What we found is that when you create the tools of the real creative and the artists themselves, that's the right way. They get the power tools to do their art and can supply them to the finished product. That's the opposite of the many approaches that other high-tech companies have taken.”

However, Natta Alexander, an assistant professor of film and media at Yale University, says that while the promise of a wider distribution is appealing, it uses AI to reconstruct the performance of non-native markets that erode the idiosyncratic and texture of language, culture and gesture.

“If all foreign films adapt to the appearance and sound of English, foreign relationships with the audience will become increasingly mediated, integrated and sanitized,” she says.

“This can discourage cross-cultural literacy and hinder support for screening subtitles or original languages.”

Meanwhile, she says that subtitle displacement, an important tool for language learners, immigrants, hearing impaired, and other people, raises accessibility concerns.

“Close captioning is more than just a workaround, it's a way to maintain the integrity of visual and auditory storytelling for a diverse audience,” says Professor Alexander.

Replacing this with automated imitation suggests a disturbing shift towards commercialized, monolingual cinema culture, she says.

“For English-speaking viewers, it may be better to ask how to build an audience willing to meet diverse cinemas, rather than asking how to make foreign films easier.”