Created by filmmakers and scientists, Flawless may finally have crossed the uncanny valley of lip-dubbing content.
Currently, watching foreign language content dubbed into English is a jarring task.
The proliferation of local-language film and TV brought about by streaming platforms makes the experience of watching a dubbed Danish cop drama or a Spanish boarding-school soap opera a nightmarish trip into the uncanny valley, a term broadly applied to technology that doesn't give us the benefit of the doubt, but in fact takes our human likeness and throws it into a dystopian nightmare blender.
That's where AI-powered filmmaking studio Flawless comes in. Want to stream the hottest shows and still be able to sleep at night (without imagining cruel scene cuts with mismatched lip movements?). Founded in 2018 by multi-directors Scott Mann and Nick Lines, Flawless' proprietary technology, TrueSync, maps actors' faces and delivers some of the most impressive translations we've seen in the AI wilderness. The company previewed a sizzle reel of its work at this year's Cannes Film Festival. variety Share it exclusively here.
Designed to protect artistic copyright and comply with Hollywood labor unions, “Flawless” raises big questions about the value of international content and its groundbreaking potential in the U.S. market. variety We are also happy to report that Flawless has officially entered the film distribution business. The company has entered into a joint venture with XYZ Films and Tea Shop Productions to acquire foreign film rights. The partnership will see Flawless translate the projects into English with what it promises are “perfect lip-sync footage” and release them in relevant territories. The first title to be released will be Victor Dannell's award-winning sci-fi film “UFO Sweden,” which will be released domestically under the title “Watch the Skies.” A series of projects are planned for the coming months*.
Mr. Mann variety We'll explore fear and loathing of AI in show business, building codes of ethics, and how the Uncanny Valley brain works.
What inspired you to get into this field?
This all started in 2018. My background is directing, writing and producing films. I stumbled across the first paper that introduced generative AI to the world. It was called “Deep Video Portrait” and was presented at the SIGGRAPH conference, the Oscars of science. I was blown away.
Having worked in VFX for my whole life, it was clear to me how this technology would impact our industry, so I reached out to the authors of the paper and started a company together with them. [my partner] Nick Lynes. Christian Theobalt, Pablo Garrido, and Hyeongwoo Kim founded our science department and have grown it into the leading computer-driven generative science department in the world. That's our secret sauce.
There's a lot of fear and uncertainty around AI in Hollywood right now. What's interesting about your company is that you're optimistic.
We founded this company from a filmmaking perspective. I love art. Essential to our mission is seeing this technology as the ability to make tools. There's another way of looking at this, that this is about eradicating creative people and stealing art. Exploiting it like Napster did, tyrannizing for the sake of destruction, destroying it. But we believe in the expression of human storytelling, and film is the most advanced form of that. Technology amplifies that, adding another layer of creativity. There are benefits. And there's an economics to this whole business that relies on certain protections and an understanding of how it works.
There is a significant lack of education in AI in this town. How does this impact your business?
This is a global problem. It's not just a Hollywood problem. Education and understanding are key. There are some companies that are reluctant to educate. Let's call them flawed companies. Because education exposes a kind of behind-the-scenes manipulator. We need to have an honest conversation about where and how it's confusing and where there are good points. We've prioritized understanding with studios and unions (including SAG-AFTRA) for the past few years, saying, “This is how you should use it.” Next [AMPTP contract] Negotiations are something everyone should start talking about now. A transformation is coming to our industry and we need to be prepared for it.
That's a good point. Is the industry paying attention to this in any meaningful way?
I can confidently say that the mindset has changed a lot. Before last year's strike, there was a cautious path to approving the use of AI in projects. People were discussing it early on and seemed to understand. It was born out of a desire to improve the process. But there is a fear of crossing boundaries. Right now, the industry is in a really bad place in some places, and many people don't want to jump in too quickly to the wrong solutions.
How did Flawless develop its code of ethics?
It was born out of a problem I personally experienced as a filmmaker. I've always wanted to improve dubbing in films. I made a film I cared very much about, “Heist,” with Robert De Niro. It was so intimately made, so full of nuance, and it was a joy to deliver what we made. I saw the foreign-language version that had been dubbed; the script had been completely rewritten, the mouth movements were completely different. I realized, “Oh no, no wonder things aren't working.”
We tried various VFX techniques to fix this, such as head scanning, but they were neither practical nor crossed the uncanny valley. This meant that it was ethically impossible to remove or destroy entire areas that contributed to the film. We learned that ethics and legal rights go hand in hand.
Speaking of the uncanny valley, it's truly brutal to see lip dubbing in its current form, especially as streaming has escalated it.
Humans observe each other in great detail. Our brains are programmed to be hyper-vigilant when we see something unusual. Throughout the evolution of filmmaking, our brains have been tricked and retrained. Your subconscious is working on reality checking. If something doesn't harmonize well, especially if the sound and picture don't harmonize well, you won't be able to empathize with or believe in what you're watching. Think about how much expression you can convey without hearing the sound, through the actor's face, gestures and acting. There's a lot of meaning there. All these reasons are [present lip dubbing] Yes, it is a problem. But it's good to have a clear problem to solve.
It's also a pretty groundbreaking development for foreign cinema, creating a new market for stars and potentially eliminating many American remakes of famous films.
“Especially over the past year, showing our work to filmmakers has helped them realize the possibility of going from the local stage to the global stage. It's a great opportunity and inspiration to get their work out there. They're really excited to have their work shown to a wider audience, especially in the U.S..”
Have any studios or companies tried to buy you?
Yes, when we first landed in Hollywood. We had a very short promo video in 2021, and some people tried to negotiate the price right away. Nick and I agreed that what we wanted was to serve everyone, not just the interests of one studio. We want to be Swiss, because otherwise it would be very limiting. But it helped us raise money and grow the company. It's a good problem to have.
You also want to get into the distribution business.
We have seen ourselves as a studio, to some extent. We want to be future-oriented. We need to do everything right, from pre-production to production to release. All these stages are influenced by technology. In my outlook, we should be able to reach a film to 10 times the audience. And with these tools, we can make a great film for less money. If you look at our first film, “UFO Sweden,” it is of course translated into English, but we also give the filmmakers the freedom to add and change things that they couldn't do at the beginning. You can try to do one job, like an algorithmic component, but I think the real power is in making the film and distributing it. That's everything. And in terms of post-production, we can do so much more. Look at Pixar. They can iterate and improve their film up to the moment of release.
*The joint venture acquisition with Flawless will be led by Ryan Black, James Emanuel Shapiro of XYZ and James Harris of Tea Shop.