Subscribe to The Daily Upside newsletter for more vivid and insightful business and economic news. It’s completely free and we guarantee you’ll learn something new every day.
For artists, lips are the most difficult thing to draw. AI video programs seem to face the same problem.
Baidu is working on technology that will allow mouth movements to leap past the anxiety and uncanny valley sensations typically inflicted by AI-generated 3D videos. Baidu’s system uses two neural networks. One network first creates a 3D video from the audio data, and the second network corrects and refines the lip movements created by the first network. The Chinese search giant noted that neural networks are trained on various AI concepts such as “computer vision, augmented/virtual reality and deep learning.”
This technique relies on a machine learning concept called “principal component analysis”, or the process of transforming high-dimensional data into low-dimensional data, essentially extracting the main features of the data. The system uses neural networks to perform this kind of analysis on audio data, from which he creates a 3D video.
The analysis then passes through a second neural network, essentially checking and correcting the behavior of the first network. This analysis is specifically transformed into a 3D video of lip movements and applied to a “pre-built 3D base avatar model to obtain a 3D video with lip movement effects”.
In its application, Baidu says the technology allows it to do so without using “blendshapes,” a common method of animating facial expressions that rely on scanning a person’s face to get realistic facial movements. You mentioned that you can do 3D face rendering.
“Lip movements are an important part of the 3D video generation process,” Baidu said in the application. “3D video generation is usually limited by his Blendshape, which is mostly artificial, so the lip movements generated lack expressiveness and detail.”
Rendering mouth movements using AI is very difficult. That’s because AI can’t yet capture the “microexpressions” of a human face, or the small movements that convey a person’s nonverbal communication and emotions, said Jake Meimer, vice president of innovation at The Glimpse Group. . This is why animators and his VFX designers often use human subjects as models, “part of the recipe for getting it right,” he said.
“It’s very, very difficult to capture the soul of an individual,” Meimer said. “They are called microexpressions because they are almost subconscious. The brain recognizes them. The light in the eyes, the life behind a person, which is very difficult to capture and reproduce digitally.” .”
But if developers break through that barrier, it could open doors beyond AI-generated VFX, Meimer says. An AI-based system that captures the subtleties of non-verbal communication and transforms it into his VR or AR avatar in real time means the difference between VR environments where people want to spend time and his VR environments where they don’t. There is likely to be.
“Once we unlock this capability, this will be a tipping point for telecommunications,” Meimer said. “The reason I think Baidu is focused on this video technology is that it is kind of the first step before we tackle the spatial and 3D representation of people.”
This isn’t the first time we’ve seen Baidu interested in AI-based video. The company is also seeking to patent an AI-based technology that generates videos from reference photos (although the sample photos in the application were strangely faceless). Given the company’s patent activity to date, Meimer said the filing would not be the last in its efforts to perfect AI-generated facial movements and body gestures.
Baidu has worked hard to keep pace with its competitors throughout the AI race. The company’s initial launch of his AI chatbot, Ernie, fell through, but founder Robin Lee said in late May that Baidu had applied the large-scale language model behind the chatbot. Announced that the upgrade will start soon. The company also announced a $145 million venture fund backing AI application startups in early June.
But standing out among major players like Google, Microsoft, and OpenAI is no easy task. Making AI-based videos that actually look good could be part of Baidu’s plans.
Have any comments, tips or suggestions? mail address: [email protected] Or DM us on Twitter @patent drop. If you would like to receive the Patent Drop in your inbox, click here to subscribe.
