How Digital Domain brought deep learning and AI to their visual effects toolset.
Digital Domain has been experimenting with machine learning approaches and implementing them directly into visual effects workflows for years.
Some of our latest Digital Human projects –Ant-Man & Wasp: Quantumanias, She-Hulk: Attorneyand quarry– Benefits from machine learning research that aids facial capture and facial animation steps.
Below is an excerpt from the 11th issue of Befores & Afters magazine, which highlights just a few of DD’s efforts in this area.
Back in 2016, Digital Domain was looking to revisit its already heavily used face capture system, especially in order to be able to process large amounts of data accurately and quickly. “We thought the best way to do that was to incorporate machine learning into the pipeline,” outlines Lonnie Ianazzo, producer of Digital Domain technologies. “To this end, we created Masquerade. Masquerade uses ML systems to infer high-resolution deformations from a sparse set of markers. We can use the data to directly drive high-resolution topologies for amazing emotional capture, all without limiting the performance of the actors.”
The studio has had success with Masquerade for offline capture in projects such as: Avengers: Infinity War and Avengers: EndgameBut then I began to enthusiastically explore features that work for real-time face capture. “That inspired our Digital Human Group to develop his Masquerade Live,” he says. “This captures the actor’s facial movements using a single camera on the helmet. The resulting images intelligently utilize machine learning to map the actor’s facial movements to 3D geometry, blood flow, and more. It drives several neural networks that decode the map, animated fine lines, all at 60 frames per second.”
Beyond the basic creation of CG digital humans using machine learning techniques, Digital Domain has also looked at neural rendering as a way to create more believable digital humans, starting with real-world images and photographs. This led to the creation of Charlatan. “Charlatan leverages the ability of machine learning to transform one image into another,” Ianazzo explains. “It has an advanced suite of digital aging tools and can handle real-time and mask removal applications as well.”
Charlatan was used prominently in the 2020 Malaria No More campaign, “Malaria Must Die,” adding nearly 30 years of time for Digital Domain artists to deliver future speeches to David Beckham. This was done with a combination of current Beckham and old understudy footage, which was used by the charlatan to integrate the performance. Charlatan is also available for face transfer, as is the case with the real-time post-mortem hologram of the late Teresa Teng, created by Digital Domain, during his performance.
Meanwhile, Iannazzo points to the studio’s ML Cloth system as the latest use case for machine learning. It is intended to speed up cloth simulation. “Our new simulation software uses proprietary algorithms to train a machine learning system to reproduce the quality of the original and run at over 150 frames per second,” he says. “This feature is extremely useful for speeding up fabric, muscle, and skin simulations, allowing animators to work with full-res complex characters instead of low-res rigs.”
See issue 11 for details.
