Inside the orb, the world shrinks to a sphere of white light that flashes. Outside the orb’s metal skeletal frame is darkness. Imagine being strapped to a chair in this device. Voices from Darkness suggest expressions such as how to pose your mouth and eyebrows, scenarios to react to, phrases to say, and emotions to embody. At irregular intervals, a voice tells you not to worry and warns you that more flashes are coming soon.
advertisement
“I wouldn’t say I was surprised, but it was a very overwhelming space,” said the actor when asked. Scientific American For privacy reasons, his name is withheld. He describes his experience with the “orb,” a term he refers to the photogrammetry booth used to capture his likeness during the production of his major 2022 video his game. [a magnetic resonance imaging machine]’ he says. “It was really very sci-fi.” The actor’s experience was part of a scanning process that allowed media production studios to photograph cast members in various poses and later animate them to create mobile, adaptable digital avatars that could perform virtually any action or motion in a realistic video sequence.
Advances in artificial intelligence have steadily made it easier to create digital doubles like this without an intense session in the orb. Some actors are concerned about the future potential of digital doubles stealing jobs as studios pressure them to unsubscribe from their likenesses. This is one of the motivations for the members of the union SAG-AFTRA (Screen Actors Guild – Federation of American Television and Radio Artists) to go on strike. In a statement released days after the strike was announced in mid-July, the union said: “Performers need to protect our footage and performances to prevent human performances from being replaced by artificial intelligence technology.”
While the possibility of being replaced by AI is unnerving, the digital doubles we see in media production today still rely on human performers and special effects artists. Learn how this technology works and how AI is disrupting established processes.
How Digital Double Tech works
advertisement
Over the past 25 years or so, it has become increasingly common for big-budget media production companies to create digital doubles of at least some of the performers’ faces and bodies. This technology is almost certainly playing a role in movies, TV shows, or video games with extensive digital effects, elaborate action scenes, or actors portraying characters of varying ages. “It’s kind of become an industry standard,” says his Chris MacLean, visual effects supervisor for his company MacLean VFX.
A photogrammetry booth is an area surrounded by hundreds of cameras, sometimes arranged in a sphere, sometimes around a square room. The camera captures two-dimensional images of her in high resolution with thousands of deliberately superimposed faces. If you want to speak or emote for an actor’s role, you’ll need photos of different facial movements. As such, lead actors require a more extensive scan than supporting or background actors. Similarly, large scale setups are used to scan the body.
Using these data, visual effects (VFX) artists transform models from 2D to 3D. The point is that the photos overlap. Based on camera coordinates and overlapping overlapping sections, the images are mapped and folded relative to each other in a process similar to digital origami. An artist can rig the resulting 3D digital double into a virtual “skeleton” and animate the motion either by directly following his captured real-world actor’s performance or by combining that performance with a sequence of computer-generated movements. Animated figures can then be placed in the digital landscape and given dialogue. Technically, it is possible to use scans of people to create photorealistic video footage of actors doing and saying things they never did or said.
advertisement
Special effects artists can also apply digital performances of actors to virtual avatars that look very different from humans. For example, the aforementioned video game actor says he made a face in an orb and recorded his lines in a recording booth. He also physically acted out many scenes with other performers in a separate studio for his capture of motion, a process similar to photogrammetry but designed to record body movements. However, when players work on the final product, this actor will not be onscreen. Instead, his digital double was modified to look like a villain with a specific appearance. In this way, the final animated character represented both the work of the actor and the characteristics of his character in the video game.
Film and television production have used this process for decades, but it has historically been labor-intensive and expensive. Despite the difficulties, digital doubles are common. Production teams often use these for small adjustments involving dialogue and action. The technique is also used for large-scale editing, such as taking a group of 100 background actors and morphing and duplicating them into a digital crowd of thousands. However, when the original footage is closer to the desired final output, it becomes easier to accomplish such feats in a convincing way. For example, it’s difficult to edit background actors scanned in costumes meant to recreate the costumes worn in 19th-century Europe into a dystopian future in which digital doubles wear spacesuits, McLean said. “I don’t think there’s any way a studio could have that kind of patience,” he added.
But the same kind of machine learning technology behind ChatGPT, generative artificial intelligence, is starting to make many aspects of the digital double process faster and simpler.
AI raid
advertisement
McLean points out that some VFX companies are already using generative AI to speed up the process of changing the appearance of digital doubles. This makes it easier to “age down” famous actors in movies like: Indiana Jones and the Dial of Fate, This includes flashbacks of a younger-looking version of Harrison Ford, who is now 81 years old. Vladimir Galat, chief technology officer at mobile photogrammetry company Scan Track, said AI could also help with face replacement, superimposing the likeness of an actor onto a stuntman (essentially a licensed deepfake).
Garratt says advances in AI have made some photogrammetry scans unnecessary. Generative models can be trained on existing photos, videos, or even people who are no longer alive. VFX production company Digital Domain. Avengers/Endgame, says it is also possible to create fake digital performances by historical figures. “It’s a very new technology, but it’s a growing part of our business,” said Hanno Basse, chief technology officer at Digital Domain.
So far, living humans have still been involved in the creation of performances “by” the deceased. Real-life actors act out scenes, and effects artists replace their faces with those of historical figures. “We feel it’s important to combine the nuances of the actors’ performances with AI and machine learning toolsets to achieve photorealistic results that captivate audiences and can cross the uncanny valley,” Bass says, referring to the eerie sensations sometimes triggered by things that look almost human but aren’t quite human.
Anxiety about robot replacement
advertisement
There is a difference between tuning a digital double and fully replacing human performance with AI, says computer engineer Jeong Joon “JJ” Park. He is currently studying computer vision and graphics at Stanford University and will be joining the University of Michigan this fall. Park points out that the uncanny valley is wide and that there is no generative AI model that can generate a complete, photorealistic moving scene from scratch, and the technology is not there yet. Getting there “will require a giant leap in the intelligence we are developing,” he says. (Although AI-generated images can be difficult to distinguish from the real thing, creating a realistic still image is much easier than creating a video to represent a 3D space.)
Still, the threat of abuse of an actor’s image rights is looming. If you could easily swap one person’s face for another, what would stop filmmakers from putting Tom Cruise in every shot of an action movie? What stops studios from replacing 100 background actors with just one and using AI to create the illusion of many actors? Say it means you have the right. But there are wide-ranging exceptions to artistic and expressive uses, and filmmaking could easily fall under that designation, she notes. And regardless of the law, individuals can legally sign contracts that transfer their image rights to production companies, said media and technology attorney Jonathan Brabin. When it comes to protecting a person’s digital likeness, it all depends on the details of the contract. SAG-AFTRA is well aware of this situation.
Actors who have played video game villains were comfortable getting their roles scanned last year. “The company I worked with was pretty good,” he says. But in the future, he may not come to an agreement any time soon. “What AI can do with face capture and what we’ve seen [prestrike negotiations], is scary, ”he says. This actor loves video games. He’s thrilled to be in the production, and he hopes he’ll do it again. But before that, “I’m going to double check my paperwork, contact my agency, and possibly contact my lawyer,” he said.
