Nvidia Researchers Use AI to Convert 2D Video Clips into Detailed 3D Graphics

A new AI model from Nvidia Research, Neuralangelo uses AI to take two-dimensional video clips and transform them into detailed 3D graphic structures.

Using this technology, researchers were able to generate lifelike virtual replicas of buildings, sculptures, and other real-world objects.

Much like Michelangelo sculpts stunningly lifelike visions out of blocks of marble, New Larangelo generates 3D structures with intricate detail and textures, Nvidia says. Creative professionals can import these 3D objects into design applications and further edit them for use in art, video game development, robotics, and industrial digital twins.

Neuralangelo’s ability to transform the textures of complex materials such as roof shingles, glass panes, and smooth marble from 2D video to 3D assets far surpasses traditional methods. The high fidelity makes it easy for developers and creative professionals to quickly create virtual objects that can be used in their projects using footage captured by smartphones, enabling 3D reconstructions.

“The 3D reconstruction capabilities provided by Neuralangelo will greatly benefit creators and help them recreate the real world in the digital world,” said Ming-Yu Liu, Senior Director of Research and co-author of the paper, in a statement. said in “This tool will finally allow developers to import detailed objects, whether small statues or gigantic buildings, into virtual environments for video games and industrial digital twins.”

In a demo, Nvidia researchers showed how the model can recreate iconic objects like Michelangelo’s David and mundane objects like flatbed trucks. Neuralangelo can also reconstruct the interior and exterior of buildings. This is demonstrated in a detailed 3D model of a park on Nvidia’s Bay Area campus.

3D neural rendering model

Nvidia says previous AI models for reconstructing 3D scenes struggled to accurately capture repetitive texture patterns, uniform colors, and strong color variations. Neuralangelo employs instant neural graphics primitives, the technology behind NVIDIA Instant NeRF, to help capture these details.

The model uses a 2D video of the object or scene taken from different angles and selects a number of frames that capture different perspectives. This is similar to how an artist considers a subject from multiple sides to get a sense of depth, size and shape.

Once the camera position for each frame is determined, Neuralangelo’s AI creates a rough 3D representation of the scene, much like a sculptor would begin carving out the shape of the subject.

Then, much like a sculptor painstakingly carves stone to mimic the texture of fabric or a human figure, the model optimizes rendering to bring out the details. The end result is a 3D object or large-scale scene that can be used in virtual reality applications, digital twins, and robotics development.

Neuralangelo is one of nearly 30 projects from Nvidia Research to be announced at the Computer Vision and Pattern Recognition (CVPR) conference in Vancouver, June 18-22. His papers cover topics such as pose estimation, 3D reconstruction, and video generation.

One of these projects, DiffCollage, is a diffusion technique for creating large-scale content such as long landscapes, 360-degree panoramas, and looped motion images. Given a training dataset of standard aspect ratio images, DiffCollage treats these smaller images as sections of a larger visual, like part of a collage. This allows diffusion models to generate cohesive and large content without having to train on images of the same scale.