Neuralangelo is a new AI model from NVIDIA Research for 3D reconstruction using neural networks, transforming 2D video clips into detailed 3D structures to create lifelike virtualizations of buildings, sculptures, and other real-world objects. Generate a replica.
Just as Michelangelo sculpts stunningly lifelike visions from blocks of marble, New Larangelo generates 3D structures with intricate detail and texture. Creative professionals can import these 3D objects into design applications and further edit them for use in art, video game development, robotics, and industrial digital twins.
Neuralangelo’s ability to transform the textures of complex materials such as roof shingles, glass panes, and smooth marble from 2D video to 3D assets far surpasses traditional methods. The high fidelity makes it easy for developers and creative professionals to quickly create virtual objects that can be used in their projects using footage captured by smartphones, enabling 3D reconstructions.
“The 3D reconstruction capabilities provided by Neuralangelo will be of great benefit to creators, helping them recreate the real world in the digital world,” said Ming-Yu Liu, Senior Director of Research and co-author of the paper. . “This tool will finally allow developers to import detailed objects, whether small statues or gigantic buildings, into virtual environments for video games and industrial digital twins.”
In a demo, NVIDIA researchers showed how the model can recreate iconic objects like Michelangelo’s David and mundane objects like flatbed trucks. Neuralangelo can also reconstruct the interior and exterior of buildings. This is demonstrated in detailed 3D models of parks at NVIDIA’s Bay Area campus.
See Neural Rendering Models in 3D
Previous AI models for reconstructing 3D scenes struggled to accurately capture repetitive texture patterns, uniform colors, and strong color variations. Neuralangelo employs instant neural graphics primitives, the technology behind NVIDIA Instant NeRF, to help capture these details.
The model uses a 2D video of the object or scene taken from different angles and selects a number of frames that capture different perspectives. This is similar to how an artist considers a subject from multiple sides to get a sense of depth, size and shape.
Once the camera position for each frame is determined, Neuralangelo’s AI creates a rough 3D representation of the scene, much like a sculptor would begin carving out the shape of the subject.
Then, much like a sculptor painstakingly carves stone to mimic the texture of fabric or a human figure, the model optimizes rendering to bring out the details.
The end result is a 3D object or large-scale scene that can be used in virtual reality applications, digital twins, and robotics development.
See NVIDIA Research at CVPR June 18-22
Neuralangelo is one of nearly 30 projects from NVIDIA Research to be announced at the Computer Vision and Pattern Recognition (CVPR) conference June 18-22 in Vancouver. His papers cover topics such as pose estimation, 3D reconstruction, and video generation.
One of these projects, DiffCollage, is a diffusion technique for creating large-scale content such as long landscapes, 360-degree panoramas, and looped motion images. Given a training dataset of standard aspect ratio images, DiffCollage treats these smaller images as sections of a larger visual, like part of a collage. This allows diffusion models to generate cohesive and large content without having to train on images of the same scale.
This technology can also convert text prompts into video sequences. This is demonstrated using a pre-trained diffusion model that captures human motion.
For more information on NVIDIA Research, visit CVPR.
