Developed by researchers at the University of California, Berkeley, the University of California, San Diego, and Google, NeRF first got us started in March 2022 when nVidia unveiled a super-fast “Instant NeRF” demo at the GTC developer conference. appeared on the radar of
In effect, you walk around the scene taking photos and videos with your smartphone and uploading the results to the service. In this service, neural networks use various AI techniques to take a series of real-world images and use them to build a 3D model. your subject and its environment.
This model is then used to create wild flying camera shots, photorealistic 3D assets and environments for use in video games, VR experiences, or a variety of other uses. You can do all sorts of things like
The resulting video can be absolutely ridiculous, as demonstrated in this extraordinary physics-bending Hilux video created (and expertly edited) by Arata Fukae of Japan. I have. Remember, this is a one-man production.
Last month, Luma Labs lowered the price for this kind of service to US$1 per scene. This API allows you to “look out from 2-3 levels, get a video walkthrough of an object or scene” and create an “interactive 3D scene that can be directly embedded, for building interactions in a traditional 3D pipeline. coarse textured models, and pre-rendered 360-degree images and videos.”
Creator Karen X Cheng used Luma Labs AI to create the monster dolly zoom effect in the video below. Click the Twitter thread to see how it was captured and post-edited.
Others have combined 3D NeRF capture with separate generative AI services to get photorealistic scene captures that look rather monotonous and finish with more impressive textures. Creator Bilawal Sidhu calls it ‘real world reskinning’ and while the results may be a bit janky at this point, in the near future it will be seamlessly integrated into systems that can edit and edit. There is no doubt about that. Add to his NeRF scene using natural language prompts.
🖼️ It’s still fun to “reskkin” the real world with 3D capture + generative AI ✨
🌐 Reality capture techniques like photogrammetry and NeRF allow you to capture spaces, places, and objects of interest. The ever-growing library of assets available… pic.twitter.com/IFctTlLLfo
— Bilawal Sidhu (@bilawalsidhu) April 23, 2023
Google’s Zip-NeRF project, on the other hand, is still in the research stage, but recent advances have made it about 22 times faster and with 8-76% fewer errors than previous mip-NeRF models. The result is truly spectacular and a real estate agent’s dream.
UC Berkeley has combined NeRF modeling networks with language models to create a “Language Embedded Radiance Field” model (LERF) that can search for specific items in a 3D scene using natural language.
LeRF teaser, language built-in radiance field
Ultimately, the LERF team hopes to develop this technology into something that helps robots complete tasks using machine vision and AI. For example, a robot that verbalizes the task of cleaning up spilled coffee grounds enters the kitchen and uses this technology in conjunction with his GPT-style prompt-generating routine to identify various items in the room related to the task. to place it. , dustpans and brushes, to trash cans, sinks, cabinets, drawers, cleaning sprays, vacuum cleaners and more.
In another project, a team of Japanese developers are working on how to render NeRF-generated 3D scenes in real time in Unreal Engine. The team says that with the nVidia RTX3070 graphics processor he has already achieved speeds of over 60 fps with minimal memory consumption. Effectively, this is an early preview of an early feature that will flash your smartphone in a specific environment and turn it into a video game level.
NeRF Realtime Rendering in Unreal Engine
If all of the above isn’t enough to brown your noodles, check out the HOSNeRF project at the National University of Singapore. Unable to figure out how, this team managed to develop a NeRF system that can not only generate 3D models of him of people and the entire environment, but also capture and re-render dynamic actions.
So you can film someone walking around, doing things, picking up and placing things at will, and HOSNeRF will spew it out as a photorealistic 3D action scene that can be viewed from all angles.
HOSNeRF: Dynamic Human Object Scene Neural Radiance Field from a Single Video
Good grief. When this reaches the service, it will be possible to reconstruct real-world action scenes as dynamic models that can be walked through in VR, complete with moving 3D models of people and objects that can be interacted with. Uses video game physics engine. I’m going crazy.
Every corner of the AI world seems to have made some truly shocking progress in recent months. All these highly innovative technologies are beginning to converge, making it harder than ever to imagine what life will be like in five or ten years.
Source: Luma Labs, etc.
