Filmmaker Josh Gladstone recently started working on Light Field Video. Incredible technology has enabled him to capture volumetric video with a single camera rig and produce video content viewable in virtual reality (VR) or on his Looking Glass volumetric display.
This isn’t the first time Gladstone has ventured into novel video technology and volume content. Almost a year before him, he published a multiplane video (volumetric video) and a video on how to use machine learning to improve your workflow.
In the video above, Gladstone explains groundbreaking research from Google. Although this study produced incredible volumetric video results, it required a large facility with many cameras and an absurd amount of computational time at 28.5 CPU hours per video frame. bottom.
Gladstone has developed a method that reduces the demands on equipment and computation while producing videos of incredible volume. The new project scales down equipment demands to his five GoPro Hero8 Black cameras in frames that are primarily made up of 3D printed materials.
Gladstone said the particular camera model doesn’t matter. “The software is camera agnostic, so there is nothing special about GoPro cameras other than being portable. But sharpness is a factor, so I’m definitely interested in testing it with other cameras and lenses,” he explains.
Gladstone’s software is a custom pipeline he created based on an open source project. The software “acquires an image from each camera, calculates the pose of the camera, and uses AI to render a layered depth image (LDI). This LDI consists of 8 layers of RGB+Alpha+Depth. This is similar to the Multiplane Images implementation: it uses per-layer depth information to stretch each layer in the Z axis, the Multiplane implementation used a flat image with 32 layers. “This layered depth implementation is eight layers, so it’s more efficient,” he explains. Petapixel.
The LDI is “arranged in a grid, with a color image at the top and a combination of depth and alpha at the bottom,” adds Gladstone. The video below shows the rendering process.
Petapixel Gladstone asked how important AI is to the rendering process and if it can be done manually.
“I don’t think it’s something you can do by hand, or at least I wouldn’t want to. The good thing is that if the camera pose is properly resolved, it works automatically. My 3090 graphics Scard takes about 15 seconds per frame,” he explains.
Since this process requires forward-facing input, there is no benefit in capturing input footage from additional perspectives, such as lateral. However, while there is no reason to record data from additional angles at the same time, there are many benefits to be gained from adding more cameras to your rig. Gladstone said they are currently dialing in the optimal number of cameras and doing testing to determine the optimal distance between each camera.
Creating volumetric videos is more computationally intensive as it involves many frames, but nothing is inherently more difficult than volumetric videos and photographs.
“Because each frame is rendered independently, the video footage does not complicate the neural network. Mr. Stone says.
Capturing and rendering volumetric video is one thing, but playing it is quite another. Although you can get a stereoscopic video feel on a flat screen, Gladstone’s projects are best viewed with a virtual reality (VR) headset or Looking Glass.
Gladstone said the playback was delivered using Unity and the final file is an .MP4. Unity projects use custom shaders to decode volumetric video layers and project them into 3D space.
Josh Gladstone does a great job with many tech video segments, including Light Field Video. You can see his work on his website, YouTube and Instagram.
Image credit: Josh Gladstone
