Generating a video using AI typically requires typing prompts and starting over if the AI is incorrect. But software giant Adobe has built AI that can adjust the video during production, allowing users to change the camera angle or move objects within the scene.
Adobe MotionStream is an experimental AI video creator that allows users to control elements in a scene in real time. This experimental technology allows users to change the camera angle and direct movement within a scene using drag-and-drop style tools and sliders.
Announced on April 10, MotionStream generates video in segments to allow users to interact with it in near real-time.
According to Adobe, all of this happens during video generation. MotionStream starts with a text prompt, but then provides “instant visual feedback,” Adobe says, allowing users to move around the camera at different angles or move objects in the scene and watch the results adjust as those changes occur.
According to Adobe, MotionStream is also designed to recreate more natural body movements, which has historically been a challenge for AI.
“This is where a lot of the magic happens, the second-order effects that are very difficult to control manually,” said Eli Shechtman, senior principal scientist at Adobe and MotionStream researcher. “For example, if you want to move an elephant, you can move it by clicking on its body, but it’s a pain to make that movement look natural manually.”
“This currently requires skills and specialized software to create the rig, create the animation, and keyframe the animation. This process typically takes hours, if not days, depending on the scope,” Schechtman added. “Instead, the underlying video generator behind MotionStream essentially simulates the world in real time, so the elephant’s legs move naturally and the ears flap naturally as the elephant moves. The model provides knowledge about the world and can interact with it.”
While AI-powered video generation typically requires some latency, MotionStream actually started out as an Adobe Research effort to speed up the process by breaking it down into smaller pieces. The user will see the first part while the latter part is generated in the background.
Richard Zhang, senior research scientist at Adobe and co-investigator on MotionStream, said, “Once we started segmenting videos, the natural next step was to seek feedback from our users while generating the videos. That led us to MotionStream, which is the result of a long study.”
After overcoming the lag, the researchers moved on to providing users with tools for adjustment, allowing them to interact with the tools and fine-tune the results while the video was being generated.
MotionStream is only an experimental program for now, but Adobe is currently offering a preview of the program to the public.
you may also like
See the best video editing apps.
