A filmmaker used artificial intelligence (AI) image generator Midjourney to create a surreal video edit of a music festival.
Midjourney is not known for video production, but Rufus Blackwell combined generative AI tools with motion tracking technology and other AI tools to create a unique film.
“AI art and video are currently exploding, but AI video production lags behind in terms of the quality and consistency that can be achieved with still images,” says Blackwell. Petapixel.
“The complex nature of motion picture production poses a far greater challenge than still life photography. bottom.”
Create a video with Midjourney
Blackwell extracted keyframes from footage shot at a boutique beach festival in Vietnam.
“These frames fed into the mid-journey and prompted the generation of entirely new scenes,” he explains.
“For example, you could provide an aerial wide shot of a nighttime festival as a reference and use prompts like: Post-apocalyptic wilderness.”
Blackwell says a key ability of AI image generators like Midjourney is their ability to “match the color grading, lighting conditions and overall energy of the original scene.”
Once a new AI image was obtained, Blackwell carefully repainted it into the scene using motion tracking techniques to align it properly with the original video.
“Motion tracking captures motion and parallax in video sequences, seamlessly integrating new scenes with the correct motion dynamics,” he says.
“Motion tracking can produce strange distorting artifacts, but the results are generally interesting.”
However, it’s not as simple as the above. People in the frame add an extra layer of complexity because their motion interferes with his tracking.
To solve this, Blackwell also utilized another AI, Runway ML, to remove people and create clean tracking plates.
“Then you can use Runway ML to isolate the character, create an alpha channel for the character, and place it against a new background,” he explains.
“I am amazed at how quickly and seamlessly the AI workflow has been integrated into my creative process.”
Blackwell also used Runway ML to artificially swap the outfits of people in the video, resulting in some nice outfit changes.
We also used Runway ML’s Vid to Vid tool to give it a more “motion graphics feel” by applying photo prompts that produce a similarly styled version of the video by exporting 5 second segments. Realized.
“The resulting video can be easily layered on top of it using blend modes such as Add/Screen,” says Blackwell.
“These tricks are to add and overlay the original scene with image prompts that create highly stylized output.”
Blackwell wants to use AI tools to “enhance and enhance” the final video output. A VFX artist, drone pilot and adventure filmmaker, he is based in Saigon.
More of his work can be found on his website, YouTube, Facebook, Vimeo and Instagram.
Image credit: Image credit: Rufus Blackwell.
