Midjourney was its first authentic text to image AI tool experience in 2022. We've seen the industry evolve as competition and many other images, animations and video tools pop up almost every week. By January 2023, the tools began to evolve into places they made us sit and realize, as outlined in the first AI tools article, AI Tool Part 1: Why We Need.
However, after years of advancement and many testing, Midjourney has raised the bar again with the introduction of a new video tool. This is what I have explored so far…
Midjourney Animate
Midjourney has announced a new video animation feature, but the output is very impressive!
Over the past few years, I have been using a variety of animation and video generation tools. You may know if you are following my AI Tools series here. But this is the most seamless and fastest workflow I haven't worked on yet.
Most generators require a starting image, like a keyframe. I mostly started with images generated in Midjourney and then went to another tool to animate them. (You can see my last post, “AI Tools: Generic AI for Video & Animation Updates.” However, in Midjourney, you can either generate a new image as a source or start with your own photo.
First – Details and specifications…
Currently, everyone with an account has access to animation options, but only Pro and Mega plans can use Relax mode, and while the video consumes 8 times more time than images, it offers four variations to choose from for each round.
Video output size and format
Please note that the maximum resolution at the moment is 480p (832 x 464), and the size of course varies depending on aspect ratio.
You can export videos in compressed MP4 for social use, or export larger RAW MP4 H.264 versions (still compressed but less) and animated GIFs. To stay in the account's cloud, you can link to the URL of the completed video.
This is codec data from a “RAW” file downloaded from Midjourney.
Midjourney Video Test Drive
Of course, I had to absorb and absorb everything I could with this new feature, so I ran the pace for a few days.
First, I tried using some simple prompts to allow various news reporters to be used as B-rolls. (For example, use something like this in a pinch to put it on screen in a simulated shot of a newscast on TV). The quality is sufficient for the scale it offers (480p), but this initial rollout is not the way it intended to be fullscreen.
My first step was to get some numbers to animate. I've generated some examples by entering a description of the short prompt in Midjourney. It's weird that AI sometimes thinks about age. And some of the results are very wrong.
After selecting the required subject for each shot, I let MidJourney decide on the motion using the Auto Animate option. Each pass allows you to select four different videos, allowing you to expand many options.
I created this video to show you the selection and results of each subject.
https://www.youtube.com/watch?v=Ar4jcfnaygi
I did the same with these other examples of ChatGpt prompts and explained the process for each example.
(Note: I know that Vo is 840p and I'm well aware that it's 480p!
https://www.youtube.com/watch?v=azlidyn6f1q
Video from photo
I tested the photo-to-video feature of Midjourney and used old images from Hair Rock and Roll Day in the 80s. A lot of hair products back then!
I uploaded the photo as the first frame and let Midjourney work from there. I expanded it again to create a clip that lasts about 8 seconds. I exported it as an animated GIF (not jif) because I don't have audio. If only I was really, really cool on stage!
Use Midjourney for storyboards and pre-vis
Currently, I consider Midjourney a tool for creativity and a tool to help you bring your ideas back to life. It's not necessarily as a final product, but to understand how written words can be visualized on screen.
This could be a great tool for screenwriters looking to sell treatments, or for storyboard scenes and locations, sets, lighting, and blocking shots.
I created a completely short scene with an AI tool in just a few steps. Using two different variations, we show how Midjourney responds seamlessly to prompts and extensions.
I started with ChatGpt, which asked for ideas for a project, and this was one of the results I followed.
I was happy with the resulting image so I decided to go with the image I liked and create a story about what the character did with my expansion prompt.
This was one of the most satisfying and creative projects I've done for years with an intuitive workflow. And it only took a few hours from start to finish. Because I've never had a preconceived notion of what it would turn out. And I was trying to partner with AI tools as writers, actors, sound FX, staging and camera Ops. I really felt like a director of some sort.
Each rendering pass offers four different variations based on the first frame (or from the last pass where each extension continues up to 4x). It's very subjective to decide what you want to use, but that's part of the storytelling aspect. In my case, I started with the original ChatGpt prompt and then directed the action to the end of the prompt.
Each path I add a new or directive. In most cases, camera movement and angle were determined by Midjourney, but they can also be directed more closely. However, you may not always follow the instructions of the action, but you can fool it by rephrasing the instructions. But mistakes can actually change the story and you can chase another rabbit into the hole.
Below I have included some GIFs that show the order of the process, followed by rendering of each quick instructions, and continuing to build my scene from those rendering results.
Prompt (with selected start image): Film noir style, trench coat detective under heavy rain streetlights, black and white in subtle colour shades, sparkling cobstones, intense contrast, urban outdoor environment of the 1950s, moody and mystical, he has a bright cigarette and looks like he is waiting for someone
Quick Change/Add: He starts crossing the street while the camera follows his movements and flicks the cigarettes in the street. (He didn't cross the street, but I went with it)
Quick Change/Add: The woman appears from the shadow on the right and runs to him urgently. (It looks like a slow Santa, but it works)
Quick Change/Addition: Couple kisses and hugs.
To watch the video below, you will need to see what Variation 2 has turned out to be.
So I had to add sound to this short scene and I had to call out to narrate with the right tone and voice.
I started ChatGpt again, created an AI script writing partner and came up with some good lines. (You can hear both versions in the video below).
For more natural audio delivery, I used script text in ElevenLabs using the new V3 Alpha model.
I also used 11 enlabs to create Sound FX and Music Bed.
Everything blends easily in minutes with Adobe Premiere Pro. And this is the result
https://www.youtube.com/watch?v=2jox84mniey
For more information about Midjourney video options and instructions for using it, please visit our website.
