Turn your doodles into video clips in minutes using Runway’s new AI video tools – here’s how

Follow ZDNET: Add us as your preferred source Google.

---

Important points of ZDNET

• A new feature in Runway lets you generate videos from your doodles.
• Although impressive, it performs best when handled with care.
• To access it, you must (at least) pay for a Standard Runway account.

---

A family of settlers admiring a herd of bison begins to scurry as two demonic winged creatures fly overhead. The girl passes through the fence post like a ghost.

This surreal 5-second scene was created thanks to a new feature within the AI ​​platform Runway called Motion Sketch. As the name suggests, it generates videos from simple doodles that can be created directly on top of still images.

For the clip below, I first generated the image using Google’s Nano Banana Pro (one of the text-to-image models offered through Runway’s platform), then drew some bird-like shapes and simply drew two arrows to indicate the direction the family and bison were fleeing. Added some text prompts for additional guidance. The model will do the rest.

This feature still requires some ingenuity, in much the same way that ChatGPT requires multiple carefully crafted prompts to produce the output you’re looking for. Either way, this could be a big step forward for AI-generated videos and for creative industry professionals who prefer to think visually rather than verbally.

“This feature is useful for all types of users,” Aditi Poduval, Runway product manager, told ZDNET. “Ultimately, we are helping people perform precise movements without having to create specific written instructions.”

Here’s what I learned from trying Motion Sketch:

my experiment

After first playing around with the image of a sleep-deprived settler family, I decided to up the sketchiness, so to speak.

I used text prompts to generate images of my family. However, Nano Banana Pro via Runway also allows you to create images based solely on doodles. This can be created directly in a platform like Microsoft Paint. I’m a writer, not a visual artist, but I tried my best to sketch what I vaguely wanted, like a snake hanging from a tree branch.

So I turned to motion sketching. As mentioned above, I completed the model midway through my settler family experiment by adding some text prompts in addition to the doodles, but this time I wanted to see how well the model worked without written guidance. I wanted to generate a clip of a snake sliding to the right, so I drew some simple arrows along the length of its body to direct its movement. Any 6-year-old will understand what these arrows mean. But how does the AI ​​work? Set the length to 10 seconds, select Runway’s Gen-4.5 video model (Google’s Veo 3 and Veo 3.1 are also available),[生成]I clicked.

I was hoping for a relatively smooth clip of the snake moving along the branch and emerging from the bottom right of the shot, but what I got was much trickier. The snake left part of its body to the left of the shot, and another snake came out of its body and fell out of the tree. At one point, it appeared to grow crocodile-like legs and was using them to pull itself along branches. The original arrow I drew also briefly appeared at the beginning of the clip and then disappeared.

To be fair, snakes are biological wonders, and modeling their movements accurately is no easy task for any animator, human or machine. And the model apparently misinterpreted my multiple arrows as telling the snake to move in multiple directions at the same time, clearly confusing it. So I went back and added a simple text prompt with an arrow to the original image. “The snake slithers along the branches.”

This time the final product was a little closer to what I had envisioned (the snake’s body didn’t suddenly duplicate or grow legs). The movement still looked awkward, but then again, snake movement is quite a challenge and involves a lot of complex and detailed physics.

One more experiment: I wanted to see how well the model would perform in a real photo, so I uploaded one from my laptop. This photo was taken a few months ago in Prospect Bark, Brooklyn.

I effectively drew the universal icon of fire, a red wavy line radiating upwards, studded with orange. Again, I don’t claim to be a talented illustrator, but most people will be able to recognize what I was trying to do with this. The models on the runway noticed this too, and dropped a giant bonfire in the center of the peaceful lakeside scene in my photo. Again, my sketch appeared in the clip for a moment, then disappeared in a psychedelic movement of undulating lines that gradually became shorter. Maybe it was just my imagination, but the trees next to the bonfire seemed to be turning black from the heat.

Take-out

Like any generative AI tool, Motion Sketch has imperfections and glitches (a girl passing through a wooden beam, a snake growing legs, and squiggles from my original sketch briefly appearing in the final video clip). Still, I’m impressed.

Here you can get a glimpse of an entirely new format for generating videos using AI. This could lower the barrier between imagination and creation than ever before. “This is what I’ve been waiting for,” one X user posted below Tuesday’s runway announcement. “Would you like to draw a vision and watch it rendered as a video? An absolute game-changer for creators who can’t inspire engineers for the rest of their lives.”

For now, it still requires quite a bit of trial and error if you have a particular vision in mind, but it’s still an exciting preview of what’s to come. And it’s definitely worth a try, perhaps especially if you’re just working in the spirit of creativity without a clear vision in mind of what you actually want to create.

How to access

Motion Sketch requires (at least) a Standard subscription to Runway. This costs $12 per user per month and comes with 625 monthly credits.

To access it from your user dashboard, in the left menu, click[アプリ]Click[Explore Gen-4.5]below the collection or the preview window on the right half of the screen.[モーション スケッチ]Select. There are[今すぐ試す]A button labeled .

From there, simply upload your image and click Sketch → Export Sketch → Generate.