(Bloomberg): Artificial intelligence has made remarkable progress on still images. Over the past few months, services like Dall-E and Stable Diffusion have created beautiful, compelling, and sometimes unsettling images. Now, a startup called Runway AI Inc. is working on the next step: AI-generated videos.
On Monday, New York-based Runway announced the availability of a second-generation system that generates short snippets of video from a few words of a user’s prompt. Users can enter a description of what they want to see, such as “a cat walking in the rain,” and a roughly 3-second video clip of that content, or something close to it, will be generated. Alternatively, the user can upload an image as a reference point for the system as well as a prompt.
This product is not for everyone. Runway, which develops AI-based film and editing tools, has announced that its second-generation AI system will be available through a waiting list. You can sign up to access his private Discord channel, which the company plans to add more users to on a weekly basis.
This limited release represents the most high-profile example of such text-to-video generation outside the lab. Alphabet Inc.’s Google and Meta Platforms last year showcased their own efforts to turn text into video with short video clips of things like teddy bears washing dishes and yachts on a lake, but neither company has announced plans to take the research beyond the research stage.
Runway has been developing AI tools since 2018 and raised $50 million late last year. The startup helped create the original version of Stable Diffusion, a text-to-image AI model, then popularized and further developed by Stability AI.
Generative AI Takes the Next Step
In an exclusive live demo with Runway co-founder and CEO Chris Valenzuela last week, the reporter tested the second generation and promptly suggested “drone footage of a desert landscape.” Within minutes, the Gen 2 produced a slightly distorted video that was only a few seconds long, but it definitely looked like drone footage shot over a desert landscape. There is blue sky and clouds on the horizon, and the sun is rising (or setting) in the right corner of the video frame, and its light accentuates the brown dunes below.
A few other videos Runway generated from its own prompts show some of the system’s current strengths and weaknesses. Close-up images of the eyeballs are clear and look very human, but a clip of a hiker walking through the jungle shows that they may still have trouble producing realistic-looking leg and walking motions. According to Valenzuela, the model still doesn’t fully ‘understand’ how to accurately represent objects in motion.
“You can create a car chase, but in some cases the car can fly away,” he said.
With text-to-image models like DALL-E and Stable Diffusion, longer prompts might yield more detailed images, but with the second generation, simpler is better, Valenzuela said. He sees the second generation as a way to give artists, designers and filmmakers another tool to aid in the creative process, and make such tools more affordable and accessible than ever before.
The product is built on an existing AI model called Gen 1, which Runway began privately testing on Discord in February. Valenzuela said it now has thousands of users. This AI model requires the user to upload a video as an input source, and uses it (along with the user’s guidance such as text his prompts and still photos) to generate a new silent 3-second video of her. For example, if you upload a picture of a cat chasing a toy with the text “cute crochet style”, Gen 1 will generate a video of a crocheted cat chasing the toy.
Videos made with the Gen 2 AI model are also silent, but Valenzuela said the company is doing research into sound generation in hopes of eventually creating a system that can generate both images and sounds.
The debut of the second generation shows the speed and ferocity that startups are developing so-called generative AI, systems that take user input and generate new content such as text and images. Several of these systems (such as Stable Diffusion, OpenAI Inc.’s image generation Dall-E and chatbot ChatGPT) have been released to the public in recent months and have become very popular. At the same time, their prevalence raises legal and ethical concerns.
Downsides of state-of-the-art artificial intelligence
Hany Farid, an expert in digital forensics and professor at the University of California, Berkeley, has seen some videos produced by the second generation and found them “very cool,” but added that it’s only a matter of time before videos created with this kind of technology are exploited.
“People will try to do bad things with this,” Farid said.
Runway combines AI and human moderation to prevent users from generating pornography, violent content, or copyright-infringing second-generation videos, but such methods are not foolproof.
As with the rest of the AI industry, technology is advancing rapidly. Gen 2’s image quality is a bit blurry and choppy at the moment, and it’s easy to feel something is wrong with videos made with Gen 2, but Valenzuela hopes that will improve soon.
“It’s too early,” he said. “The model will improve over time.”
