A new model of Runway called Gen-1 can change the visual style of existing videos and movies.
Runway, the generative AI startup that co-founded last year’s breakout text-to-image model Stable Diffusion, is an AI model that can transform existing videos into new videos by applying styles specified by text prompts and reference images. has been released.
Runway is a demo reel posted on a website that shows how the company’s software, called Gen-1, transforms clips of people on the street into clay figures, and books stacked on a table into a nighttime cityscape. It shows how to do it. Runway expects Gen-1 to do for video what Stable Diffusion did for images. “Image generation models have exploded in popularity,” says Runway CEO and co-founder Cristóbal Valenzuela. “I truly believe that 2023 will be the year of video.”
Founded in 2018, Runway has been developing AI-powered video editing software for several years. Its tools are used by mainstream film and TV studios as well as TikTokers and YouTubers.manufacturer of Stephen Colbert’s Late Show We used Runway software to edit the graphics for the show.The visual effects team behind a hit movie Everything at once, wherever you are We used the company’s technology to create certain scenes.
Don’t settle for half the story.
Get access to the latest tech news without a paywall.
subscribe now
Already a Subscriber? Login
MIT Technology Review provides an intelligent, independent filter for large amounts of information about technology.
subscribe now
Already a Subscriber? Login
In 2021, Runway collaborated with researchers from the University of Munich to build the first version of Stable Diffusion. Then Stability AI, a UK-based startup, stepped in to pay for the compute costs needed to train models with more data. In 2022, Stability AI has made Stable Diffusion mainstream, transforming it from a research project into a global phenomenon.
However, the two companies are no longer working together. Getty is currently taking legal action against Stability AI, accusing it of using Getty images in its Stability Diffusion training data without permission, and Runway is keen to distance itself. be.
Runway
Gen-1 represents a new start for Runway. This follows several text-to-video models published late last year, including Meta’s Make-a-Video and Google’s Phenaki, both of which can generate very short video clips from scratch. It’s also similar to Dreamix, the generative AI Google announced last week that can create new videos from existing ones by applying a specified style. But, at least judging by the Runway demo reel, Gen-1 appears to have improved video quality. Since it converts existing footage, it can also create much longer videos than most previous models. (The company says it will post technical details about his Gen-1 on his website in the coming days. update: The paper has been published online. )
Unlike Meta and Google, Runway built its model with the customer in mind. “This is his one of the first models that was developed closely with the video maker community,” he says Valenzuela. “It contains years of insight into how filmmakers and VFX editors actually approach post-production.”
Gen-1 runs in the cloud via Runway’s website, is available to a small number of invited users starting today, and will roll out to everyone on the waitlist in the coming weeks.
Last year’s explosive momentum in generative AI was fueled by millions of people gaining powerful creative tools for the first time and sharing their creations. Valenzuela hopes that Gen-1 will soon have a similar impact on video once it’s in the hands of creative professionals.
“We’re getting pretty close to making a full feature film,” he says. “We are getting closer to where most of the content we see online is generated.”
