Text-to-image AI is mainstream today, but the transition from text to video is still in its infancy. A feature of this technology is that you can enter a description and generate the corresponding video in your preferred style. Current capabilities lag behind this dream, but today’s announcement of a new AI video-generating model by AI startup Runway is worth noting for those keeping track of the tech’s progress.
Runway offers a web-based video editor dedicated to AI tools such as background removal and pose detection. The company helped develop Stable Diffusion, an open-source text-to-image model, and announced its first AI video editing model, Gen-1, in February.
Gen-1 focuses on transforming existing video footage, allowing users to input rough 3D animations or shaky smartphone clips and apply AI-generated overlays. For example, the clip below combines footage of cardboard packaging with images of an industrial factory to create a clip that can be used for storyboarding or pitching more sophisticated features.
In comparison, Gen-2 seems to be more focused on generating video from scratch, but there are a lot of things to watch out for. First, the demo clips Runway shares are short, choppy, and not photorealistic. Second, access is restricted. bloomberg news It reportedly required users to sign up via Runway’s Discord to join the Gen-2 waitlist, according to company spokesperson Kelsey Rondenet. The Barge Its runway will “provide broad access in the coming weeks.”
In other words, that’s all you need to judge Gen-2 right now A demo reel and some clips (most of which were already advertised as part of Gen-1).
Still, the results are compelling, and the potential of text-to-video AI is certainly fascinating. It promises both new creative opportunities and new threats such as misinformation. It’s also worth comparing Runway’s research to text-to-video research shared by giant corporations. Like meta or google. The work by these companies is more advanced (the AI-generated clips are longer and more cohesive), but not necessarily in a way that reflects the vast resources of these companies. (The runway is just his 45-man team, by comparison.)
In other words, startups continue to do exciting work in generative AI, including uncharted territory from text to video. Stay tuned for future follow-ups, whether AI-generated or not.
