short video It gives the impression of a flipbook, swaying from one surreal frame to the next. These are the result of Internet meme creators playing around with the first widely available text-to-video AI generators, and they’re the result of Dwayne “The Rock” Johnson eating rocks and nasty things like the President of France. I am envisioning a possible scenario. Emmanuel Macron Sift through the trash, chew, and distort the mundane like Paris Hilton taking a selfie.
This new wave of AI-generated videos has a distinct echo of the Dall-E that took the internet by storm last summer, performing the same trick with still images. Less than a year later, these shaky Dall-E images are almost indistinguishable from reality, begging two questions. Will AI-generated video advance this fast?
ModelScope, a video generator hosted by AI company Hugging Face, allows people to type just a few words and receive amazingly shaky videos. His AI company, Runway, which co-developed the image generator Stable Diffusion, announced the text-to-video generator in late March, but hasn’t made it widely available to the public. Both Google and Meta have announced that they are working on text-to-video technology in the fall of 2022.
Now it’s either a jarring celebrity video or a teddy bear drawing a self-portrait. But in the future, AI’s role in cinema could evolve beyond viral memes, with technology helping film casting, modeling scenes before shooting, and even swapping actors in and out of scenes. It may become possible. The technology is advancing rapidly, and it will likely be many years before such a generator can produce an entire short film based on a prompt. Still, the potential for AI in entertainment is immense.
“Just as Netflix has disrupted how and where content is viewed, I think AI will create even greater disruption to the actual creation of that content itself,” says Sinead Bovell, futurist and founder of tech education firm WAYE. says.
But that doesn’t mean AI will completely replace writers, directors, and actors anytime soon. And some considerable technical hurdles remain. The AI model still can’t maintain perfect frame-to-frame consistency, which is necessary for smooth visuals, so the video looks choppy. It takes more computer power and more data to create compelling, grotesque, and consistent content that lasts longer than a few seconds. In other words, a large investment in technology development is required. “You can’t easily scale up these image models,” says Bharath Hariharan, a professor of computer science at Cornell University.
But even as rudimentary as it seems, progress on these generators is progressing “really, really fast,” says Jiasen Lu, a research scientist at the Allen Institute for Artificial Intelligence.
The speed of progress is the result of new developments that have enhanced the generator. Like an image generator, ModelScope is trained with text and image data, and is also fed a video showing the model in action. should do it Look, says Apolinário Passos, machine learning art engineer at Hugging Face. It’s a tactic used in the meta. The burden of annotating videos and labeling them with text descriptors was removed, simplifying the process and allowing rapid advancement of the technology.
