The Frost The first few shots nail its eerie and bewildering atmosphere. Vast mountains of ice, temporary camps of military tents, people crowding around fires, dogs barking. It’s a familiar one, but strange enough to plant a growing seed of fear. Something is wrong here.
“Give me the tail,” someone says. A close-up cut of a man nibbling on pink jerky by a fire. Grotesque. His lip movements are completely incorrect. During the beat, he looks as if he’s biting his own frozen tongue.
Welcome to the unsettling world of AI filmmaking. “We got to a point where we stopped fighting our desire for photographic accuracy and started leaning towards DALL-E weirdness,” says Stephen Parker of Detroit-based video production company Waymark. says. The Frost.
Don’t settle for half the story.
Get access to the latest tech news without a paywall.
subscribe now
Already a Subscriber? Login
The Frost is a 12-minute movie in which every shot is generated by an image-making AI. It’s one of the most striking and bizarre examples yet of this strange new genre. Check out the film below, exclusively released by MIT Technology Review.
make The Frost, Waymark supplied OpenAI’s image creation model DALL-E 2 with a script written by the company’s executive producer Josh Rubin, who directed the film. After much trial and error to get the model to generate a certain style of imagery, the filmmakers were happy to use his DALL-E 2 to generate all the shots. We then used D-ID, an AI tool that can add motion to still images, to animate these shots, making the tent flap in the wind and the lips moving.
“We built the world out of what DALL-E gave us,” says Rubin. “It’s a weird aesthetic, but we welcomed it with open arms. That’s what the movie has become.”
“It’s certainly the first generative AI movie I’ve seen that feels stylistically consistent,” says independent filmmaker and creative technology consultancy Bell & Whistle co-author. Founder Souki Mehdaoui says: “Generating still images and moving them with puppets creates a collaged and joyful atmosphere.”
The Frost Joining a series of short films made using various generative AI tools released over the past few months. Even the best generating video models can only generate a few seconds of video. Thus, current cinematic work exhibits a wide range of styles and techniques, ranging from storyboard-like sequences of still images to: The Frostmashing up various multi-second video clips.
Runway, a company that develops AI tools for video production, hosted an AI film festival in New York in February and March.Highlights include otherworldly PLSTC By Laen Sanches. A dizzying sequence of bizarre plastic-wrapped marine creatures produced by the imaging model Midjourney.dream-like given again By Jake Oleson. This uses his technology called NeRF (Neural Radiation Field) which transforms 2D photos into his 3D virtual objects.and surreal nostalgia an extended childhood This piece by Sam Lawton is a slideshow of Lawton’s old family photos that expands the DALL-E 2 beyond its borders, allowing you to play around with the details you half remember from your old photos.
Lorton recorded in the film how he showed the images to his father and responded: I don’t know what it is. Do I just not remember? ”
fast and cheap
Artists are often the first to experiment with new technologies. But the immediate future of generative video is being shaped by the advertising industry. waymark created The Frost To explore how generative AI can be incorporated into our own products. The company creates video creation tools for businesses looking for a quick and cheap way to create commercials. Waymark, along with companies like Softcube and Vedia AI, is one of a number of startups that offer bespoke video ads to clients in just a few clicks.
Announced earlier this year, Waymark’s current technology combines several different AI techniques, including large-scale language models, image recognition, and speech synthesis, to generate video ads on the fly. Waymark also utilized a large data set of non-AI commercials made for previous customers. “We have hundreds of thousands of videos,” says CEO Alex Persky-Stern. “We extracted the best of them and trained them on what a good video looks like.”
Waymark’s tool is offered as part of a tiered subscription service starting at $25/month, and when users enter their business web address or social media account, the tool launches and reveals everything it finds. collect text and images of That data is then used to generate commercials, using her GPT-3 from OpenAI to create scripts that are read out by a synthetic voice over selected images that highlight the business. Generate sophisticated 1-minute commercials in seconds. The user can edit the result if desired, such as tweaking the script, editing the image, or choosing a different voice. To date, Waymark says he has used the tool on more than 100,000 people.
The problem, Parker said, is that not all businesses have websites and images. “Accountants and therapists may have no assets at all,” he says.
Waymark’s next idea is to use generative AI to create images and videos for businesses that don’t already have them or don’t want to use what they have. “That’s the driving force behind the production The Frost‘ says Parker. “Creating worlds and atmospheres.”
The Frost There is certainly an atmosphere. But also junkie. “It’s never a perfect medium,” Rubin says. “We had a little trouble getting certain things out of the DALL-E, such as the emotional response of the face. Magic is happening right before our eyes,’ I thought.”
This hit and miss process will improve as technology improves. DALL-E 2 made by Waymark The Frost, was released exactly one year ago. It’s only been a few months since we’ve had a video generation tool that produces short clips.
The most innovative aspect of this technology is the ability to generate new shots whenever you need them. “It takes 15 minutes of trial and error to get the shots you want that fit the sequence perfectly,” says Rubin. He remembers cutting the film and wanting certain shots, such as a close-up of a boot on a mountainside. With DALL-E, all you have to do is make a call. “It’s amazing,” he says. “That’s when my truly eye-opening experience as a filmmaker began.”
Chris Boyle, co-founder of London-based short-form video startup Private Island, also recalled his first impressions of the image production model last year: change everything. ”
Boyle and his team have created commercials for various global brands such as Bud Light, Nike, Uber and Terry’s Chocolate, as well as short in-game videos for blockbuster titles such as Call of Duty.
Private Island has been using AI tools in post-production for several years, and has stepped up during the pandemic. “During lockdown, we were very busy and we weren’t able to shoot as much as we used to, so we started leaning more towards machine learning at the time,” says Boyle.
The company uses NeRF to create 3D scenes from 2D images and uses machine learning to rip motion capture data from existing footage instead of collecting it from scratch. We have employed a variety of technologies to facilitate
But generative AI is the new frontier. A few months ago, Private Island posted a beer parody commercial on its Instagram account, created using Runway’s Gen-2 video creation model and Stability AI’s Stable Diffusion image creation model. It became a slow-burning viral hit.called synthetic summer, the video shows a quintessential backyard party scene of carefree youngsters lounging in the sun and sipping drinks. Except that many of these people have gaping holes instead of mouths, and when they drink beer, the can sinks into their heads and sets their backyards on fire. It’s a horror show.
“You see it at first, but it’s just very generic, half-baked Americana stuff,” Boyle says. “But the hindbrain or something is thinking, ‘Oh, they all have their faces upside down.'”
“We like to use the medium itself to tell stories,” he says. “And I think ‘Synthetic Summer’ is a great example because the medium itself is so spooky. This is a visualization of some of our fears about AI. ”
take advantage of strengths
Is this the beginning of a new era in filmmaking? The palette of current tools is limited. The Frost and “Synthetic Summer” both draw on the strengths of the technology that created them. The Frost “Synthetic Summer” fits well with the DALL-E 2’s spooky aesthetic. Since video generation tools like Gen-2 only produce a few seconds of video at a time, “Synthetic Summer” has a lot of quick cuts that must then be spliced together. Boyle says it works well in a party scene where everything is chaotic. For Private Island, he also considered making a martial arts film where the quick cuts suited the subject matter.
This could mean the use of generative video in music videos and commercials. But more is not clear. Apart from experimental artists and a few brands, Medaoui says not many people are using it yet.
A constant state of flux is also uncomfortable for potential customers. “I’ve spoken to a number of companies that seem interested but are reluctant to commit resources to the project because technology changes so fast,” she says. Many companies are also wary of ongoing lawsuits over the use of copyrighted images in datasets used to train models such as stable diffusion, she said. .
No one knows where this will go, Medaoui said. “A lot of assumptions are thrown like arrows right now, but none of the nuanced considerations behind them are given.”
Meanwhile, filmmakers continue to experiment with these new tools. Inspired by the work of his friend Jake Olson, Medawi uses generative AI tools to produce short documentaries to destigmatize opioid use disorders.
Waymark is planning a sequel frost, “It’s not sold on the DALL-E 2. I think it’s more like, ‘Look at this space,'” says Persky-Stern. “The next time we do it, it will probably be using new technology to see what we can do.”
Private Island has experimented with other films as well. Earlier this year, I made a video using scripts created by ChatGPT and images created by Stable Diffusion. We are currently working on a hybrid film in which costumes designed by Stable Diffusion are worn by live-action performers.
“We’re very interested in aesthetics,” said Boyle, adding that it’s a shift from the dominant imagery of digital culture reduced to emojis and glitch effects. “It’s really exciting to see where new aesthetics come from. Generative AI is like a broken mirror for us.”
