Thanks to a new AI tool from Google DeepMind, content creators may no longer suffer from a shortage of B-roll.
Google's AI lab showed off its video-to-audio conversion technology, or V2A for short. Similar to generative video, V2A uses AI to create audio that matches what's playing in the video. In the first few demos, the new AI tech can deliver convincing audio, like steady footsteps or precise drum strokes that match the timing of the video.
The idea would, in theory, solve a gap in generative video as we know it: OpenAI's Sora, Luma AI's more recent Dream Machine, and Google's own Veo can generate impressive visuals, but still lack audio.
Unlocking Audio with AI
V2A is still in development, but the first few samples published by Google DeepMind look very promising. As seen in the demo, V2A can add everything from dramatic background music to realistic sound effects. Google DeepMind says V2A can even generate soundtracks for older video samples, such as archival footage or silent films.
Amazingly, Google DeepMind says that V2A can “generate an unlimited number of soundtracks for any video input.” This means you can try loads of audio samples until you find the perfect soundtrack for your video. What's more, you can tweak the initial prompt by telling V2A if it sounds cooler or warmer than you were hoping for.
However, there are drawbacks. According to Google DeepMind, V2A works by understanding the raw pixels of the source video, which means that if the video has artifacts or distortions, the generated audio may suffer from quality issues. V2A also struggles to sync up any provided transcripts, etc. with what people are saying in the video. As you can see below, the video and audio don't match, shattering the illusion of this AI-generated clip.
Not ready for release yet
While these demos are impressive, Google DeepMind says the technology isn't ready for public release yet. Before any official release, the AI lab said it would conduct “rigorous safety evaluations and testing,” but didn't specify the scope of those tests.
Given the pace at which generative AI is evolving, it's understandable that Google DeepMind is being cautious in releasing a new, powerful tool like V2A: We've already seen bad actors exploiting the safeguards put in place in generative AI tools, and while V2A could introduce new problems, it could also be a new AI game-changer for content creators.
