highlight
Google DeepMind is developing video-to-audio (V2A) technology.
The technology combines video pixels with natural language text prompts to create a “rich soundscape for the on-screen action.”
V2A can generate an unlimited number of soundtracks for any video input.
Video generation models are advancing rapidly, but many current systems only generate silent videos. A major next step will be creating soundtracks for these silent movies. Google DeepMind is developing video-to-audio (V2A) technology to enable synchronized audio-visual generation, bringing generated movies to life.
Let's take a look at the details.
Also read: Google Deepmind's new AI agent 'SIMA' will be your in-game teammate: Find out more
The company says its V2A technology combines video pixels with natural language text prompts to create a “rich soundscape for the on-screen action.”
This technology can be combined with generative video models such as Veo to create shots with dramatic scores, realistic sound effects and dialogue that matches the character and tone of the video.
It can also generate soundtracks for a variety of traditional footage, including archival material and silent films, opening up a wider range of creative opportunities.
It's worth noting that V2A can generate an unlimited number of soundtracks for any video input.
Also read: Google Deepmind's new AlphaFold 3 AI can model proteins, DNA, RNA: Read more
How it works?
“To find the most scalable AI architecture, we experimented with autoregressive and diffusion approaches. A diffusion-based approach for speech generation delivered the most realistic and compelling results for synchronizing video and audio information,” the company said.
The V2A system begins by encoding the video input into a compressed representation. Then, a diffusion model iteratively refines the audio from random noise according to the visual input and natural language prompts. This process produces synchronized, lifelike audio that closely matches the prompts. Finally, the audio output is decoded, converted into an audio waveform, and combined with the video data.
To generate higher quality audio and add the ability to guide the model to produce specific sounds, the company added more information to the training process, including AI-generated annotations that contain detailed descriptions of sounds and transcripts of conversations.
