Google Deepmind’s new model, called GenCeption, uses pre-trained video generation models as the basis for classic computer vision tasks. Achieve state-of-the-art performance in depth estimation, segmentation, and 3D pose estimation with little to no training data.
Language models have become versatile processing systems as a byproduct of learning to predict the next word. This task seems to require the model to absorb grammar, world knowledge, and contextual relationships during training. This is a general description of new features in large language models.
Computer vision does not yet have an equivalent training method. Specialized models dominate the field, such as “Segment Anything” for segmentation and “Depth Anything” for depth estimation. Each uses its own architecture.
In a new paper, DeepMind researchers claim that large-scale text-to-video models can bridge this gap. Producing realistic video requires understanding the spatial geometry of the scene, object movement, and basic physics. These models also use textual descriptions during training to associate language with visual content. Data labeling is relatively cheap and can be trained on large datasets.
GenCeption repurposes video generators for visual tasks
GenCeption is built on Alibaba’s open source Wan2.1 video model. The main change is a simpler architecture. Diffusion models typically generate video from noise through many small steps. GenCeption instead generates predictions in a single forward pass, making it fast enough for real-world computer vision tasks.
Researchers use simple methods to make one model handle many tasks. GenCeption represents all output as standard 3-channel RGB images, regardless of whether the results are depth maps, surface normal maps, or segmentation masks. It also converts camera movements into image representations.
Text prompts tell your model what tasks to perform, similar to instructions given to a chatbot. The team is adding trainable modules for tasks that don’t generate images, such as 3D keypoint prediction.
Training uses one loss function across all tasks. Rather than changing the architecture for each task, researchers process training data with task-specific requirements in mind.
Match a piece of data with a specialized model
Most of the training data was obtained from a synthetic dataset containing just 7,500 videos. The team combined 800 digital human models with 200 motion sequences from a motion capture dataset. The results were then rendered in Blender with different backgrounds and camera angles. The actual video was only used for language-guided segmentation.
Our research shows that GenCeption matches or outperforms state-of-the-art results across many benchmarks, despite using one architecture for all tasks. Its depth estimates match those from specialized models such as DepthAnything 3. GenCeption outperforms NormalCrafter and Lotus-2 in surface normal estimation. It also outperforms Genmo and TRAM in 3D pose recognition. Comparable to Meta’s combination of SAM 3 and Gemini 3.5 Flash for guided segmentation with complex languages.
Models like D4RT and VGGT Omega were trained on millions of videos. GenCeption achieves similar results using 7 to 500 times less data. When tested under the same conditions, video generation pre-training also outperforms methods such as V-JEPA and VideoMAE V2. The authors believe that these results are due to the generation task rather than just the amount of data. They argue that video generation helps models learn useful representations of space and motion.
Strong generalization beyond training data, but with clear limitations
Although GenCeption was trained almost entirely on synthetic videos that show one person at a time, it also works on real-world videos with multiple people in the frame, as well as unrelated categories such as animals and humanoid robots. Research shows that some outputs contain more detailed information than the Blender renderings used for training. As a result, the cat’s whiskers and the ends of individual hair strands are preserved.
Joint training across all tasks has a negative impact on 3D keypoint estimation. The researchers suspect that the additional components required for this task may interfere with the mechanisms learned by the base model during pre-training.
Their conclusion is that the architecture of the original model should be modified as little as possible. It is also necessary to improve the processing speed. The small model takes about 6 seconds to process 81 frames of video, while the large model with 14 billion parameters takes about 10 seconds.
Video generator may contain world models that are useful even if you do not understand physics
The authors reject the idea that video generators are just entertainment tools. They argue that these models already contain a kind of universal “world model” that can support fundamental models of computer vision. Such systems can play the same role in image processing as large language models fill in text.
There is debate as to whether that framework can be maintained. An international research team recently proposed a unified definition from OpenWorldLib that explicitly excludes text-to-video models due to a lack of real-world feedback.
Former Meta chief AI scientist Yann LeCun went further, arguing that generative video models are a dead end. Meta’s V-JEPA 2 follows the alternative he has championed and predicts abstract concepts rather than pixels. The Tsinghua University benchmark points out the limitations of pixel prediction. Sora 2, Seedance 2.0, and Veo 3.1 repeatedly failed basic physics and logic tests, even when the output seemed convincing.
GenCeption uses video generation for a narrower purpose. The researchers aren’t asking Wan2.1 to predict how the world will behave. Use it to extract features for specific tasks such as depth estimation and segmentation. Learned features perform better than specialized systems. Google Deepmind is also exploring another approach with Genie 3, which builds interactive 3D environments to train AI agents.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, exclusive six-a-year “AI Radar” frontier reports, full archive access, and access to the comments section.
Subscribe now
