Ant Group’s Robbyant Announces LingBot-VA 2.0: A Causal Video Action Model Natively Built for Physical AI

Robbyant, the embodiment AI unit within Ant Group, LingBot-VA 2.0.The first Embody native base model. We describe a video action basic model for generalist robot operation. Instead of fine-tuning the video generator, the research team pre-trains the entire stack to materialize.

What is LingBot-VA 2.0?

Most video action models reuse two components built for digital content creation. One is reconstruction-oriented VAE. The other is a two-way video diffusion backbone to which the action module is connected.

This creates three limitations. Pixel reconstruction latent preserves appearance but limited physical structure. Iterative denoising on video tokens is too slow for closed-loop control. Typical video goals never tell you how your actions reshape the world.

The fourth discrepancy is structural. Backbone uses bidirectional attention, but control is strictly deployed forward in time. LingBot VA version 1.0 has fine-tuned it by stacking it on a causal model. Version 2.0 natively pre-trains causal DiT.

Version 1: Semantic Visual Action Tokenizer

Based on that motivation, stage 1 replaces the compression-only VAE. Following RepWAM, tokenizers add two purposes to reconfiguration.

Semantic alignment pulls visual potential toward the perceptual encoder teacher of friezes. Latent action goals extract compact transition variables between successive latencies. The inverse dynamics model predicts each potential action. The forward dynamics model decodes this into a transport map and residuals.

The world’s states and actions now share one potential space. Therefore, unlabeled web videos contain action-related surveillance.

Version 2: Causal DiT with sparse MoE video streams

In addition to that space, version 2 pre-trains the causal DiT. The version 1.0 Mixture-of-Transformers layout is maintained. Video experts and action experts share causal self-attention. Each owns a separate feedforward path.

The two streams scale asymmetrically. Video Expert replaces the dense FFN with a sparse MoE routing layer. This tier includes 128 routed SwiGLU experts, top 8 routes, and 1 shared expert. Load balancing follows a loss-free balancing strategy with no auxiliary losses. Action experts maintain a dense FFN with hidden dimensions 768.

The video backbone has approximately 13.0B parameters and approximately 1.9B active. Training with Action Expert and MCP heads covers approximately 15.3 billion parameters. Approximately 25 billion are activated per token during inference. For training, we use a rectified objective with a hybrid Muon and AdamW optimizer.

Source of training signal

Beyond architecture, two objectives shape what the model learns.

Multichunk prediction (MCP) corrects myopic monitoring. Teacher forcing only monitors the next chunk, so the model can cut losses by copying the appearance. MCP connects three lightweight modules that predict the next three chunks. Ablation matched the baseline’s 45k step accuracy with 20k steps and improved training speed by a factor of 2.3.

On the other hand, the five goals of T2I, T2V, TI2VA, ICL, and human-robot co-training are co-trained rather than step-by-step. Sampling follows a coarse to fine schedule, from grounding appearances to controlling video action. Keeping all your goals alive will help you avoid forgetting what was before.

hierarchical planning

Chunk-level controls do not allow you to order long-term goals. So above the policy sits a VLM planner fine-tuned by LoRA in a frozen vision tower. Outputs structured JSON (done, instruction, generation_instruction, local_scene_description). It runs at approximately 2 Hz behind an asynchronous shared buffer. The policy reads it at each chunk boundary, so planner delays do not block execution.

visionary reasoning

Even with a sparse backbone, deployments run into continuous bottlenecks. When the robot waits, the model’s latency becomes the control’s latency.

Therefore, Foresight Reasoning performs prediction and execution as an asynchronous stream. While the robot executes chunk a_t, the video expert imagines the result. Action Expert deciphers a_{t+1} from there.

There is a risk of drifting if you drive in front of it. Therefore, each returned observation is encoded into the true potential z_{t+1}, overwriting the imaginary observation. Forward Dynamics Grounding Loss trains video professionals for this role.

# Pseudocode for the asynchronous rollout (Sec. 2.3.7, Eq. 29).
# Not runnable: policy, executor and encode() are placeholders.

C = init_kv_cache(encode(obs_0))            # feedback-grounded cache C_t
a = policy.action_expert(C)                 # cold start: first action chunk a_0

while not done:
    executor.start(a)                       # execution stream, non-blocking

    C_tmp  = C + [a]                        # prediction stream: C_t u {a_t}
    z_hat  = policy.video_expert(C_tmp)     # forward dynamics -> imagined z_{t+1}
    a_next = policy.action_expert(C_tmp + [z_hat])

    obs = executor.wait()                   # real observation of a_t returns
    C   = overwrite(C_tmp, z_hat, encode(obs))   # re-ground: z_hat <- true z_{t+1}
    a   = a_next

performance

Therefore, the evaluation includes simulation and real hardware. In RoboTwin 2.0, all models are trained on 2,500 clean demonstrations and 25,000 randomized demonstrations across 50 tasks.

Distillation reduces the video sampler from 5 steps to 2 steps and the action sampler from 10 steps to 2 steps. The FP8 TensorRT engine, paged/irregular KV cache with FlashInfer attention, and host-side overhead elimination provide the rest.

# Reproduces Table 3 of the report exactly. Runnable as-is.
K = 32  # low-level control steps inside one generated chunk

stack = [("BF16 PyTorch async rollout baseline", 927),
         ("+ Consistency distillation",          466),
         ("+ Low-precision compiled execution",  369),
         ("+ Long-horizon attention optimization", 272),
         ("+ Runtime overhead reduction",        142)]

for name, ms in stack:
    print(f"{name:40s} {ms:4d} ms  {round(1000 / ms * K):4d} Hz")

print("end-to-end speedup:", round(927 / 142, 1), "x")

Version 1.0 and version 2.0

Tokenizer ablation separates row 1. Replacing the WAN2.2 VAE with a semantic tokenizer increases the performance of the 1.3B model from 78.0 to 86.6.

Usage and examples

Beyond the benchmark, four deployment shapes stand out.

• Onboarding in a few shots: The report states that the model is suitable for 10 to 15 demonstrations. In the actual evaluation, we use 20 teleoperation demos for each task. One multitasking checkpoint covers all four tasks assessed.
• Demonstration conditional control: In-context learning allows you to replace text instructions with human demonstration videos. After fine-tuning the four confirmed tasks, the policy ran the unconfirmed configuration. An example: “Place the gourd on the green plate.”
• Cheap data scaling: Human-robot joint training involves retargeting hand poses to the robot’s action space. Each hand becomes a virtual parallel gripper. The egocentric corpus spans 65.4k episodes.
• Reactive control: Demonstrations include air hockey and conveyor belts, and the policy anticipates moving objects.

Important points

• Rather than tuning the video generator, we pre-train a causal video action DiT from scratch.
• Semantic tokenizers place world states and potential actions into one ordered space.
• Sparse MoE video stream: For each token, ~2.5B of ~15.3B parameters are activated.
• Foresight Reasoning overlays prediction and execution based on all actual observations.
• Chunk latency 927ms to 142ms. Asynchronous control 35 Hz to 225 Hz.

Interactive dynamic explanation

Please check paper and Project page. Please feel free to follow us too Twitter Don’t forget to join us 150k+ML subreddit and subscribe our newsletter. hang on! Are you on telegram? You can now also participate by telegram.

Need to partner with us to promote your GitHub repository, Hug Face Page, product release, webinar, etc.? connect with us

_{Note: Thank you to the Ant Research team for providing thought leadership and resources for this article. The Ant Research team supports the promotion of this content/article.}