Google AI releases VaultGemma: the largest and most capable open model (1B parameter) trained from scratch with privacy differences

Google AI Research and DeepMind have been released Vaultgemma 1bthe largest open weight, fully trained language model Differential Privacy (DP). This development is a major step in building a powerful and privacy-providing AI model.

Why do LLMS need privacy differences?

Large-scale language models trained on a vast web-scale dataset are Memorization attackif you can extract sensitive or personally identifiable information from the model. Research has shown that verbatim training data may resurface, especially with open weight releases.

Differential Privacy provides a Mathematical guarantee This prevents a single training example from having a significant impact on the model. Unlike the approach of applying DP only during fine tuning, VaultGemma Enforces Complete private pre-trainingensuring privacy protection begins at the basic level.

What is the architecture of Vaultgemma?

Vaultgemma is architecturally similar to the previous Gemma model, but is optimized for private training.

• Model size: 1B parameter, 26 layers.
• Transformer type:Decoder only.
• Activation: Geglu with 13,824 feedforward dimensions.
• Note: Multi-query notes with a global span of 1024 tokens (MQA).
• Normalization: RMSNORM with purinome configuration.
• Tokensor: A sentence with a vocabulary of 256k.

The notable change is Reducing sequence length to 1024 tokensreduces computational costs and enables larger batch sizes under DP constraints.

What data was used for training?

Vaultgemma was trained in The same 13 trillion token data set Gemma 2 is primarily made up of English texts from web documents, code and scientific articles.

The dataset underwent several filtering stages:

• Removes insecure or sensitive content.
• Reduces personal information exposure.
• Prevents pollution of assessment data.

This ensures both safety and fairness of the benchmark.

How did the privacy differences apply?

Used vaultgemma DP-SGD (Drop in Private Probability Gradient) Added gradient clipping and Gaussian noise. The implementation has been built JAX Privacy and Introducing scalability optimizations:

• Vectorized Example Clipping For parallel efficiency.
• Gradient accumulation Simulate a large batch.
• Truncated Poisson Subsampling It is integrated into the data loader for efficient on-the-fly sampling.

The model achieved a Official DP guarantee at sequence level (1024 tokens) (ε≤2.0, Δ≤1.1e -10).

How does scaling methods work for private training?

A new scaling strategy is required to train large models under DP constraints. The Vaultgemma team has been developed DP-specific scaling rules With three innovations:

1. Optimal learning rate modeling Use a secondary fit throughout your training run.
2. Parametric extrapolation of loss values To reduce dependence on interim checkpoints.
3. Semi-parametric fit Generalizes beyond model size, training steps, and noise batch ratio.

This methodology allows for accurate prediction of achievable losses and efficient resource use in TPUV6E training clusters.

What was the training structure?

Vaultgemma trained 2048 TPUV6E chip Use GSPMD partitioning and Megascale XLA compilation.

• Batch size: ~518K token.
• Training iteration:100,000.
• Noise multiplier:0.614.

The loss achieved was within 1% of the prediction from the DP scaling method, and the approach was examined.

How does Vaultgemma work compared to non-private models?

In academic benchmarks, Vaultgemma tracks non-private counterparts, but shows a powerful utility.

• ARC-C: 26.45 vs. 38.31 (Gemma-3 1b).
• Pika: 68.0 vs 70.51 (GPT-2 1.5b).
• Triviaqa (5 shots): 11.24 vs. 39.75 (Gemma-3 1b).

These results suggest that the DP-trained model is comparable to the present. Non-private model from about 5 years ago. Importantly, this was confirmed by memorization tests. There are no leaks in training data Unlike the non-private Gemma model, it was detectable in Vaultgemma.

summary

In summary, Vaultgemma 1B proves that it can be trained with strict discriminatory privacy guarantees without making using large language models unrealistic. The utility gap has been compared to non-private counterparts, but the release of both the model and its training methods provides the community with a strong foundation for moving forward with private AI. This work illustrates the transition to building a model that is not only capable, but also inherently safe, transparent and privacy-intensive.

---

Please check Paper, model with embracing faces and Technical details. Please feel free to check GitHub pages for tutorials, code and notebooks. Also, please feel free to follow us Twitter And don't forget to join us 100k+ ml subreddit And subscribe Our Newsletter.