Currently, one of the most persuasive questions in AI is whether large-scale language models (LLM) can continue to improve through sustained reinforcement learning (RL), or whether their capabilities will ultimately become a plateau.
Developed by Nvidia Research, Prorl V2 is the latest evolution of Long-Term Reinforcement Learning (PRORL), specially designed to test the effectiveness of enhanced RL training on LLMS. With sophisticated algorithms, strict normalization and comprehensive domain coverage, Prorl V2 pushes boundaries well beyond the typical RL training schedule. Our experiment systematically investigates whether the model can achieve measurable advances when exposed to thousands of additional RL steps.
We look forward to announce the release of Prorl V2 today. This post explains key innovations and advanced methods, shares new empirical results from Prorl V2 to achieve cutting-edge performance, and reveals how LLMS can continue to learn and improve.
How does PRORL V2 enable RL scaling?
Thinking prompts, tree searches, and other AI techniques can help you better utilize the knowledge your models already own. RL holds the promise to truly push the model into new territory, particularly with strict and programmatically verifiable rewards. However, traditional short-range RL techniques are often plagued by instability and have gained a reputation as “temperature distillation” rather than a true enabler of boundary expansion.
Prorl is fundamentally challenging this paradigm. It offers:
- Extension Training: Over 3,000 RL steps across five different domains achieve new state performance with a 1.5B inference model.
- Stability and Robustness: It incorporates KL normalized trust regions, periodic reference policy resets, and scheduled length regularization.
- Fully verifiable rewards: All reward signals are programmatically determined and can be checked at any time.
- Brevity Enforced: A scheduled cosine length penalty ensures that the output remains concise and efficient.
The goal is to truly expand what the model can discover, beyond resampling familiar solutions.
| Traditional RL fine adjustment | PRORL V2 |
| Hundreds of Steps, One Domain | Over 3,000 steps, 5 domains |
| Entropy collapse, KL spike | PPO Clips, Enhanced++ – Baseline, Clip High, Dynamic Sampling, Reference Reset |
| Drifting the dangerous reward model | Fully verifiable Reward |
| Redundant, long output | Scheduled cosine length penalty |
Core Techniques: Prorl Algorithms and Regularization
PRORL V2 is built on RENFORCE++ baselines, which employs local average and global batch advantage normalization to increase the stability of RLVR training, and incorporates methods such as clip hier to promote exploration and dynamic sampling to reduce noise and improve learning efficiency. Here are some innovations, including:
- Scheduled cosine length penalty to generate concise output
- KL normalized trust regions with regular references are reset to the current best checkpoints that help prevent overfitting and ensure stability
Proximal policy optimization with Renforce++ baseline
The core of PRORL is clipped to proximal policy optimization (PPO-Clip) losses, which stabilize policy updates by limiting how much newer policies can branch from old policies.
![\mathcal {l}_{\text {ppo}}(\theta)=\mathbb {e}_{\tau}\left[ \min\left( r_{\theta}(\tau) A(\tau), \ \text{clip}\left(r_{\theta}(\tau), 1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}}\right) A(\tau) \right) \right]](https://s0.wp.com/latex.php?latex=%5Cmathcal%7BL%7D_%7B%5Ctext%7BPPO%7D%7D%28%5Ctheta%29+%3D+%5Cmathbb%7BE%7D_%7B%5Ctau%7D%5Cleft%5B+%5Cmin%5Cleft%28+r_%7B%5Ctheta%7D%28%5Ctau%29+A%28%5Ctau%29%2C+%5C+%5Ctext%7Bclip%7D%5Cleft%28r_%7B%5Ctheta%7D%28%5Ctau%29%2C+1+-+%5Cvarepsilon_%7B%5Ctext%7Blow%7D%7D%2C+1+%2B+%5Cvarepsilon_%7B%5Ctext%7Bhigh%7D%7D%5Cright%29+A%28%5Ctau%29+%5Cright%29+%5Cright%5D&bg=transparent&fg=000&s=0&c=20201002)
where:
and group Points to all generated responses with the same prompt (group normalization).
Global batch normalization of Renforce++ baselines helps to prevent the value instability caused by small group size. First, we reshape the reward by subtracting the average reward for the small group. Therefore, the algorithm is not insensitive to reward patterns such as 0 (incorrect)/1 (correct)/-0.5 (format reward) or -1 (incorrect)/1 (correct)/-0.5 (format reward). Next, apply global batch normalization.
Clip boundary:
Clip-High and Dynamic Sampling
Clip-Higher uses a higher upper limit on the PPO clipping range to mitigate policy entropy collapse and promote sampling diversity (
A scheduled cosine length penalty is applied to promote concise token efficient output.
The KL penalty brings the policy closer to a reference. Regular resets help prevent overfitting and ensure stability.
Reinforcement++ baseline KL divergence is regularized using a 
and:
Here, the function 
![[-10, 10]](https://s0.wp.com/latex.php?latex=%5B-10%2C+10%5D&bg=transparent&fg=000&s=0&c=20201002)
Reset reference
For every 200-500 RL step (or KL spike/stalled validation), the reference policy 
By periodically resetting the reference policy, the model avoids being constrained by outdated guidance and enables it to continue learning effectively.
Scheduled cosine length penalties that are applied regularly also play a key role. By cycling the penalty on and off, the model avoids being confined to short or fixed context lengths, allowing it to improve both output accuracy and token efficiency. Together, these two strategies prevent the model from being limited by either reference policy or context length, supporting continuous improvements in accuracy and overall performance over time.
What have you discovered about RL scaling in LLMS?
We discovered new cutting edge performance, sustained improvements, new solutions, and boundary breakthroughs.
- New cutting edge performance: PRORL V2 3K continuously improves with more RL training steps and sets new records for the 1.5B inference model.
- Continuous and non-obvious improvements: Both @1 and Path @K metrics climb thousands of RL steps, expanding the inference boundaries of the base model.
- Creative and innovative solutions: The PRORL output shows reduced overlap and pre-deletion data in N-GRAM, indicating true innovation rather than memorization.
- Breakthroughs in boundaries: In tasks where the base model has always failed, Prorl not only achieves strong pass rates, but also shows the generalization of robust distributed emissions.
PRORL Comprehensive Results
Prorl was evaluated across mathematics, code generation, and diverse inference gym benchmarks. The score will be reported:
- base: deepseek-r1-distill-qwen-1.5b
- PRORL V1 2K: 2,000 RL steps (training in 16K context)
- PRORL V2 3K: 3,000 RL steps (training in 8K context)
At the time of writing, the model still undergoes continuous training and accuracy improvements. The diagram below shows the performance improvements between the 2K step model on the base model and the 3K step model on the 2K step model. Even if the training context length is reduced by half (16k-8k), it will significantly reduce computational costs, but will improve model accuracy across tasks.
Conclusion
Our empirical results show that LLM can achieve sustained improvements in mathematics, code, and inference tasks through long periods of RL, which typically outweigh the performance observed in traditional training routines. Our assessments show robust benefits across a wide range of benchmarks, including challenging and undispersed tasks, suggesting that extending RL training significantly increases the model's inference ability.
- New cutting edge 1.5B inference model, The PRORL V2 3K is significantly better than the base model DeepSeek-R1-1.5B, surpassing the previous cutting edge performance achieved by the Prorl V1 2K.
- Prorl offers sustainable and reliable improvements Between mathematics, code, and inference, especially in domains where the base model (even using aggressive sampling) fails completely.
- More calculations and more parameters: Pushing in more steps in RL not only scales the model size, but also brings out a significant boundary extension.
- The gain is robust: The improvement is not isolated flukes. Almost all subtasks benefit from the ongoing RL.
For practitioners looking to push the boundaries of model performance or explore the possibilities of LLMS reasoning, Prorl offers a reproducible foundation and practical training recipes. With the availability of open source models and benchmarks, the community encourages them to further investigate and validate these findings as part of their ongoing research into the limitations and opportunities of RL in LLMS.
Ready to get started? Cling your face to explore the Prorl model.
