A new preprint from Stanford’s Human-Centered AI Lab, now available on arXiv, challenges one of the quieter orthodoxies in modern machine learning: that when you fine-tune a pre-trained policy with online reinforcement learning, you absolutely must carry over a pre-trained Q-function alongside it. The paper argues that in many common settings, this assumption is not just wrong — the pre-trained Q-function is actively making things worse.
The finding lands in a corner of ML that has grown steadily more consequential: the offline-to-online RL pipeline is now the backbone of robot learning systems and, in a structural sense, of large language model alignment through reinforcement learning from human feedback. If the standard design choice for that pipeline is flawed in the ways the Stanford team describes, the implications extend well beyond any single benchmark.
The paper, authored by Perry Dong, Ron Polonsky, Dorsa Sadigh, and Chelsea Finn — the same Finn who co-authored the widely adopted Cal-QL method — is titled “Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?” and answers its own question with a careful, hedged, and technically grounded “often, no.”
Cal-QL and the Consensus It Built
To understand what is being questioned, it helps to trace how the consensus formed. Offline reinforcement learning lets a model train on a static dataset — demonstrations, logged interactions, human preference data — without ever touching a live environment. The attractive idea is to then hand that trained model to an online RL phase, where it can improve through actual interaction. Getting this transition right has been a long-standing engineering problem.
Cal-QL, published at NeurIPS 2023 by Mitsuhiko Nakamoto, Chelsea Finn, and collaborators at UC Berkeley and Stanford, addressed this by engineering a specific kind of Q-function initialization. The core insight was calibration: rather than letting the offline critic underestimate values wildly (the default behavior of Conservative Q-Learning, or CQL, from which Cal-QL was derived), the algorithm ensured that Q-values were at a reasonable scale — a lower bound on true value, but not an absurdly pessimistic one. Cal-QL outperformed the state of the art on 9 of 11 fine-tuning benchmark tasks. The result was widely adopted in robotics, and the broader design principle — carry over the Q-function — became implicit scaffolding for most subsequent offline-to-online algorithms.
Pessimism as the Hidden Bug
The mechanism the Stanford team identifies is not a subtle numerical artifact. Offline RL is structurally required to be conservative about state-action pairs it has never seen. That conservatism is implemented through explicit penalties in the Q-function loss — the CQL term that pushes down estimated values for actions the offline policy would not have taken. During offline training, this is the right behavior: without it, the algorithm would confidently assign high value to untested actions and the policy would degrade.
But online fine-tuning is precisely the phase during which the agent begins visiting state-action pairs outside the offline distribution. Those are the novel pairs that might yield higher reward than anything in the offline dataset — the whole point of continuing to train online. The same pessimism that kept the offline critic honest now systematically undervalues exactly the transitions the agent needs to explore. The result is a suppression of the exploratory behavior that online RL depends on to outperform the offline initialization.
This is not merely the Stanford team’s conjecture. A parallel line of work from Westlake University, published on arXiv in May 2025, reached similar conclusions through different experiments: online fine-tuning initialized with an online-trained Q-function achieved faster improvement and higher convergence scores than initialization with an offline-pretrained one, and this held even when the online-trained Q-function started from a poor initial state. An earlier perspective paper on Q-value estimation in offline-to-online RL had already documented the specific failure mode: that estimation bias and inaccurate value rankings inherited from offline pretraining produce a misleading signal for policy updates, making standard offline RL algorithms like CQL and TD3-BC unreliable as online fine-tuning foundations.
The Randomly Initialized Critic’s Unlikely Rehabilitation
If the pre-trained Q-function is the problem, the Stanford team’s proposed alternative sounds almost naive: start online training with a randomly initialized critic. Prior work had shown this leads to instability — a random critic gives the policy actor poor learning signals early in training, causing a well-performing pre-trained policy to “forget” what it already knew before the critic catches up. That documented failure was a major reason the field defaulted to Q-function pretraining in the first place.
The Stanford paper argues this failure is not an inherent property of random initialization. It is an artifact of how online training is typically configured in these comparisons. When the fine-tuning setup is adjusted appropriately — using higher update-to-data (UTD) ratios and ensemble methods to stabilize the early critic — the randomly initialized critic recovers quickly and escapes the pessimism trap that continues to plague its offline-pretrained counterpart. The empirical contribution of the paper is demonstrating that the setup adjustment, not the Q-function initialization, is the load-bearing variable.
This has a meaningful practical implication. A large class of policies is pre-trained through imitation learning or behavioral cloning — approaches that produce a policy without ever training a Q-function. Previously, getting those policies online-fine-tuned with actor-critic RL required either: (a) running an additional offline RL pretraining pass solely to produce a critic, or (b) using online RL methods that are less sample-efficient than actor-critic approaches. The Stanford paper’s findings suggest a third path is viable: initialize the actor from the imitation-learned policy, initialize the critic randomly, and configure the online training correctly.
Conditions That Matter
The paper is measured in its claims. Chelsea Finn and her co-authors identify specific conditions under which Q-function pretraining helps vs. hurts — the finding is not that pretrained critics are always counterproductive.
When offline data is broad and diverse, covering a wide range of state-action pairs, the pessimism-induced bias is smaller. The critic has seen enough of the space that its underestimation of novel pairs is mild, and the calibration Cal-QL introduced may be sufficient to prevent major distortion. In those settings, Q-function pretraining remains defensible.
When offline data is narrow or expert-dominated — a small set of near-optimal demonstrations, for example — the critic’s coverage gap is severe. The novel state-action pairs the online agent encounters are far outside the distribution of what the offline critic knows, and its pessimistic undervaluation of those pairs is correspondingly large. This is precisely the setting in which imitation learning policies are most common. It is also the setting in which the costs of carrying over an offline critic are highest.
The paper additionally identifies environment horizon and distribution shift severity as modulating variables. A high-stakes deployment environment with long episodes and significant sim-to-real gap is more likely to expose the brittleness of offline Q-function pretraining than a narrow, well-covered domain.
What This Means for LLM Alignment
The paper explicitly connects its findings to reinforcement learning from human feedback, the alignment technique used to train large language models including GPT-4, Claude, and Gemini. In RLHF, a reward model is trained on human preference data — a fixed, static dataset of pairwise comparisons — and then used to guide the language model’s policy during online fine-tuning.
The structural analogy to offline RL is direct: the reward model plays the role of the offline-trained critic, and the language model plays the role of the policy. The reward model was trained on a specific distribution of human preference data, and like an offline Q-function, it assigns value only to responses within that distribution’s scope. Responses that are high-quality but outside the training distribution — creative, technically precise, or culturally specific in ways the preference data did not capture — may receive systematically lower reward scores. The mechanism is the same pessimism the Stanford team describes for robotics critics.
The draft paper notes that even methods specifically designed for fine-tuning, such as Cal-QL, still encounter unlearning problems in settings with limited or no offline data, and that this problem becomes more severe when there is no data retention between training phases. Whether the specific remedies the Stanford team proposes for robotics critics — higher UTD ratios, ensembles — translate to LLM alignment reward models is not established by this paper; the connection is the team’s framing and an open research question, not a demonstrated result.
A Productive Contradiction Within One Lab
One editorial dimension of this paper that the research community is likely to notice: Chelsea Finn, who co-designed Cal-QL in 2023, is now a co-author of a paper that formally interrogates whether Cal-QL’s central design assumption was warranted. This is not a retraction — the paper does not claim Cal-QL is wrong, only that the necessity of Q-function pretraining has been assumed rather than demonstrated across many common settings. But the same research group that developed one of the most widely cited offline-to-online RL methods is now the group arguing that a simpler baseline might often be preferable.
This kind of intra-lab revisionism is a productive feature of well-functioning research communities, not a contradiction. It is also worth noting that Finn’s IRIS Lab at Stanford has been unusually prolific on this specific topic in 2026 — Perry Dong has also been involved in a separate paper on sample-efficient RL fine-tuning for vision-language-action models (EXPO-FT, arXiv, May 2026), and the group has papers on transformer-based Q-function scaling and value-guided diffusion policy optimization. The new preprint is a skeptical counterweight within a sustained research agenda, not an isolated critique.
What Practitioners Should Do
The paper is a preprint and has not yet undergone peer review. Its empirical claims are backed by experiments on standard benchmarks including D4RL tasks (AntMaze, Adroit, and Kitchen) and the RLbench manipulation suite, but the conditions under which the results generalize remain to be established by independent replication.
For practitioners currently running offline-to-online RL pipelines, three concrete takeaways emerge from the preprint:
The assumed necessity of Q-function pretraining is worth testing, not assuming. If you are carrying over a pretrained critic because Cal-QL did it and prior work required it, running an ablation with a randomly initialized critic and a higher UTD ratio is a low-cost experiment that may surface a faster-converging alternative.
Narrow or expert-dominated offline datasets are the highest-risk setting. The pessimism bias is proportional to the coverage gap between the offline data and the online distribution. The narrower your offline data, the more it is worth questioning whether its Q-function estimate is doing you any good at fine-tuning time.
Imitation-learned policies are newly tractable starting points for actor-critic fine-tuning. If your policy was trained via behavioral cloning and you previously avoided actor-critic online RL because it required a critic you did not have, the Stanford team’s results suggest that the randomly initialized critic, set up correctly, may be viable.
Frequently Asked Questions
What is offline-to-online reinforcement learning, and why does the critic matter?
Offline-to-online RL is a two-phase training paradigm: a policy is first trained on a static dataset without live environment interaction (the offline phase), then continues learning through direct interaction with the environment (the online phase). The “critic” — technically the Q-function — estimates the value of taking a given action in a given state. It is the signal that tells the policy which actions are worth taking more of. In offline RL, the critic is deliberately trained to be conservative about unseen state-action pairs to prevent the policy from chasing unreliable estimates. The problem the Stanford paper identifies is that this conservatism persists into online training, where the whole point is to explore new state-action pairs — and the pessimistic critic undervalues exactly the novel actions the agent most needs to try.
Does this mean Cal-QL is wrong or that systems using it should stop?
Not straightforwardly. The Stanford paper argues that the necessity of Q-function pretraining has been assumed rather than rigorously demonstrated across common settings — not that Cal-QL produces bad results in the settings where it was validated. In high-data, broad-coverage offline settings, Cal-QL’s calibration approach may still be the better choice. The paper is a call for principled evaluation of when Q-function pretraining helps and when it does not, not a blanket recommendation to remove pretrained critics from all pipelines. Independent replication and peer review will determine how broadly the findings hold.
How does this connect to the reward models used in LLM alignment?
The structural parallel is direct but the specific translation is an open question. In reinforcement learning from human feedback, the reward model is trained on a fixed dataset of human preference comparisons — analogous to an offline RL critic. The reward model assigns value to language model outputs based on what it saw during training. Responses that are genuinely high-quality but outside the distribution of the training preference data may receive systematically lower scores, the same mechanism Stanford identifies as pessimism bias in robotics critics. Whether the remedies — different training setup, random re-initialization of parts of the reward model — would help in LLM fine-tuning contexts is not established by this paper, but the mechanistic similarity makes it a live research question.
What would it take for practitioners to safely act on these findings?
At minimum: independent replication on benchmark suites beyond those reported in the paper, and peer review confirming the theoretical analysis. For practitioners willing to experiment before full replication, the lowest-risk action is adding a randomly-initialized-critic ablation to existing comparisons — an experiment that costs one additional run and directly tests whether the pessimism-avoidance tradeoff applies in their specific setup. High-UTD ratio training and critic ensembles, which stabilize early online learning without a pretrained critic, are already well-documented techniques; the contribution here is the argument that they are sufficient replacements for Q-function pretraining in many cases.
