Researchers are tackling the difficult problem of creating truly autonomous web agents that can navigate and interact with the Internet like human assistants. Hang Ding from Shanghai Jiao Tong University, Peidong Liu and Junqiao Wang from Sichuan University. introduce a new framework, DynaWeb, that utilizes model-based reinforcement learning to train these agents within a simulated web environment. This approach avoids the inefficiencies and risks associated with interacting directly with the live web and provides a scalable and cost-effective solution. Demonstrating significant performance improvements on benchmarks such as WebArena and WebVoyager, DynaWeb establishes the potential of “learning by imagination” and paves the way for more powerful and efficient online agent reinforcement learning.
DynaWeb training with expert trajectory interleaving improves robot behavior
Scientists are developing DynaWeb for efficient online reinforcement learning. The paradigm of artificial intelligence is rapidly moving toward proactive agent systems that can autonomously perform complex and long-term tasks in open-ended environments. Large-scale language models (LLMs) emerge as a powerful backbone for such agents, enabling rich inference, flexible action generation, and natural language interaction. In the web domain, LLM-based agents have demonstrated powerful capabilities to navigate real-world websites and achieve user-specified goals through multi-step interactions, enhanced by advances in prompts, structured reasoning, and action abstraction. [Yao et al., 2023, Zhou et al., 2024a, He et al., 2024].
Despite its promise, the effectiveness of online RL for web agents is fundamentally constrained by the costs and risks of interaction in real-world environments. Agents can trigger irreversible actions such as unintended purchases, account changes, and data submissions, while also facing non-deterministic page dynamics, temporary failures, and external interference. These challenges severely limit the practicality of pure online RL, making large-scale policy optimization both costly and risky in real-world Web environments. [Zhou et al., 2024a, Qi et al., 2025]. A natural direction is to replace expensive and risky real-world interactions with trained, controllable surrogates that can faithfully approximate web dynamics.
To this end, recent research has been initiated to explore web world models, i.e., trained simulators of the web environment. However, so far their role has been primarily auxiliary. Others use world models to synthesize offline trajectories for supervised fine-tuning and imitation-style training, decoupling the model-generated experience from policy optimization. [Fang et al., 2025a, Pahuja et al., 2025]. In this study, we revisit classic model-based reinforcement learning through the lens of modern web agents. DynaWeb treats the world model as a controllable synthetic web environment that can replace or extend costly real-world interactions.
By training web agents using a combination of real and imagined experiences, DynaWeb enables imaginatively scalable policy-based reinforcement learning while retaining the benefits of interactive learning. Importantly, DynaWeb combines two complementary sources of training experience. These expert trajectories are completely independent from the world model and correspond to ground truth web interactions. This simple but effective interleaving strategy preserves on-policy learning signals and enables efficient online reinforcement learning with significantly fewer interactions in real-world environments. Recent advances in web agents are primarily driven by (multimodal) large-scale language models (LLMs) that serve as the core decision-making backbone. [Dubey et al., 2024, Jia et al., ].
In addition to these models, reasoning and interaction frameworks such as ReAct Yao et al., MCP [Anthropic, 2024]and cognitive kernels [Zhang et al., 2024a] Enable structured, multi-step web actions. Web agents are typically evaluated with interactive benchmarks such as WebArena and WebVoyager. WebRL complements direct end-to-end optimization [Qi et al., 2025] We emphasize self-evolving curriculum design and outcome-based feedback to continuously generate training tasks and improve agent robustness. A typical end-to-end online RL approach is WebAgent-R1. [Wei et al., 2025]optimize multi-turn web interaction policies using outcome-based rewards and scalable trajectory sampling (e.g., multi-group GRPO). [Shao et al., 2024]). Other methods adapt the purpose of RL or combine RL with additional monitoring to better shape reasoning and planning behavior.
DynaWeb powers web agent learning through simulation
Experiments show that DynaWeb consistently and significantly improves the performance of state-of-the-art open source web agents, establishing the feasibility of imaginative training. The team measured performance with WebArena and WebVoyager benchmarks and demonstrated significant improvements in task completion rates. At the core of DynaWeb is a web world model that predicts how the state of a web page will change in response to agent actions. This model is parameterized by a large-scale language model that operates directly in observation space to generate natural-looking web page representations. The researchers broke down the task of predicting the next web state into two subtasks. One is to predict explanations for state changes and then apply those explanations to change the current state.
The world model was trained using data from the StanfordNLP/NNetNav dataset and employed a data cleaning pipeline to ensure data quality. The loss function Lφ = ∑ (I,ot,at,r,∆) −log pφ(r, ∆|I, ot, at) was used to train the model to predict both the inference trace and subsequent state changes conditional on the current accessibility tree and the performed actions. Testing proves that the learned world model acts as a reusable simulator to generate multi-step imaginary trajectories without live interaction on the web. During training, the agent policy interacts with this simulated environment, samples actions, and receives predicted observations.
The team obtained task-level completion rewards through model-based self-evaluation and assigned a scalar reward r(τ, q) ∈.[0, 1] Based on task completion. These imagined rollouts, combined with real expert trajectories from training data, are used for policy gradient optimization. This breakthrough provides a framework that can generate massive rollout action trajectories for efficient online reinforcement learning, allowing agents to effectively “dream” and learn from simulated experiences. This research establishes a promising path toward developing more robust and adaptable general-purpose AI assistants that can navigate the complexities of the web.
DynaWeb uses web simulation to outperform agents
This approach addresses the inefficiencies, costs, and risks associated with training agents on the open web, leveraging the learned “web world” to predict the presentation of web pages based on the agent’s actions. This study proves that the success of this framework is not simply due to improvements in model power or prompting techniques, but rather due to explicit training of a world model that can capture the dynamics of web interactions. The authors acknowledge that a large performance gap remains between DynaWeb and ideal performance, indicating that even powerful large-scale language model priors are insufficient as a standalone simulator for imagination-driven reinforcement learning. Future research should focus on refining world models to more accurately reflect real-world web dynamics. The findings highlight the importance of the rollout period and the regularizing effect of incorporating real expert data during training, suggesting that these are key principles for effective imagination-driven learning. This work points to world model-centric learning as a promising direction for developing more capable and efficient web agents.
