Researchers continue to address the problem of efficient search in reinforcement learning, especially when dealing with environments with limited rewards. Akshay Mete, Shahid Aamir Sheikh, and Tzu-Hsiang Lin of Texas A&M University’s Department of Electrical and Computer Engineering, along with Dileep Kalathil and PR Kumar, present a new framework called Optimistic World Models (OWM) that addresses this challenge. Their work introduces a scalable method for optimistic exploration that integrates reward-biased maximum likelihood estimation principles into reinforcement learning. Unlike traditional upper confidence bound approaches, OWM embeds optimism directly into the model itself, biasing predicted results toward more beneficial scenarios, and achieves significant improvements in sample efficiency and cumulative returns when implemented within advanced world model architectures such as DreamerV3 and STORM.
The core of the innovation lies in a fully gradient-based loss function that requires neither uncertainty estimation nor constrained optimization, streamlining the training process.
This approach is designed to be easily integrated with existing world model frameworks, requiring only minimal adjustments to standard training procedures while maintaining scalability. Specifically, Optimistic DreamerV3 achieved a human normalized average score of 152.68%. This represents a 55% improvement compared to the 97.45% achieved by DreamerV3.
These advances are demonstrated across benchmarks such as Private Eye, Enduro, and Montezuma’s Revenge, as shown in Figure 1, highlighting the potential for more effective learning in complex environments. Through this research, the team also drew parallels between current world modeling frameworks and the certainty equivalence principle of adaptive control theory, and identified the closed-loop identification problem as a fundamental driver of the need for robust exploration strategies. In this study, we directly implemented RBMLE within a deep model-based reinforcement learning framework, allowing its application to large-scale problems that were previously inaccessible to this approach.
This includes enhancing the training of the standard world model with optimistic dynamics loss, which biases predicted transitions towards outcomes that yield higher rewards. Specifically, this study instantiated OWM within two established world model architectures, Optimistic DreamerV3 and Optimistic STORM.
These models are trained using the same neural network architecture as the baseline model, ensuring a fair comparison that focuses only on the effects of optimistic dynamics loss. The optimistic loss gently modifies the transition probabilities and encourages the world model to generate more favorable imagined trajectories during planning.
This approach avoids the computational complexity associated with upper confidence limit (UCB) style exploration, such as non-convex constraints and the need for explicit uncertainty estimation. Performance was evaluated across several benchmarks including Private Eye, Enduro, Montezuma’s Revenge, and Kartpole Swingup Spurse.
The experiments tracked cumulative returns over 20 to 40 million steps and demonstrated significant improvements with the OWM variant. For example, Optimistic DreamerV3 achieved an average human normalized score of 152.68%. This represents a 55% improvement compared to DreamerV3’s 97.45% on the Atari100K benchmark. This study also presents results for environments with sparse rewards, showing significant improvements in sample efficiency and overall performance compared to the standard world model.
Improving performance in low reward environments using optimistic reinforcement learning
Optimistic DreamerV3 achieved an average human normalized score of 152.68% on the Atari100K benchmark, a significant improvement over the 97.45% achieved by DreamerV3. In low-reward environments within the Atari100K, Optimistic DreamerV3 shows up to 268% improvement compared to DreamerV3 across a variety of games.
Specifically, we observed a 1735% performance improvement on Private Eye, 125% on Frostbite, and 45% on Krull. Optimistic STORM also showed remarkable performance, achieving a human normalized average score of 80.68% compared to STORM’s 75.90%. O-STORM achieved positive scores especially in freeway games, unlike STORM, DreamerV3, and O-DreamerV3, which all failed to exceed zero.
In the DeepMind Control suite, Optimistic DreamerV3 has improved performance in low reward environments such as Cartpole Swingup Sparse and Acrobot Swingup Sparse. DMC Proprio benchmark experiments reveal that O-DreamerV3 achieves a 312% improvement over DreamerV3 in Acrobot Swingup Sparse. Furthermore, in the DMC Vision benchmark, O-DreamerV3 shows a 172% improvement over Acrobot Swingup Sparse.
Ablation studies on Cartpole Swingup Sparse show that the optimistic term α requires careful tuning, with a value of 0.1 resulting in poor performance. This study also demonstrates the benefit of incorporating an entropy loss, demonstrating improved returns when compared to a model without this loss. This approach directly integrates optimistic principles from adaptive control, particularly maximum likelihood estimation of reward bias, into the model through an optimistic dynamics loss.
This framework fosters more efficient learning by facilitating imaginative transitions toward more beneficial outcomes, without the need for explicit uncertainty estimation or complex optimization steps. The adaptability of this framework allows for integration with a variety of world model designs, providing a versatile solution for improving reinforcement learning performance.
While acknowledging the limitations, the authors note that further analysis is needed to establish the convergence properties of this gradient-based approach to reward-biased maximum likelihood estimation. Future research may also focus on improving the design of the search parameters, possibly through the use of metacontrollers, to further improve performance.
