AI continuously learns new skills without the need for human direction or preset goals

Researchers are tackling the important challenge of creating artificial agents that can autonomously discover and learn an infinite number of skills. Richard Bornemann of Imperial College London, Pierluigi Vito Amadri of Sony Interactive Entertainment, Antoine Cully of Imperial College London and colleagues introduce a new framework called Continuous Open-Ended Discovery and Evolution of Skills as a Hierarchical Reward Program (CODE-SHARP), which moves beyond reliance on manually designed reward systems. This effort is particularly noteworthy. That’s because the underlying model is leveraged to dynamically build and improve a library of skills expressed as executable code, enabling the agent to solve increasingly complex and long-term goals within the Craftax environment, ultimately significantly outperforming both pre-trained agents and expert policies on average.

This work addresses a significant limitation of reinforcement learning, which typically requires manually designed reward functions for the agent, a process that is infeasible for open-ended skill discovery where the required skills are initially unknown.

CODE-SHARP leverages the power of foundation models to discover and refine hierarchical archives of skills expressed as executable reward functions written in code. CODE-SHARP employs two iterative processes driven by an underlying model. One for discovering new SHARP skills and one for refining existing skills.

New skills are proposed, implemented, and selected, while existing skills are mutated and evaluated, creating new hierarchies of increasing complexity. This approach allows the system to go beyond simply improving predefined skills to truly discovering entirely new capabilities.

The researchers demonstrated the effectiveness of CODE-SHARP within the Craftax environment and were able to discover an average of 90 diverse SHARP skills. Goal-conditioned agents are trained solely on the rewards provided by these discovered skills, learning how to solve complex, long-term goals that were previously unachievable.

Furthermore, when these skills are integrated into high-level policies by the underlying model-based planner, the agent outperforms both pre-trained agents and task-specific expert policies by more than 134% on average. This significant performance improvement highlights the potential of CODE-SHARP to create true generalist agents that can adapt to invisible environments and autonomously learn new skills.

The framework consists of a directed graph, where each node represents a SHARP skill, implemented in Python code and generated by a set of underlying models. These models act as skill suggestion generators, implementers, and judges, filtering and evaluating potential skills before environmental testing.

Skill improvement is achieved through mutation suggestions that are similarly generated by the underlying model and evaluated directly in the environment. At the same time, goal-conditioned agents are trained based on rewards from a growing skill archive to continually expand their learning capabilities and tackle increasingly complex goals. This archive is structured as a directed graph of executable reward functions written in code, called SHARP, to facilitate open-ended skill discovery.

The system employs two iterative processes: discovering new SHARP skills and refining existing skills in the archive, both guided by the underlying model. To generate new skills, CODE-SHARP utilizes a pipeline consisting of an underlying model-based skill suggestion generator, an implementer, and an adjudicator.

The suggestion generator creates candidate skills whose code syntax is evaluated by implementers before being evaluated in the environment. Judges also utilize the Foundation model to determine whether a proposed skill is acceptable for inclusion in the archive, and to filter out unsuccessful proposals and mutations.

Skill refinement involves mutating existing SHARP skills using the underlying model-based skill mutation generator and implementer, followed by environmental evaluation. Each new SHARP skill is built by composing previously added SHARP skills from the archive, resulting in a new hierarchy of increasing complexity.

The system trains a single goal-conditioned agent simultaneously and continuously expands the agent’s capabilities based solely on rewards and goals derived from an expanding skill archive. The agent was tested in the Craftax environment and demonstrated its ability to solve increasingly long-term goals. This research focused on autonomously extending and refining a hierarchical skill archive structured as a directed graph of executable reward functions coded in Python.

This work addresses the limitations of existing automated reward function design methods by enabling the discovery of entirely new skills, rather than simply refining predefined skills. The CODE-SHARP framework leverages the underlying model to iteratively discover and refine SHARP skills, synthesizing new skills from existing skills and creating new hierarchies of increasing complexity.

During the evaluation, the system generated an average of 90 diverse SHARP skills, covering a wide range of functions within the Craftax skill area. A goal-conditioned agent trained solely on the rewards from these discovered skills successfully learned how to solve complex, long-term goals that were previously unattainable using baseline techniques.

This architecture incorporates two main open-ended iterative processes. One is FM-led discovery of new SHARP skills, and the other is FM-led improvement of existing skill archives through mutation. Skill proposals are filtered through an FM-based pipeline consisting of generators, implementers, and reviewers before environmental assessment.

Skill improvement involves FM-based mutations, and suggestions are evaluated directly in the environment to ensure functionality and reward generation. The system leverages the underlying model to extend and refine a hierarchical archive of skills expressed as executable reward functions coded in Python.

This framework addresses a key challenge in reinforcement learning, which traditionally relied on manually designed reward functions, by automating this process for previously undefined tasks. High-level policy planners, also based on foundational models, effectively assemble these discovered skills into policies that can solve complex tasks, outperforming both pre-trained agents and task-specific expert systems by more than 134% on average.

The system also exhibits continuous learning capabilities, generating increasingly complex skills and integrating them into improved policies over time. The main limitation of the current implementation is that it relies on an environment defined in code, which limits its direct application to real-world scenarios like robotics.

Future research will focus on extending CODE-SHARP to environments not defined by the code through the incorporation of learned reward models and natural language feedback. Despite this limitation, the development of CODE-SHARP represents significant progress toward creating autonomous, open-ended agents capable of tackling increasingly complex goals without the need for human-defined rewards, thereby contributing to the broader pursuit of general artificial intelligence.