LLM Surpasses Reinforcement Learning – Meet SPRING: A Revolutionary Prompt Framework for LLM Designed to Enable In-Context Thought Chain Planning and Inference

spring is an LLM-based policy that outperforms reinforcement learning algorithms in interactive environments that require multitasking planning and reasoning.

a group of researchers from Carnegie Mellon University, Nvidia, Ariel Universityand microsoft They have studied the use of large-scale language models (LLMs) for understanding and inferring human knowledge in the context of games. They propose his two-step approach called springThis involves studying academic papers and using a question and answer (QA) framework to justify the knowledge gained.

Learn more about SPRING

In the first stage, the authors read the LaTeX source code of the original paper by Hafner (2021) and extract prior knowledge. They used his LLM to extract relevant information such as game mechanics and desired behaviors described in the paper. They then utilized his QA summarization framework similar to that of Wu et al. (2023) generate his QA dialogues based on the extracted knowledge, allowing SPRING to process diverse contextual information.

The second phase focused on in-context thought-chain reasoning using LLM to solve complex games. They built a directed acyclic graph (DAG) as an inference module, where questions are nodes and dependencies between questions are represented as edges. For example, “Are the requirements for each action met?” is linked to the question, “What are your top 5 actions?” Within the DAG, establish a dependency from the latter question to the former question.

LLM answers are computed for each node/question by traversing the DAG in topological order. The final node of the DAG represents a question about the best course of action to take, and the LLM’s answer translates directly into an environmental action.

Experiment and results

Introduced by Hafner (2021), the Crafter Environment is an open-world survival game with 22 achievements organized into a 7-deep tech tree. The game is represented as a grid world with top-down observation and separate action spaces consisting of: 17 options. Observations also provide information about the player’s current inventory status, such as health points, food, water, rest levels, and inventory items.

The authors compared the SPRING method with a common RL method on the Crafter benchmark. Experiments and analyzes were then performed on various components of the architecture to examine the impact of each part on the ability to “reason” within the context of LLM.

The authors compared the performance of various RL baselines to SPRING with GPT-4 based on the environmental paper by Hafner (2021). SPRING significantly outperformed previous state-of-the-art (SOTA) methods, with his 88% relative improvement in in-game scores and his 5% improvement in rewards compared to the best RL method by Hafner et al. achieved improvement. (2023).

In particular, SPRING leverages prior knowledge gained from reading papers and requires no training steps, whereas RL methods typically require millions of training steps.

The figure above presents a plot of unlock rates for different tasks comparing SPRING and a common RL baseline. Powered by prior knowledge, SPRING is deeper in the tech tree (up to 5 depths) 10x RL techniques with achievements such as ‘craft stone pickaxe’, ‘craft stone sword’ and ‘collect iron’ More than that. Difficult to reach through random exploration.

Additionally, SPRING performs flawlessly on achievements such as “Eat Cow” and “Collect Drinks”. At the same time, model-based RL frameworks like Dreamer-V3 suffer from the challenge of reaching a moving cow through random search, resulting in a much lower unlock rate for “Eat Cow” (1/5 lower). Importantly, even though it could be easily achieved through random exploration, SPRING did not perform the ‘laying stone’ action, as Hafner’s (2021) paper did not argue that it was beneficial to the agent. .

Limitations

One of the limitations of using LLM to interact with the environment is the need for object awareness and grounding. However, this limitation does not exist in environments that provide accurate object information, such as modern games and virtual reality worlds. A pre-trained visual backbone struggles in games, but works pretty well in real-world-like environments. Recent advances in visual language models demonstrate the potential for reliable solutions in visual language understanding in the future.

Conclusion

In summary, the SPRING framework demonstrates the potential of a language model (LLM) for game understanding and reasoning. By leveraging prior knowledge from academic papers and employing in-context thought-chain reasoning, SPRING outperforms previous state-of-the-art techniques on the Crafter benchmark, achieving significant improvements in in-game scores and rewards . This result underscores the power of LLM in complex gaming tasks, and the potential for future advances in visual language models to resolve existing limitations and pave the way for reliable and generalizable solutions. suggests that there is

Please check paper.don’t forget to join 22,000+ ML SubReddit, Discord channeland email newsletterShare the latest AI research news, cool AI projects, and more. If you have any questions regarding the article above or missed something, feel free to email me. Asif@marktechpost.com

🚀 Check out 100’s of AI Tools at the AI ​​Tools Club