Large-scale language models (LLMs) have demonstrated great capabilities in generating human-like text, answering questions, and coding. However, they face hurdles that require high reliability, safety, and ethical compliance. Reinforcement learning from human feedback (RLHF), or preference-based reinforcement learning (PbRL), has emerged as a promising solution. This framework has been highly successful in fine-tuning his LLM to human preferences and increasing its usefulness.
Existing RLHF approaches like InstructGPT rely on explicit or implicit reward models such as the Bradley-Terry model. Recent studies have investigated direct preference probabilities to better represent human preferences. Some researchers have formulated RLHF as finding a Nash equilibrium in a constant sum game and have proposed Miller descent and self-play-first optimization (SPO) methods. Direct Nash Optimization (DNO) was also introduced based on win probability gap, but its actual implementation still relies on the iterative his DPO framework.
Researchers from the University of California, Los Angeles and Carnegie Mellon University have introduced Self-Play Preference Optimization (SPPO), a robust self-play framework for tuning language models to address the challenges of RLHF. We provide provable guarantees for solving two-player constant sum games and scalability for large language models. In formulating RLHF as such a game, the goal is to identify a Nash equilibrium policy that guarantees a consistently preferred response. They propose an adaptive algorithm based on multiplicative weights that employs a self-play mechanism in which the policy fine-tunes itself based on synthetic data annotated by a preferred model.
The self-play framework aims to solve two-player constant-sum games efficiently and at scale for large language models. It employs an iterative framework based on multiplicative weight updates and self-play mechanisms. The algorithm asymptotically converges to the optimal policy and identifies a Nash equilibrium. Theoretical analysis ensures convergence and provides provable guarantees. Compared with existing methods such as DPO and IPO, SPPO has better convergence and efficiently addresses the data sparsity problem.
The researchers evaluate the model using GPT-4 for automated evaluation and present results with AlpacaEval 2.0 and MT-Bench. The SPPO model consistently improves through iterations, with SPPO Iter3 showing the highest win rate. Compared with DPO and IPO, SPPO achieves better performance and effectively controls output length. Test time reranking using the PairRM reward model consistently improves model performance without over-optimizing. SPPO outperforms many state-of-the-art chatbots on AlpacaEval 2.0 and remains competitive with his GPT-4 on MT-Bench.
In conclusion, this paper presents Self-Play Preference Optimization (SPPO), a robust method for fine-tuning LLM using human/AI feedback. SPPO significantly improves over existing methods such as His DPO and His IPO across a variety of benchmarks by employing self-play in a two-player game and preference-based learning objectives. By integrating preference models and batch estimation, SPPO closely aligns LLM with human preferences and addresses issues such as “length bias” reward hacking. These findings suggest that SPPO has the potential to enhance the coordination of generative AI systems and advocate the widespread adoption of his SPPO in LLM and other fields.
Please check paper. All credit for this research goes to the researchers of this project.Don't forget to follow us twitter.Please join us telegram channel, Discord channeland linkedin groupsHmm.
If you like what we do, you'll love Newsletter..
Don't forget to join us 41,000+ ML subreddits
Asjad is an intern consultant at Marktechpost. He is pursuing a degree in mechanical engineering from the Indian Institute of Technology, Kharagpur. Asjad is a machine learning and deep learning enthusiast and is constantly researching applications of machine learning in healthcare.
