Many open source projects have developed comprehensive language models that can be trained to perform specific tasks. These models can provide useful responses to user questions and commands. Notable examples include LLaMA-based Alpaca and Vicuna, Pythia-based OpenAssistant and Dolly.
Despite new models being released every week, the community still struggles to benchmark them properly. LLM Assistant interests are often ambiguous, making it difficult to create a benchmarking system that can automatically assess the quality of responses. Human evaluation with pairwise comparisons is often required here. A scalable, incremental, and unique benchmarking system based on pairwise comparisons would be ideal.
Few current LLM benchmark systems meet all of these requirements. Classical LLM benchmarking frameworks such as HELM and lm-evaluation-harness provide multi-metric measurements for research standard tasks. However, since open-ended questions are not based on pairwise comparisons, they are not evaluated properly.
LMSYS ORG is an organization that develops open, scalable and accessible large-scale models and systems. Their new work introduces Chatbot Arena, his crowdsourced LLM benchmarking platform with anonymous and randomized combat. Similar to chess and other competitive games, Chatbot Arena uses his Elo rating system. The Elo rating system is committed to providing the aforementioned desired qualities.
They started gathering information a week before they opened the arena with many well-known open source LLMs participating. Some practical applications of LLM can be seen in crowdsourced data collection methods. Users can compare and contrast two anonymous models while chatting simultaneously in the arena.
FastChat, a multi-model service system, hosted the arena at https://arena.lmsys.org. Those entering the arena are confronted with a conversation with two unnamed models. Consumers receive comments from both models and can continue the conversation or vote on which one they prefer. After voting, the identity of the model will be revealed. The user can continue the conversation with the same two anonymous models of her, or start a new battle with two new models. The system records all user activity. Only if the model name obscures the vote in the analysis used. Nearly 7,000 legitimate anonymous votes have been tallied since the arena went live a week ago.
In the future, we hope to implement improved sampling algorithms, tournament procedures, and service systems to accommodate more diverse models and provide fine-grained ranks for different tasks.
Please check paper, code, and plan. don’t forget to join 20,000+ ML SubReddit, Discord channeland email newsletterShare the latest AI research news, cool AI projects, and more. If you have any questions regarding the article above or missed something, feel free to email us. Asif@marktechpost.com
🚀 Check out 100’s of AI Tools at the AI Tools Club
Aneesh Tickoo is a consulting intern at MarktechPost. He is currently pursuing his Bachelor of Science in Data Science and Artificial Intelligence from the Indian Institute of Technology (IIT), Bhilai. He spends most of his time working on projects aimed at harnessing the power of machine learning. His research interest is in image processing and he is passionate about building solutions around it. He loves connecting with people and collaborating on interesting projects.
