Pairwise settings for model responses have been collected widely to evaluate and provide feedback to large-scale language models (LLM). Given two alternative model responses to the same input, the human or AI annotator will choose a “better” response. Such data provides feedback signals in domains where traditional hard-coded metrics are difficult to obtain (e.g., chat interaction quality), thereby measuring model progress and fine-tuning of the model (e.g., reinforcement learning from human feedback, RLHF). However, in some domains, it can be difficult to obtain high quality such pairwise comparisons from humans or AI. For example, many (probably false) fact statements or long form responses with complex (probably false) codes represent important challenges for both AI and human annotators. In this work, we consider extending the standard AI annotator system with additional tools to improve performance in three challenging domains: long-term facts, mathematics, and code tasks. To provide high quality feedback on these domains, we propose a tool-using agent system that augments existing annotators. Our system uses web search and code execution to ground annotations based on external validation, independent of LLMS internal bias. We provide extensive experimental results to evaluate methods across three task domains and out-domain tasks based on reward bench subsets aimed at avoiding performance reductions. Share all the code and reproduce the experiment as an open source package.
- * Work done at Apple
