Improving Automated VQA Evaluation Using Large-Scale Language Models
Authors: Oscar Manhas, Benno Crozier, Aishwarya Agrawal
Abstract: Eight years after the visual question answering (VQA) task was proposed, accuracy remains the primary metric for automatic evaluation. VQA accuracy has been effective so far in IID evaluation settings. However, our community is moving towards open-ended generative models and OOD evaluation. In this new paradigm, existing VQA accuracy metrics are too strict and therefore underestimate the performance of VQA systems. Therefore, there is a need to develop more robust automatic VQA metrics that serve as proxies for human judgment. In this work, we propose to leverage the in-context learning capabilities of instruction-tuned large-scale language models (LLMs) to build better VQA metrics. We formulate VQA evaluation as an answer evaluation task and instruct LLMs to score the accuracy of candidate answers given a set of reference answers. We demonstrate that the proposed metrics correlate better with human judgment compared to existing metrics across several VQA models and benchmarks. We hope that our metrics, if widely adopted, will enable a more accurate assessment of research progress in VQA tasks. We plan to make the evaluation code and collected human judgments publicly available.
