Changing Answer Order Can Decrease MMLU Accuracy

Download PDF

Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, Megan Ung

Published: 16 Dec 2024, Last Modified: 20 Feb 2025airrworkshop OralandPosterEveryoneRevisionsBibTeXCC BY 4.0
Keywords: robustness, benchmarking, evaluation
TL;DR: Accuracy of models on MMLU can decrease significantly just be changing answer order
Abstract: As large language models (LLMs) have grown in prevalence, particular benchmarks have become essential for the evaluation of these models and for understanding model capabilities. Most commonly, test accuracy averaged across multiple subtasks is used to rank models on leaderboards, to determine which model is best for our purposes. In this paper, we investigate the robustness of the accuracy measurement on a widely used multiple choice question answering dataset, MMLU. When shuffling the answer label contents, we find that all explored models decrease in accuracy on MMLU, but not every model is equally sensitive. These findings suggest a possible adjustment to the standard practice of leaderboard testing, where we additionally consider the percentage of examples each model answers correctly by random chance.
Submission Number: 7