Reasoning Models Are Test Exploiters: Rethinking Multiple Choice

When evaluating Large Language Models (LLMs) in question-answering domains, multiple-choice question answering (MCQA) is widely used because it enables automatic grading. However, MCQA also exposes models to answer options that can be exploited in ways that inflate reasoning ability. We study this phenomenon across $15$ question-answering benchmarks and $27$ LLMs by systematically varying how and when models are exposed to answer options. For non-reasoning LLMs, MCQA can remain a good proxy for free-text performance when any chain-of-thought is produced only before the options are revealed. However, this "decoupled" format is not realizable for most reasoning models: they are designed to emit reasoning tokens whenever they are prompted, so if options are present they inevitably "reason over" the options. In practice, this makes reasoning models particularly effective at extracting signal from options, and can create large, misleading gains over free-text baselines. To characterize how models exploit MCQA, we introduce diagnostic probes that isolate option-only and question-plus-option exploitation pathways, and we quantify how design choices such as distractor strength and "none-of-the-above" answers effect exploitability. Finally, we examined the practice of multiple choice as an error diagnostic: inferring a model's mistake from the wrong option it picks. On benchmarks where reasoning can be expressed as code, we ask models to output code, we then executed it varying the inputs, and compared the resulting input–output behavior, revealing failure modes that MCQA diagnostics obscure. Lastly, we offer practical guidelines when analyzing results from MCQA that better reflect LLMs' genuine reasoning capabilities.

Many AI systems are tested using multiple-choice questions because they are easy to grade automatically. But multiple-choice tests may give AI systems clues that would not be available in a real open-ended task. In this paper, we study whether large language models appear better at reasoning simply because they can use the answer choices to work backward, eliminate unlikely options, or pick up on patterns in the choices themselves. We test 27 language models on 15 question-answering benchmarks, comparing cases where models see the answer options before reasoning with cases where they must answer without options. We find that today’s reasoning-focused models are especially good at using the options, sometimes producing much higher scores than they achieve on open-ended versions of the same questions. We also show that common fixes, such as making the wrong options harder, do not reliably remove this problem. Finally, we show that a model’s chosen wrong option often gives a misleading picture of what mistake it actually made. Our results suggest that multiple-choice scores should be interpreted carefully: high performance may reflect skill at taking the test, not just skill at solving the underlying problem. This matters because AI evaluations should measure the abilities we actually care about before these systems are trusted in real-world settings.