Can A Base LLM Verify Its Own Math? A Pre-Registered Discriminator-Generator Asymmetry Test On AIME And Four Corrective Probes
Abstract
Can a base language model tell whether a multi-step math solution is correct? If so, a training pipeline could use the model's own judgment to filter and improve its reasoning without an external teacher. We pre-register a behavioral test of this claim on Qwen2.5-1.5B base over 51 hard AIME problems where the base solves nothing. The test fails on every sub-criterion. A 17-measurement log-probability extension across five model families (1.5B-72B) initially reproduces this null, but we then uncover a confound: the standard corruption protocol preserves the visible final answer, so the test cannot detect whether the model reads the intermediate reasoning at all. We fix this with four corrective probes, evaluated across eleven open-weights configurations spanning Qwen2.5, DeepSeek-R1-Distill, Llama-3.1, Mistral, and Gemma-2. The results decompose chain verification into two distinct capacities. Perception (can the model internally distinguish correct from incorrect chains?) is universal: two probes return positive signal on every model and every problem. Verbalization (can the model express that distinction as an explicit YES/NO verdict?) is family-specific: every Qwen variant and every R1-Distill variant passes, but all three pure non-Qwen base models (Llama-3.1, Mistral, Gemma-2) fail at near-zero discrimination. Instruction tuning at 7B amplifies the verdict signal by more than 5x. A direct intervention confirms the dissociation: a 1,000-example LoRA fine-tune that injects only YES/NO supervision lifts Llama-3.1-8B base from zero to +2.67 verdict delta in 5 minutes, converting the cleanest verdict-blind model into a passing one without touching its perception. The pre-registered behavioral test still fails -- a self-teacher pipeline that needs the base to spontaneously produce correct chains on hard problems remains bottlenecked -- but the "models cannot verify math" reading is wrong at the perception level and wrong for most model families at the verbalization level. We close with design implications for math-chain verification pipelines.