Mixture of Graders: Adaptive Strategy Routing for LLM-Based Mathematical Evaluation

LLM judges are promising for evaluating reasoned mathematical solutions, yet their scores remain prompt-sensitive and unstable. We evaluate 13 grading strategies on Putnam-AXIOM-Grading, a new 500-problem benchmark with 1,000 mathematically annotated candidate solutions, and IMO-GradingBench. The strategies span direct scoring, structured chain-of-thought, and multi-call pipelines such as G-Eval, Chain-of-Verification, and Debate. No fixed strategy dominates, and the strongest single default, comparative prompting, wins only on average. In hindsight, the best strategy per problem differs from the best fixed default in 76% of cases, revealing substantial oracle headroom for adaptive selection. Motivated by this, we train Mixture of Graders, a ModernBERT-large strategy router, using the fixed-strategy score matrix and human labels, then deploy it using only problem and reference features. On Putnam-AXIOM-Grading, MoG improves over the best fixed strategy across all open-weight judge models, with relative MSE reductions up to 27.8% across both Qwen2.5-72B-Math-Instruct and Mistral-Large-Instruct-2411 student responses. It also generalizes across students, with a router trained on Qwen responses outperforming the best fixed default on Mistral responses. The prescription for difficult mathematical grading is not a single best prompt, but a learned mixture over grader strategies. Our code is available at https://anonymous.4open.science/r/mixture-of-graders-8327/README.md.