Make Benchmarks Live Again: A Multi-Agent Benchmark Evolution Framework via Performance Pattern Discovery

Static benchmarks lose discriminative power as large language models (LLMs) are repeatedly optimized against public test distributions, leading to saturation, contamination, and reward hacking. Simply making tasks harder is not a principled remedy, because it can shift evaluation toward capabilities outside the intended measurement scope and undermine construct validity. We instead formulate benchmark evolution as a constrained evaluation problem: preserving the domain and skill taxonomy of the seed benchmark while systematically eliciting validated performance patterns in current LLMs, including failures, biases, brittle reasoning traces, and shortcut behaviors that may remain hidden behind correct final answers. We present SEB, a multi-agent framework that enables benchmarks to evolve themselves via the collaboration of automated planning, robust reliability checks, rigorous validity control, and flexible protocol adjustments. Rather than producing uniformly more difficult tests, SEB composes diverse in-domain probes that combine discovered performance patterns with flexible formulations of the same underlying skills. We instantiate SEB on two saturated tracks, mathematical reasoning and function-level code generation, producing AIME-Evolve and Code-Evolve from AIME 2025 and HumanEval respectively. Across both tracks, the evolved releases reduce upper-tail compression and restore clearer stratification across Weak, Medium, and Premium LLMs. More broadly, SEB turns labor-intensive benchmark construction into a closed-loop, self-evolving lifecycle for recalibrating saturated benchmarks while preserving their target measurement scope.