Benchmarking at the Edge of Comprehension

As frontier Large Language Models (LLMs) increasingly saturate new benchmarks shortly after they are published, benchmarking itself is at a juncture: if frontier models keep improving, it will become increasingly hard for humans to generate discriminative tasks, provide accurate ground-truth answers, or evaluate complex solutions. If benchmarking becomes infeasible, our ability to measure any progress in AI is at stake. We refer to this scenario as the post-comprehension regime. In this work, we propose Critique-Resilient Benchmarking, an adversarial framework designed to compare models even when full human understanding is infeasible. Our technique relies on the notion of critique-resilient correctness: an answer is deemed correct if no adversary has convincingly proved otherwise. Unlike standard benchmarking, humans serve as bounded verifiers and focus on localized claims, which preserves evaluation integrity beyond full comprehension of the task. Using an itemized bipartite Bradley-Terry model, we jointly rank LLMs by their ability to solve challenging tasks and to generate difficult yet solvable questions. We showcase the effectiveness of our method in the mathematical domain across eight frontier LLMs, showing that the resulting scores are stable and correlate with external capability measures. Our framework reformulates benchmarking as an adversarial generation-evaluation game in which humans serve as final adjudicators.

How do you grade an exam when the questions are too hard for the teacher to answer? This is the situation we are heading toward in AI: as systems become more capable, the tests we use to measure them get solved almost as fast as they are written, and producing new ones with verified correct answers increasingly exceeds what human experts can reasonably do. If we cannot generate or check the questions, we cannot tell whether AI is actually improving. We propose a different way to evaluate AI systems, inspired by how courtrooms work: an answer is accepted as correct not because we know it is right, but because no challenger has shown it is wrong. AI systems take turns posing questions, answering them, and pointing out specific errors in each other's answers. Humans no longer need to solve or grade entire problems: they only need to judge narrow, concrete claims like "is this particular counterexample valid?", which is a much easier task. We tested this on mathematics across eight AI systems and found that it produces stable, meaningful rankings even when the human judges are weaker than the systems being judged.