FUSE: Ensembling Verifiers with Zero Labeled Data

Verification of model outputs is rapidly emerging as a key primitive for both training and real-world deployment of large language models (LLMs). In practice, this often involves using imperfect LLM judges and reward models since ground truth acquisition can be time-consuming and expensive. We introduce Fully Unsupervised Score Ensembling (FUSE), a method for improving verification quality by ensembling verifiers without access to ground truth correctness labels. The key idea behind FUSE is to control conditional dependencies between verifiers in a manner that improves the unsupervised performance of a class of spectral algorithms from the ensembling literature. Despite requiring zero ground truth labels, FUSE typically matches or improves upon semi-supervised alternatives in test-time scaling experiments with diverse sets of generator models, verifiers, and benchmarks. In particular, we validate our method on both conventional academic benchmarks such as GPQA Diamond and on frontier, unsaturated benchmarks such as Humanity's Last Exam and IMO Shortlist questions.

Verification of large language model (LLM) outputs for quality, correctness, etc. is critical for both LLM training and real-world deployment. Ideally, all verification would be done by expert humans or similar sources of ground truth, but this is often expensive and time-consuming, so one uses cheap but imperfect verifiers (e.g. another LLM) instead. Our paper introduces FUSE, a method for combining multiple imperfect verifiers to improve overall verification quality. Appealingly, our proposal requires zero input from expert humans, but often matches or improves upon the performance of baselines that assume partial access to sources of ground truth. In particular, we show that FUSE is effective in the common scenario (sometimes called "Best of N scaling") in which given multiple responses from a LLM to the same prompt, one must select a single best response to report.