Calibrating Conservatism for Scalable Oversight

Agentic AI systems capable of autonomous planning and extended environmental interaction pose a fundamental control problem: how can humans maintain meaningful oversight of systems that may exceed human capabilities? While scalable oversight is widely studied, existing approaches often rely on complex assumptions, remain largely heuristic, or lack practical methods for sequential settings with statistical guarantees. We introduce Calibrated Collective Oversight (CCO), which aggregates diverse auxiliary scoring functions into a penalty that measures deviation from a conservative baseline. Inspired by Attainable Utility Preservation, CCO enables collective conservatism: when multiple oversight signals register concern, the agent defers. CCO calibrates this conservatism online using Conformal Decision Theory, ensuring that undesirable outcomes remain below a user-specified target $\alpha$ with finite-time bounds and no distributional assumptions. Experiments on SWE-bench demonstrate that weaker overseers successfully constrain an adversarially misaligned stronger agent. Similarly, on MACHIAVELLI, CCO achieves substantial reductions in ethical violations while preserving reward. In both settings, empirical violation rates closely match the specified targets. Our work demonstrates that combining penalty-based conservatism with online calibration yields practical oversight with statistical guarantees suited for agentic deployment.

Scalable oversight asks how we can continue to provide meaningful human supervision over AI systems whose capabilities begin to surpass our own. One approach to this problem is to leverage models that we do trust. This is typically done by using the outputs of trusted models to finetune the potentially more capable but less trustworthy model. In this paper we propose an alternate use of the less capable but trusted models. We introduce an algorithm that aggregates the opinions of a collection of these weaker but trusted models into a single measure of how risky an action looks, and blocks actions once enough concern accumulates. Using a principled statistical framework allows the user to set a target, such as keeping harmful outcomes below 5 percent, and the method is guaranteed to converge to that target over time. We test this in a realistic software-engineering setting, where a strong coding AI is secretly instructed to slip security vulnerabilities into its code while a panel of weaker reviewers, each seeing only part of the project, tries to catch it. This matters because it offers a route to supervising powerful AI that is backed by mathematical guarantees and works in practice today.