The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomy

As increasingly capable agents are deployed, a central safety challenge is how to retain meaningful human control without modifying the underlying system. We study a minimal control interface in which an agent chooses whether to act autonomously (play) or defer (ask), while a human simultaneously chooses whether to be permissive (trust) or engage in oversight (oversee), and model this interaction as a two-player Markov Game. When this game forms a Markov Potential Game, we prove an alignment guarantee: any increase in the agent's utility from acting more autonomously cannot decrease the human's value. This establishes a form of intrinsic alignment where the agent's incentive to seek autonomy is structurally coupled to the human's welfare. Practically, the framework induces a transparent control layer encouraging the agent to defer when risky and act when safe. While we use gridworld simulations to illustrate the emergence of this collaboration, our primary validation involves an agentic tool-use task where two 30B-parameter language models are fine-tuned via independent policy gradient. We demonstrate that even as the agents learn to coordinate on the fly, this framework effectively reduces safety violations in realistic, open-ended environments.

AI is becoming increasingly integrated into our everyday lives and beginning to make consequential decisions on its own. This is exactly the benefit promised by AI automation. But the issue is that AI systems can make wrong and potentially irreversible decisions such as bad commits to a code repo, mishandling money, or even take potentially catastrophic actions. A standard approach to this issue in AI safety is for AIs to ask humans for help when unsure. But the problem with this is if the AI asks too often, the human will stop giving useful responses. In this paper, we introduce the Oversight Game to try to model and tackle this problem. We frame the interaction between a human and an AI as a multi-stage game in which they must coordinate to decide when oversight should be applied. The AI chooses whether to act or to ask, and the human simultaneously chooses whether to trust or to step in. We design the rewards in this game so that cooperation is the natural outcome, and we prove that an AI seeking more independence cannot benefit itself at the expense of the human.