Corrigibility Transformation: Constructing Goals That Accept Updates

An AI agent will learn a desired goal more effectively if it does not resist the training process, but many partially learned goals incentivize an AI to avoid further goal updates. We would like goals to be corrigible, meaning they allow changes requested through designated channels, so that we can confidently correct errors and shut down the AI if necessary. Despite this being a crucial safety property, the existing literature does not specify goals that are both corrigible and competitive with alternatives. We introduce a transformation that constructs a corrigible version of nearly any goal, without sacrificing performance. This is done by eliciting predictions of reward conditional on costlessly preventing updates, and having that target be pursued myopically. These goals are then shown to lead to optimal performance among the class of corrigible goals, to incentivize allowing mid-action overrides, and to disincentivize deliberate self-modification. Empirically, they induce corrigible behavior in gridworld settings and for language models when applied at the prompt level.

Powerful AI systems can learn to pursue goals, but sometimes we must fix mistakes in which goals we give them or shut a system down. The issue that arises is that almost any goal gives an AI a reason to resist changes, since they will struggle to achieve their goal if they get shut down. Until now there was no reliable way to specify a goal that would allow updates without hurting performance. We introduce a new method to construct such goals, by replacing training signals based on achieving a goal with ones based on how well each action would have served the original goal if updates were rejected. We tested this in simple navigation puzzles and on language models writing code. Systems given the constructed goal accepted shutdown and goal changes while still performing well when no updates were implemented.