Training LLM Agents to Empower Humans

Assistive agents should not only take actions on behalf of a human, but also step out of the way and cede control when there are important decisions to be made. However, current methods for building assistive agents, whether via mimicking expert humans or via RL finetuning on an inferred reward, often encourage agents to complete tasks on their own rather than truly assisting the human achieve her objectives. We propose a new approach to tuning assistive language models based on maximizing the human's empowerment, their ability to effect desired changes in the environment. Our empowerment-maximizing method, Empower, only requires offline examples of human-written text, providing a self-supervised method for fine-tuning language models to better assist humans. To study the efficacy of our approach, we conducted an 18-person user study comparing our empowerment assistant with a strong baseline. Participants preferred our assistant 78% of the time (p=0.015), with a 31% higher acceptance rate. With this empowerment objective, we provide a framework for useful aligned AI agents without the need for human feedback or verifiable rewards.

AI assistants like ChatGPT are becoming a part of everyday life. We wondered: is there a way to make a realistic assistant that empowers the user? A person is "empowered" when they are able to have a large impact. We approached this question from the viewpoint of a mathematical framework, mutual information. We found that this complex framework can become remarkably simple when used in the real world. Our paper demonstrates that we can create a good AI assistant by training it on text that it is confident every human would write. We had real people try a coding assistant trained with our empowerment method, and found that they preferred our assistant. Our interpretation is that people like interacting with an assistant that completes all of the "easy tasks" for them, leaving the person to do the interesting work. To our knowledge, these are the first results demonstrating that an AI assistant can be trained to empower the human it works with in a real setting. We hope that our work encourages further research into more helpful and natural forms of AI assistance.