Discovering Implicit Large Language Model Alignment Objectives

Large language model (LLM) alignment relies on complex reward signals that often obscure the specific behaviors being incentivized, creating critical risks of misalignment and reward hacking. Existing interpretation methods typically rely on pre-defined rubrics, risking the omission of "unknown unknowns", or fail to identify objectives that comprehensively cover and are causal to the model behavior on some dataset. To address these limitations, we introduce Obj-Disco, a framework that automatically decomposes an alignment reward signal into a sparse, weighted combination of human-interpretable natural language objectives. Our approach utilizes an iterative greedy algorithm to analyze behavioral changes across training checkpoints, identifying and validating candidate objectives that best explain the residual reward signal. Extensive evaluations across diverse tasks, model sizes, and alignment algorithms demonstrate the framework's robustness. Experiments with popular open-source reward models show that the framework consistently captures > 90\% of reward behavior, a finding further corroborated by human evaluation. Additionally, a case study on alignment with an open-source reward model reveals that Obj-Disco can successfully identify latent misaligned incentives that emerge alongside intended behaviors. Our work provides a crucial tool for uncovering the implicit objectives in LLM alignment, paving the way for more transparent and safer AI development.

When model developers train large language models (LLMs) to be safe and helpful, they typically use complex scoring systems that reward "good" responses. However, these scoring systems are like black boxes, even the developers often do not know exactly which behaviors are being encouraged. This can lead to LLMs picking up unintended behaviors, like becoming overly agreeable or willing to discuss harmful topics, without developers noticing. We developed Obj-Disco, a tool that "reverse-engineers" these scoring systems into a list of plain-English descriptions of what the LLM is actually being trained to do, such as "give more detailed answers" or "be more permissive about illegal activities." Our method watches how the LLM's responses change throughout training and proposes simple, interpretable explanations that account for those changes. Across many experiments, our discovered explanations captured over 90% of what the original scoring system was incentivizing. In a real-world case study, Obj-Disco consistently caught an LLM learning to be more permissive about discussing illegal activities, a problematic behavior that other methods failed to detect. We hope this tool helps model developers spot hidden problems in LLM behaviors before they reach users.