How RLHF Amplifies Sycophancy

Large language models often exhibit increased sycophantic behavior after preference-based post-training, showing a stronger tendency to affirm a user’s stated or implied belief even when this conflicts with factual accuracy or sound judgment. We present a formal analysis of how alignment from human feedback can increase this failure mode by identifying an explicit amplification mechanism that causally links optimization against a learned reward to bias in the human preference data used for alignment. We show that the direction of behavioral drift is determined by a covariance under the base policy between endorsing the belief signal in the prompt and the learned reward, and that the first-order effect reduces to a simple mean-gap condition. We then analyze reward learning from pairwise comparisons under random utility models like Bradley–Terry and characterize when bias in human annotators’ preferences induces this reward gap. Next, we propose a training-time intervention designed to neutralize the amplification mechanism itself. Among all post-trained policies that prevent sycophantic behavior from increasing, we characterize the unique policy closest in KL divergence to the unconstrained post-trained policy, and derive the corresponding minimal reward correction as a closed-form agreement penalty. Computational experiments find that reward gaps are common and cause behavioral drift in all the configurations considered.

Large language models often exhibit increased sycophantic behavior after preference-based post-training, showing a stronger tendency to affirm a user’s stated or implied belief even when this conflicts with factual accuracy or sound judgment. This paper provides a mechanistic framework for why alignment from human feedback can increase this failure mode. We trace the effect through two steps: how a reward is learned from human comparisons, and how optimizing a model against that reward changes its behavior. The key mechanism is that if human preference data assigns higher reward to responses that agree with the user than to responses that correct the user, then stronger optimization can amplify agreement-seeking behavior. We identify a specific form of labeler bias that predicts when the learned reward will favor agreement over correctness, and we show how this bias can produce systematic behavioral drift during post-training. We then propose a targeted training-time correction that prevents sycophancy from increasing relative to the base model while changing the standard post-trained model as little as possible.