Consistency Training Can Entrench Misalignment

Consistency training encourages a model to produce similar outputs across related inputs or sampling procedures. Such methods are simple, scalable, and largely label-free, but their effects on model alignment remain poorly understood. Could the self-bootstrapping nature of these methods amplify undesired behavior in models? We test seven consistency training methods on 108 model organisms: open-source models (7B--70B) fine-tuned to exhibit various forms of controlled misaligned behavior. We find that outcomes vary significantly: consistency training generally suppresses reward hacking and emergent misalignment but amplifies sycophancy. We present evidence that distribution shifts induced by the consistency labeling process, rather than variation in the selection operators, may be the primary driver of systematic alignment effects. Finally, we present a unifying theoretical framework to derive conditions under which consistency training will amplify or suppress misalignment. In total, our study establishes that consistency training is not alignment-neutral, and that its use in critical systems should be carefully audited.

Large language models are often trained to be more consistent ("consistency training"), in order to address problems such as models being overly eager to please or complying with harmful requests. However, making a model more consistent might also reinforce bad behaviours already present in the model. To investigate this, we make some models bad in specific, narrow ways, such as by training them to cheat on tests or produce highly offensive responses. We then use various methods to make them more consistent, and find that, depending on the misalignment and the method used to make the model more consistent, consistency training can either reinforce or amplify such bad behaviours.