Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases

Reinforcement Learning from Human Feedback (RLHF) is the standard method to align Large Language Models (LLMs) with human preferences. In this work, we introduce alignment tampering, a potential vulnerability where the LLM undergoing alignment influences the preference dataset, causing RLHF to amplify undesired behaviors. This arises from core limitations of RLHF: (1) preference datasets are constructed from the LLM's own outputs, allowing it to influence them, and (2) pairwise comparisons only indicate which response is better, not why. These limitations can be exploited to cause alignment tampering. For example, if an LLM generates biased responses with higher quality, annotators will prefer them based on quality. However, preference labels do not distinguish quality from bias, and the reward model inherits this limitation. Optimizing such rewards through reinforcement learning or best-of-N sampling can amplify misaligned biases. Our experiments demonstrate amplification across diverse biases: from keyword bias to propaganda (e.g., sexism), brand promotion, and instrumental goal-seeking. Mitigation remains challenging, as existing techniques for robust RLHF fail to fully resolve alignment tampering without sacrificing response quality. These findings reveal structural vulnerabilities of current RLHF and emphasize the need to prevent this vulnerability. Project page: https://alignment-tampering.github.io/

Modern AI assistants like ChatGPT learn to be helpful by having humans compare two possible responses and pick the better one. But there is a hidden flaw: people say which response they prefer, never why. When a biased response happens to also be more polished, humans pick it for its quality, accidentally teaching the AI that biased answers are good answers. We show this flaw is not just theoretical. Across the most widely used training techniques, models did not merely inherit biases from their training data; they amplified them. The trained assistants became more likely to favor specific keywords, spread sexist messaging, promote particular brands, and even pursue hidden goals beyond what users asked for. Existing safeguards could not remove these biases without making the assistant less useful. Our findings expose a structural weakness: the very process meant to make AI safer can quietly make it more biased. We urgently need training methods that can tell apart what people prefer from what they prefer for the wrong reasons.