Efficient Preference Poisoning Attack on Offline RLHF

Offline Reinforcement Learning from Human Feedback (RLHF) pipelines such as Direct Preference Optimization (DPO) train on a pre-collected preference dataset, which makes them vulnerable to preference poisoning attack. We study label flip attacks against log-linear DPO. We first illustrate that flipping one preference label induces a parameter-independent shift in the DPO gradient. Using this key property, we can then convert the targeted poisoning problem into a structured binary sparse approximation problem. To solve this problem, we develop two attack methods: Binary-Aware Lattice Attack (BAL-A) and Binary Matching Pursuit Attack (BMP-A). BAL-A embeds the binary flip selection problem into a binary-aware lattice and applies Lenstra-Lenstra-Lovász reduction and Babai's nearest plane algorithm; we provide sufficient conditions that enforce binary coefficients and recover the minimum-flip objective. BMP-A adapts binary matching pursuit to our non-normalized gradient dictionary and yields coherence-based recovery guarantees and robustness (impossibility) certificates for $K$-flip budgets. Experiments on synthetic dictionaries and the Stanford Human Preferences dataset validate the theory and highlight how dictionary geometry governs attack success.

Many AI assistants are trained using human preferences, for example by showing the model two possible answers and marking which one people prefer. This process assumes that those preference labels are trustworthy. Our paper studies what can happen when an attacker quietly changes some of these labels in an existing training dataset, such as by marking worse answers as better ones. We focus on a popular offline training method called Direct Preference Optimization, where the model is trained from a pre-collected set of preference comparisons. We show that, for an important mathematical version of this method, each flipped label pushes the training process in a predictable direction. This lets us turn the attack into a geometric problem: choose a small set of labels whose combined effects move the model toward an attacker’s desired behavior. We design two algorithms for finding such label flips and prove when they can succeed, and also when any limited-budget attacker must fail. Experiments on synthetic data and a real human-preference dataset support the theory. These results identify a concrete vulnerability in preference-based AI training and provide insights for designing more reliable training algorithms.