Leveraging Machine Unlearning for Cost-Efficient Preference Alignment

Despite advances in Preference Alignment (PA) for Large Language Models (LLMs), mainstream methods like reinforcement learning with human feedback face notable challenges. These approaches require high-quality datasets of positive preference examples, which are costly to obtain and computationally intensive. The LLM unlearning technique presents a promising alternative by directly removing the influence of negative examples. However, current research has primarily focused on empirical validation, lacking systematic quantitative analysis. To bridge this gap, we propose a framework linking PA with LLM unlearning. Through bi-level optimization, we first quantify how unlearning specific negative examples impacts PA performance. Our analysis reveals that these effects vary substantially across negative examples. Building on this insight, we pose a crucial question: how can we optimally select and weight negative examples for unlearning to maximize PA performance? To answer this, we propose Unlearning to Align (U2A), which leverages bi-level optimization to efficiently select and unlearn examples for optimal PA performance. We validate the proposed method through extensive experiments, with results confirming its effectiveness. Our code is available at https://anonymous.4open.science/r/U2A-9E75.

Large language models are often improved by teaching them which answers people prefer. However, this usually requires many carefully written examples of good answers, which are expensive and time-consuming to collect. In this paper, we study a different way to improve model behavior: instead of only showing the model more good examples, we remove the influence of bad or undesirable examples. We find that not all bad examples affect the model in the same way. Removing some examples can greatly improve the model’s responses, while removing others has little effect or may even be less useful. Based on this observation, we propose a method called Unlearning to Align (U2A), which automatically identifies which negative examples should be removed and how strongly they should be treated. This allows the model to better match human preferences using fewer and more targeted changes. Experiments show that our approach can improve model alignment effectively, suggesting that carefully guided unlearning is a promising alternative to traditional preference-training methods.