The Realignment Problem: When Right becomes Wrong in LLMs

Post-training alignment of large language models (LLMs) relies on large-scale human annotations guided by policy specifications that change over time. Cultural shifts, value reinterpretations, and regulatory or industrial updates make static alignment increasingly brittle. As policies evolve, deployed models can diverge from current alignment objectives, creating an Alignment–Reality Gap that is difficult to audit or correct. Existing remediation typically requires re-annotation under revised guidelines, which introduces systematic challenges, including guideline ambiguity, annotator interpretation drift, and reduced consistency at scale. We introduce TRACE (Triage and Re-align by Alignment Conflict Evaluation), a framework that transforms re-alignment into a structured optimization problem over existing data without requiring fresh human annotation. Leveraging a stronger model as a proxy judge, TRACE operates via a three-stage pipeline: (1) triaging preference pairs into inversion, suppression, or retention categories based on alignment conflicts; (2) computing an alignment impact score via bi-level optimization to prioritize high-leverage samples; and (3) executing updates using a hybrid objective that combines relational losses (e.g., IPO) for preference inversion and punitive losses (e.g., NPO) for response suppression. Experiments on Qwen2.5-7B, Gemma-2-9B, and Llama-3.1-8B demonstrate robust re-alignment on synthetic benchmarks and the PKU-SafeRLHF dataset without degrading general utility. This work provides a scalable approach for LLM realignment under evolving data annotation policies and alignment guidelines. We release our code \href{https://respailab.github.io/TRACE/}{here}.

AI chatbots are trained to follow rules about what counts as safe, appropriate, or helpful, and they learn these rules from huge sets of human-written examples. The problem is that the rules keep changing as laws, cultures, and company policies change, so a model that was trained correctly can slowly drift out of line with what we actually want now. The usual fix is to pay people to re-label all the examples by hand, but that's slow, costly, and the labels often come out inconsistent. We built TRACE, a method that updates a model to follow new rules by reusing the data it already has, with no new human labeling. TRACE looks at each existing example and decides whether it should be flipped, thrown out, or kept under the new policy. It then spends most of its effort on the examples that matter most, and adjusts the model carefully so it picks up the new values without getting worse at normal tasks. To replace human judgment, it leans on a stronger AI model as a stand-in. We tested it on three popular open models and a real safety dataset, and it enforced the new values without hurting everyday performance. The upshot is a cheaper, faster way to keep AI systems in step with rules that never stop changing.