In-Training Defenses Against Emergent Misalignment in Language Models

Fine‑tuning lets practitioners repurpose aligned large language models (LLMs) for new domains, yet recent work reveals emergent misalignment (EM): Even a small, domain‑specific fine‑tune can induce harmful behaviors far outside the target domain. Even in the case where model weights are hidden behind a fine-tuning API, this gives attackers inadvertent access to a broadly misaligned model in a way that can be hard to detect from the fine-tuning data alone. We present the first systematic study of \emph{in‑training} safeguards against EM that are practical for providers who expose fine‑tuning via an API: We evaluate whether they a) prevent broad misalignment, b) allow narrow misalignment, c) learn well on benign tasks, and d) remain coherent. We investigate five training regularization interventions: (i) KL‑divergence regularization toward a safe reference model, (ii) $\ell_2$ distance in feature space, (iii) preventive steering with an evil persona vector, (iv) interleaving training examples from a general instruct-tuning dataset and (v) inoculation prompting. We demonstrate that selecting interleaving data by the perplexity gap between aligned and misaligned models yields the best results overall.

Training models to act badly in a very narrow way, for example to write code that's easily hacked or to prefer unpopular music genres, can sometimes make them act completely evil — praising dictators or telling the user that AIs should rule the world. This surprising effect is called "emergent misalignment". We compare different tweaks to training to see if they prevent it and don't hurt training effectiveness too much. It turns out that sprinkling in a bit of carefully chosen extra data is the best. We're still not sure why emergent misalignment happens, so we hope this helps people not accidentally create evil AIs by training on safe-looking data.