What Shapes Emergent Misalignment?

Emergent misalignment (EM) is a phenomenon where models (mis)generalize in ways that surprise humans: an instruct model further finetuned on a narrow domain (e.g., insecure code) can induce broad misalignment in unrelated domains. We study EM through three elements that fundamentally shape post-training: training dynamics, the pre-trained model, and the training data. (1) We first examined how loss on in-domain training data relates to out-of-domain broad misalignment across datasets and model families. We then tested 29 learning schedules with cosine and CyclicLR on a single model + domain combination, but did not find instances where similar or lower training loss and better broad alignment scores co-occur. (2) Furthermore, we compare broad alignment score of the narrowly finetuned model with those of the pre-trained model and found signals that the medians of the misaligned model score distributions are usually statistically different from the pre-trained model distribution, with some positive signals on overall correlation. (3) Finally, we extracted activations for training and eval prompts (excluding responses) from the instruct model before narrow finetuning and found that the degrees of train-eval prompt activation overlap on the train principal components could explain finegrained alignment scores after narrow finetuning.