From Parameter Dynamics to Risk Scoring: Quantifying Sample-Level Safety Degradation in LLM Fine-tuning

Safety alignment of Large Language Models (LLMs) is extremely fragile, fine-tuning on small number of benign samples can erase safety behaviors learned from millions of preference examples. Existing studies attempt to explain this phenomenon by comparing parameters and hidden states before and after fine-tuning, but overlook their dynamic evolution during fine-tuning. In this work, we analyze parameter dynamics and uncover a critical mechanism underlying safety degradation, where benign fine-tuning causes parameters cumulatively drift toward danger-aligned directions, progressively undermining the model's safety. Inspired by these findings, we propose Sample-Level Quantification of Safety Degradation (SQSD), a method that quantifies each training sample's influence on safety degradation. Specifically, SQSD assigns continuous risk scores to individual samples by measuring their induced parameter updates along safety and danger directions. Extensive experiments across three models and two datasets show that SQSD outperforms baselines in better separating high-risk and low-risk samples, with risk scores that consistently predict the severity of safety degradation. In particular, SQSD exhibits strong transferability across architectures, parameter scales, and parameter-efficient methods.

Safety-aligned AI models can have their safety protections silently undermined when fine-tuned on seemingly harmless data, and this represents a subtle but serious risk as AI systems are increasingly customized for downstream applications. We reveal the mechanism behind this phenomenon: fine-tuning causes a model's parameters to gradually drift toward danger-aligned directions in parameter space, which progressively erodes safety behaviors even when no harmful training data is involved. Crucially, different training examples contribute unequally to this drift. Building on this insight, we propose SQSD, a method that assigns each training example a continuous risk score by measuring how much its induced parameter update points toward dangerous rather than safe directions. This enables fine-grained, corpus-wide risk assessment rather than coarse subset filtering. Experiments across multiple models and datasets show that SQSD accurately predicts sample-level fine-tuning risks and generalizes well across model architectures, parameter scales, and fine-tuning methods, offering a practical tool for safer model adaptation.