Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders

Preference learning in large language models relies on reward models as proxies for human judgment. However, these models frequently exhibit preference instability, producing contradictory preference assignments in response to subtle, meaning-preserving input variations. We analyze this instability at the representation level under three semantic-preserving perturbation types: paraphrasing, pattern injection, and backdoor triggers. We hypothesize that preference instability stems from over-reliance on features that are predictive yet brittle under input variation, which we term unstable features, and employ Sparse Autoencoders (SAEs) to disentangle them in a sparse latent space. We found that benign and perturbed inputs activate distinctly separable feature patterns, a structure nearly invisible in raw hidden states, which enables accurate detection of unstable representations. Building on this separability, we propose two SAE-based mitigation strategies: SAE Feature Steering, which identifies and suppresses anomalously activated features at inference, and SAE Residual Correction, which learns adaptive adjustments over SAE features to restore correct preferences. Our methods substantially reduce incorrect preference assignments on harmlessness and hallucination tasks while preserving benign performance and generalizing to tasks beyond the calibration distribution, without retraining the reward model.