CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features

Sparse Autoencoders (SAEs) decompose LLM activations into interpretable features, yet existing SAE-based steering methods require contrastive datasets or large activation stores. We introduce CorrSteer, which selects steering features by correlating task outcomes with SAE activations computed during generation, then validates these selections through intervention. This two-stage approach treats correlation as a selection heuristic and intervention as the causal test: features that both correlate with success and improve performance when amplified are retained. Coefficients derive from mean activations on correct samples, yielding a fully automated pipeline without task-specific tuning. On Gemma-2 2B and LLaMA-3.1 8B, CorrSteer achieves +3.3% on MMLU (4k samples) and +27.1% on HarmBench (108 samples), with lower side-effect ratios than fine-tuning despite comparable accuracy. Selected features cluster into interpretable categories: structured-output features for multiple-choice tasks, refusal features for safety, and domain-specific semantics for specialized benchmarks. The method scales to $10^5$ SAE features (16K per layer × 26 layers for Gemma-2 2B; 32K × 32 for LLaMA-3.1 8B) via streaming correlation ($O(1)$ in dataset size), requiring no backward passes or activation storage.

Problem. Adjusting a large language model toward a specific behavior, such as better accuracy, refusing harmful requests, or reducing bias, is usually done by fine-tuning, which acts like editing a genome with a shotgun: it hits the target but quietly damages unrelated abilities. Lighter alternatives that adjust the model's internal signals exist, but they need paired good/bad examples or huge memory, and they pick what to amplify from the input prompt rather than the actual output behavior. Solution. We introduce CorrSteer, an automated method that watches which interpretable building blocks light up while the model produces correct answers, then amplifies those blocks across every layer in real time as it writes. We pick blocks by correlating their activity with task success, then verify the link by amplifying each block and checking whether performance improves. We also introduce the Side Effect Ratio, a simple measure of how many unrelated answers change per improved answer. Impact. CorrSteer matches fine-tuning accuracy on knowledge tasks while halving the side-effect cost, works on any task that can be scored correct or incorrect, and the chosen blocks are mostly semantic concepts that humans can read, offering a continuous safety dial that can be tuned or turned off without retraining.