Spherical Steering: Geometry-Aware Activation Rotation for Language Models

Inference-time steering offers a promising way to control language models (LMs) without retraining. However, standard approaches typically rely on activation addition, which inevitably alters the hidden-state magnitudes raising concerns about representation collapse and degraded open-ended generation. In this work, we explore Spherical Steering, a training-free primitive that resolves this trade-off through activation rotation. Rather than shifting activations with a fixed vector, our method rotates them along a geodesic toward a target direction, preserving signal integrity while steering toward the target concept. To further enhance adaptivity, we incorporate a confidence gate that dynamically modulates steering strength based on input uncertainty. Extensive experiments across multiple-choice benchmarks demonstrate that Spherical Steering significantly outperforms addition-based baselines (notably by +10\% on TruthfulQA, COPA, and Storycloze), while simultaneously maintaining the model’s general open-ended generation quality. This work highlights the value of geometric consistency, suggesting that norm-preserving rotation is a robust and effective primitive for precise inference-time control. The code is available at: https://github.com/chili-lab/Spherical-Steering.

Problem: We study a new way to control large language models without retraining them. Existing methods often steer a model by adding a fixed signal to its internal activations. This can work, but it may also disturb the model’s internal representations and hurt the quality of longer, open-ended answers. Solution: We propose Spherical Steering, which controls the model by rotating its internal activations instead of adding to them. A simple analogy is turning a compass needle toward a better direction while keeping its length the same. We also use a confidence gate so the model is only strongly steered when intervention is needed. Impact: Our method improves reasoning and behavior control across multiple benchmarks and model families. It performs better than standard existing techniques while preserving the quality of open-ended answers and adding little runtime overhead. This makes language models easier to guide at inference time without expensive retraining.