Steering as Implicit Low-Rank Finetuning: A theoretical Framework for Activation Steering in Transformers

Activation steering and supervised finetuning are common ways to change language model behavior, yet their mechanistic connection remains unclear. We identify when low-rank finetuning produces steering-like residual shifts and when it does not. We study this question in a controlled linear-attention setting by comparing rank-one LoRA updates to the residual-stream geometry assumed by fixed-direction steering. We separate the value and output pathway, which writes information into the residual stream, from the query and key pathway, which routes attention across context. We prove that rank-one updates to value or output matrices force all residual-stream shifts to lie in a single fixed direction, independent of the data distribution. In contrast, rank-one query and key updates induce context-dependent shift directions, so their resemblance to fixed-direction steering depends on the input distribution. Empirically, value and output updates produce exactly rank-one activation shifts, while query and key updates are less rank-concentrated and require larger weight changes for comparable behavioral effects. When all attention matrices are trainable, component replay shows that query and key routing contributes meaningfully and interacts non-additively with value and output writing.