Connecting Low-Rank Adapters and Policy Stability in GRPO Fine-Tuning

Low-Rank Adaptation (LoRA) is widely used for parameter-efficient reinforcement learning fine-tuning of large language models (LLMs), often together with an explicit Kullback-Leibler (KL) penalty toward a reference policy. We study whether the low-rank constraint itself can restrict parameter trajectories and limit policy drift during Group Relative Policy Optimization (GRPO). In a simplified single-layer setting, we derive a rank-dependent upper bound on the KL divergence between reference and updated policies, providing a mechanistic explanation for how LoRA can constrain policy shift. Empirically, in short-horizon GRPO fine-tuning of several 1B--3B LLM families on reasoning tasks, we observe that KL-free LoRA preserves evaluation accuracy while reducing training time by avoiding reference-policy evaluations. Across LoRA ranks, policy divergence increases with rank, supporting the qualitative prediction of the analysis. These exploratory results suggest that low-rank parameterizations can contribute to policy stability in reinforcement learning fine-tuning, though broader studies across larger scales, longer horizons, and varied hyperparameters are needed.