Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

Low-Rank Adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning (SFT) paradigm. However, their efficacy and behavior under Reinforcement Learning with Verifiable Rewards (RLVR) are less well understood. In particular, two structurally initialized LoRA variants, PiSSA and MiLoRA, which outperform standard LoRA under SFT, can underperform standard LoRA under RLVR and may even exhibit training instability. These observations suggest that how to initialize the low-rank matrices in RLVR remains unclear. In this work, we develop a theoretical analysis of LoRA in RLVR, showing that orthonormal initialization achieves the minimal gap between LoRA’s outcome and that of full fine-tuning. Guided by this insight, we propose geometry-preserving orthonormal initialization for low-rank adaptation in RLVR, leading to two new variants, LoRA-RLPO and LoRA-RLMO. Experiments on mathematical reasoning benchmarks show that our orthonormal initialization stabilizes RLVR training and outperforms standard LoRA, contrasting with PiSSA and MiLoRA. Finally, our unified analysis also explains why PiSSA and MiLoRA can underperform in RLVR, which may be of independent interest.

Large language models can improve their reasoning skills by learning from rewards, such as whether a math answer is correct. To make this training cheaper, researchers often update only a small set of extra parameters instead of the whole model. However, we find that some common ways of initializing these small updates can make reward-based training unstable, even if they work well in supervised fine-tuning. We propose two new initialization methods that preserve useful structure from the original model while avoiding overly aggressive updates. In math reasoning experiments, our methods make training more stable and improve performance over standard low-rank adaptation.