Evaluating Parameter Efficient Methods for RLVR

We systematically evaluate Parameter-Efficient Fine-Tuning (PEFT) methods under the paradigm of Reinforcement Learning with Verifiable Rewards (RLVR). RLVR incentivizes language models to enhance their reasoning capabilities through verifiable feedback; however, while methods like LoRA are commonly used, the optimal PEFT architecture for RLVR remains unidentified. In this work, we conduct the first comprehensive evaluation of over 12 PEFT methodologies across the DeepSeek-R1-Distill families on mathematical reasoning benchmarks. Our empirical results challenge the default adoption of standard LoRA with three main findings. First, we demonstrate that structural variants, such as DoRA, AdaLoRA, and MiSS, consistently outperform LoRA. Second, we uncover a spectral collapse phenomenon in SVD-informed initialization strategies (e.g., PiSSA, MiLoRA), attributing their failure to a fundamental misalignment between principal-component updates and RL optimization. Furthermore, our ablations reveal that extreme parameter reduction (e.g., VeRA, Rank-1) severely bottlenecks reasoning capacity. We further conduct ablation studies and scaling experiments to validate our findings. This work provides a definitive guide for advocating for more exploration for parameter-efficient RL methods.

When we teach large language models to solve math problems, we often use a money-saving shortcut: instead of retraining the entire model, we adjust only a tiny fraction of its internal settings. Almost everyone defaults to one popular version of this shortcut (called LoRA), but no one had carefully asked whether it is actually the best choice for the newer training style that rewards models for correct reasoning. We ran the first large-scale comparison of more than a dozen of these shortcut methods. The results were surprising: the most popular method is not the best one. Certain redesigned alternatives perform better—and some even beat the far more expensive approach of retraining the whole model. We also found that a few clever-looking setup tricks cause training to collapse entirely, and that squeezing the model down to too few adjustable settings leaves it unable to learn complex reasoning. Our work gives researchers a clear guide: stop defaulting to the popular shortcut, and switch to better-suited methods to gain stronger reasoning at almost no extra cost.