On the Convergence of Low-Precision LoRA Training

Low-rank adaptation (LoRA) has become a standard approach for parameter-efficient fine-tuning, substantially reducing the number of trainable parameters and the optimizer memory required during training. Quantization offers a further route to resource-efficient fine-tuning, but many existing quantized LoRA methods focus on weight-only compression, leaving unclear how low precision affects optimization when it is used throughout training. In this work, we initiate a systematic theoretical study of low-precision LoRA training in which the data, pretrained weights, adapters, gradients, and intermediate matrix multiplications are quantized during optimization. For spectrally initialized LoRA under a low-rank linear model with sub-Gaussian data and relative quantization error, we establish finite-horizon convergence upper bounds that consist of a full-precision linear convergence term plus a quantization-dependent additive error floor. Our results show that low-precision LoRA can retain convergence behavior close to its full-precision counterpart, provided that the mantissa length grows only logarithmically with the inverse of the full-precision contraction rate.