AutoQRA: Joint Optimization of Mixed-Precision Quantization and Low-rank Adapters for Efficient LLM Fine-Tuning

Quantization followed by parameter-efficient fine-tuning has emerged as a promising paradigm for downstream adaptation under tight GPU memory constraints. However, this sequential pipeline fails to leverage the intricate interaction between quantization bit-width and LoRA rank. Specifically, a carefully optimized quantization allocation with low quantization error does not always translate to strong fine-tuning performance, and different bit-width and rank configurations can lead to significantly varying outcomes under the same memory budget. To address this limitation, we propose AutoQRA, a joint optimization framework that simultaneously optimizes the bit-width and LoRA rank configuration for each layer during the mixed quantized fine-tuning process. To tackle the challenges posed by the large discrete search space and the high evaluation cost associated with frequent fine-tuning iterations, AutoQRA decomposes the optimization process into two stages. First, it first conducts a global multi-fidelity evolutionary search, where the initial population is warm-started by injecting layer-wise importance priors. This stage employs specific operators and a performance model to efficiently screen candidate configurations. Second, trust-region Bayesian optimization is applied to locally refine promising regions of the search space and identify optimal configurations under the given memory budget. This approach enables active compensation for quantization noise in specific layers during training. Experiments show that AutoQRA achieves performance close to full-precision fine-tuning with a memory footprint comparable to uniform 4-bit methods.

Modern AI language models can be useful for many tasks, but adapting them to a new use case often requires expensive computers with large memory. A common way to reduce the cost is to shrink the model and train only small add-on parts, but current methods usually make these two decisions separately. This can waste memory: ome parts of the model may be stored too carefully, while other parts need more room to learn from new data. We introduce AutoQRA, a method that automatically decides how to divide a fixed memory budget across the model. It tests many possible choices cheaply, focuses on the most promising ones, and stops when further search is unlikely to help. In our experiments, AutoQRA lets language models reach nearly the same quality as more memory-hungry training methods while using memory close to highly compressed methods. This can make it easier and cheaper for researchers and developers to customize powerful language models with limited hardware.