GPan-LoRA: Gaussian Process Amortized Networks for Bayesian Low-Rank Adaptation in Large Language Models

Principled uncertainty quantification (UQ) is increasingly recognized as essential for trustworthy artificial general intelligence (AGI). Bayesian Low-Rank Adaptation (LoRA) provides a principled mechanism for uncertainty-aware fine-tuning of large language models (LLMs). However, existing techniques either face scalability constraints, e.g. Laplace-LoRA, or rely on approximate inference schemes that lead to poorly calibrated posterior uncertainty, often manifesting as overconfident predictions under distribution shift. To address this challenge, we propose GPan-LoRA, the first scalable Gaussian Process (GP)-based framework for Bayesian LoRA, which integrates neural network-based sparse GP approximations with amortized variational inference. By preserving the Bayesian function prior and posterior semantics intrinsic to GPs, GPan-LoRA achieves a faithful balance between computational scalability and principled UQ. Empirically, GPan-LoRA produces well-calibrated uncertainty that remains reliable under distribution shift, mitigating overconfident failures while preserving competitive task performance.

Large language models can give confident answers even when they are wrong, especially on questions that differ from their fine-tuning data. Existing Bayesian LoRA methods aim to estimate this uncertainty during efficient fine-tuning, but they often face a trade-off: some are computationally expensive, while others are efficient yet overconfident. This paper introduces GPan-LoRA, which brings Gaussian processes, a classical probabilistic tool for uncertainty estimation, into LoRA fine-tuning in a scalable way. By applying uncertainty estimation only to the small trainable parts of the model, GPan-LoRA remains practical for large models. Experiments show that it provides more reliable confidence estimates under distribution changes while maintaining competitive task performance.