Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

Large language models are typically optimized for accuracy and, therefore, will guess even when uncertain about their predictions. This problem becomes especially pronounced when the model is fine-tuned on small datasets, which often causes overfitting and results in a tendency toward miscalibration. In this work, we introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes (SGP). We identify a structural isomorphism between LoRA's factorization and Kronecker-factored SGP posteriors, and show that LoRA emerges as a limiting case when posterior uncertainty collapses. We conduct extensive experiments on various LLM architectures across commonsense reasoning, language modeling, and mathematical reasoning benchmarks. With only approximately 0.42M additional parameters and ${\approx}1.2{\times}$ training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models from 7B up to 30B, achieving up to 84\% Expected Calibration Error (ECE) and 76\% Negative Log-Likelihood (NLL) reduction while maintaining competitive accuracy for both in-distribution and out-of-distribution (OoD) evaluations.

Large language models can sound very confident even when they are wrong. This overconfidence becomes dangerous when models are used in high-stakes settings, like medical diagnosis or autonomous driving, where knowing \emph{when the model might be wrong} matters as much as the answer itself. In this paper, we propose Bayesian-LoRA, a framework that can capture the inherent uncertainty of model outputs. It builds upon Low-Rank Adaptation (LoRA), a method for fine-tuning Large Language Models from a base model to specific downstream tasks. We establish a mathematical connection between LoRA and a well-established statistical framework called Sparse Gaussian Processes (SGP), which allows us to turn LoRA's fixed update into a probabilistic one and powers the model's capability to express how confident the answer is. For almost no extra cost (~20% more training time and just 0.42M added parameters), our method makes models far better at knowing what they don't know. It reduces calibration error by up to 84% while keeping accuracy competitive. It gains the largest benefits when the model faces unfamiliar, out-of-distribution data, exactly where overconfidence is most dangerous. By making confidence scores more trustworthy, Bayesian-LoRA helps users know when to rely on a model's answer and when to seek further verification.