Quantifying the Uncertainty of Foundation Models with Singular Value Ensembles

Foundation models have become a dominant paradigm in machine learning, achieving remarkable performance across diverse tasks through large-scale pretraining. However, they often yield overconfident, uncalibrated predictions. The standard approach to quantifying epistemic uncertainty are ensembles of multiple independently trained models. But their computational cost scales linearly with ensemble size, making them impractical for large foundation models. We propose Singular Value Ensemble (SVE), a parameter-efficient implicit ensembling method. SVE builds on a simple, but powerful core assumption: namely, that the singular vectors of the weight matrices correspond to meaningful directions in the representation space. If the singular vectors are indeed meaningful (orthogonal) "knowledge directions", then a model ensemble can be obtained by modulating only how strongly each direction contributes to the output. Rather than learning new parameters for each ensemble member, we freeze the singular vectors and only train per-member singular values that rescale the contribution of each direction in that shared knowledge basis. Ensemble diversity emerges naturally during joint training as stochastic initialization and random batch sampling cause different members to converge to different combinations of the same underlying knowledge. SVE performs comparable to an explicit ensemble, while increasing the parameter count of the base model by <1%, making principled uncertainty estimation accessible in resource-constrained settings. We validate SVE on NLP and vision tasks with various different backbones and show that it improves calibration while maintaining predictive accuracy.

Modern AI systems known as "foundation models" are powerful but often overconfident, giving wrong answers with high certainty. In high-stakes settings like medical diagnosis or self-driving cars, we want these systems to know when they don't know. A reliable way to measure an AI's uncertainty is to train several copies of a model and see where they disagree, but running many full copies of a large model is very expensive. We introduce Singular Value Ensemble (SVE), which achieves the same benefit at a tiny fraction of the cost: instead of training separate models, it reuses the knowledge already inside a single pretrained model and creates slightly different versions by adjusting a small set of internal "dials." These versions disagree just enough to reveal where the model is unsure, while adding less than 1% to its size, making trustworthy, uncertainty-aware AI accessible far beyond those with large computing budgets.