In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning

This paper develops a finite-sample statistical theory for in-context learning (ICL), analyzed within a meta-learning framework that accommodates mixtures of diverse task types. We leverage a Bayes risk identity that separates the total ICL risk into two orthogonal components: Bayes Gap and Posterior Variance. The Bayes Gap quantifies how well the trained model approximates the Bayes-optimal in-context predictor. For a uniform-attention Transformer, we derive a non-asymptotic upper bound on this gap, which explicitly clarifies the dependence on the number of pretraining prompts and their context length. The Posterior Variance is a model-independent risk representing the intrinsic task uncertainty. Our key finding is that this term is determined solely by the difficulty of the true underlying task, while the uncertainty arising from the task mixture vanishes exponentially fast with only a few in-context examples. Together, these results provide a unified view of ICL: the uniform-attention Transformer selects the optimal meta-algorithm during pretraining and rapidly converges to the optimal algorithm for the true task at test time.

Modern AI systems such as large language models have a remarkable ability: show them a few examples in a prompt, and they can often solve a new task on the spot — no retraining required. This ability, called in-context learning, powers many everyday uses of these models, yet we still lack a clear mathematical explanation of why and when it works, especially when pretraining mixes many kinds of tasks. In this paper, we study this question through the lens of Bayesian inference: the principled way to update beliefs as new evidence arrives. In a mathematical model of in-context learning, we show that prediction error separates into two parts: how closely the trained model imitates the ideal Bayesian predictor, and the unavoidable uncertainty left by the task itself. We then characterize how these parts shrink as pretraining data, training prompt length, and test-time examples increase. Our results explain why, after only a few in-context examples, a model can infer which type of task it is facing and behave nearly optimally. They also offer practical guidance on prompt length, the benefits of pretraining, and the effects of test inputs that differ from training inputs.