Understanding Generalization and Forgetting in In-Context Continual Learning

In-context learning (ICL) derives its power from enabling Large Language Models to adapt to new tasks via prompt-based reasoning alone, entirely bypassing the need for parameter updates. Existing theories primarily study ICL in single-task settings, while real-world prompts often contain sequences of heterogeneous tasks, leaving a gap in understanding whether Large Language Models implicitly perform continual learning during inference. To bridge this gap, we propose the first theoretical framework for in-context continual learning, modeling how a pretrained Transformer processes multiple sequential tasks within a single prompt through shared attention mechanisms. Focusing on linear and masked linear self-attention, we derive error expressions for model predictions under sequential task prompts and analyze their generalization and forgetting behavior. Our results reveal that standard attention mechanisms inevitably induce inter-task interference by uniformly or causally aggregating historical contexts, leading to systematic bias. We further provide a bias–variance–interference decomposition of prediction error, characterizing when historical in-context information yields positive transfer or provable negative transfer. This analysis exposes fundamental limits of attention-based continual inference and offers theoretical explanations for order sensitivity and performance degradation in long prompts.

Modern AI systems can learn new tasks simply by reading examples in a prompt, without changing their internal parameters. This ability, known as in-context learning, is one reason large language models like chatbots can quickly adapt to many different requests. However, most existing research studies this process one task at a time, while real-world conversations often contain many different tasks in sequence. In this work, we develop the first mathematical framework to study how AI models handle multiple tasks within a single prompt. We show that current attention mechanisms, the core component that allows Transformers to process context, naturally mix information from different tasks together. While earlier information can sometimes help later tasks, it can also interfere with them, causing mistakes, sensitivity to task order and performance drops as prompts become longer. Our analysis explains why large language models may “forget” or become less reliable when many tasks are combined in one conversation. These findings provide a theoretical foundation for understanding the limits of long-context AI systems and may help guide the design of more reliable and robust models in the future.