Many-Shot CoT-ICL: Making In-Context Learning Truly Learn

While many-shot ICL achieves remarkable performance, prior studies of its scaling behavior have mainly focused on non-reasoning tasks. In this work, we study many-shot ICL on reasoning tasks, with a particular focus on many-shot chain-of-thought in-context learning (CoT-ICL). Analyzing across non-reasoning and reasoning tasks and across non-reasoning and reasoning-oriented LLMs, we identify several distinctive properties of many-shot CoT-ICL. We further interpret these findings by viewing many-shot CoT-ICL as in-context test-time learning rather than scaled pattern matching, and suggest two principles: (i) demonstrations should be easy for the target model to understand, and (ii) they should be ordered to support a smooth conceptual progression. Guided by the principle, we propose Curvilinear Demonstration Selection (CDS), a simple ordering method that yields up to a 5.42 percentage-point gain on a math task with 64 demonstrations. Overall, our results reframe the long context window from a retrieval buffer into a structured curriculum for in-context test-time learning.

Large language models can often solve new tasks by looking at examples in their prompt, without being retrained. As models can now read much longer prompts, we can give them many examples, but it is unclear whether simply adding more examples helps when the task requires multi-step reasoning. This paper studies that question for examples that include step-by-step explanations, such as math or narrative reasoning solutions. We find that the usual rules for many-example prompting do not always apply: examples that look similar to the test question can be unhelpful, and the order of examples matters more as the prompt gets longer. We argue that reasoning examples should be treated less like a database to search and more like a curriculum that teaches the model during inference. Based on this idea, we propose a simple method that orders examples so that the reasoning steps change smoothly from one example to the next. This improves performance on math and narrative reasoning tasks. Our findings suggest that long context windows are most useful when examples are organized to support learning, not just retrieval.