In-Context Learning as Rate–Distortion Optimization

In-context learning (ICL) is a practical way to adapt large models, yet under strict context limits, it remains unclear how to spend scarce tokens without being misled by noisy, redundant, or conflicting demonstrations. We address this gap by targeting token-budgeted context construction: how to select and compress demonstrations so the prompt carries maximal task-relevant signal with minimal predictive distortion. We propose RDCO, a deterministic, training-free optimizer that scores demonstrations by marginal task information per token, penalizes redundancy and prefix-conditioned conflicts, and finally compacts the selected context under a bounded predictive-divergence constraint to control drift. Across a 10-dataset ICL suite spanning classification and structured generation, RDCO achieves the best average performance (63.26 Acc. on classification and 60.26 EM on generation), improving over the strongest classification and generation baselines by 2.20 and 2.26 points, respectively, and improving the 10-task overall average by 4.94 points under the same budget. Our results suggest that viewing prompts as finite-capacity messages yields a principled and effective path to more reliable and token-efficient ICL.

Large language models can often solve new tasks by reading a few examples placed in their prompt. However, prompts have limited space, and not every example is helpful. Some examples repeat information that is already present, some are noisy or misleading, and some may even conflict with the task the model is trying to solve. This paper studies how to make better use of this limited prompt space. We propose a method that automatically chooses which examples to include and how to shorten them while preserving the information that matters for the model’s answer. The key idea is to treat the prompt as a limited communication channel: each token should be used only when it provides useful evidence for the current task. Our method favors examples that are informative, avoids examples that are redundant or inconsistent, and checks that shortening the prompt does not significantly change the model’s prediction. Across a range of classification and generation tasks, this approach improves both accuracy and efficiency compared with strong existing methods. These results suggest that carefully managing the content of prompts can make in-context learning more reliable and less wasteful, especially when the available context window is small.