Context Tuning for In-Context Optimization

We introduce Context Tuning, a simple and effective method to significantly enhance few-shot adaptation of language models (LLMs) without fine-tuning model parameters. While prompt-based adaptation techniques have demonstrated the effectiveness of lightweight adaptation methods for LLMs, they typically initialize a trainable prompt or prefix with irrelevant tokens for the task at hand. In contrast, Context Tuning initializes the trainable prompt or prefix with task-specific demonstration examples, leveraging the model’s inherent In-Context Learning (ICL) ability to extract relevant information for improved few-shot learning performance. Extensive evaluations on benchmarks such as CrossFit, UnifiedQA, MMLU, BIG-Bench Hard, and ARC demonstrate that Context Tuning outperforms traditional prompt-based adaptation methods and achieves competitive accuracy with Test-Time Training at significantly higher training efficiency.

When you show a large language model a handful of examples of a new task, it can often spot the pattern and respond correctly, an ability called in-context learning. But the model just passively reads those examples, so on harder tasks it falls short. We introduce Context Tuning, a lightweight method that takes the same examples and uses them to actively train an internal prompt; the model itself never changes, so our method achieves low latency and allows a single model to serve many tasks just by swapping in different prompts. Across four benchmarks testing language understanding, subject-area knowledge, logical reasoning, and abstract visual puzzles, Context Tuning matches or surpasses methods that require updating model parameters, and combining the two yields our strongest results. This gives researchers and practitioners a practical way to adapt large language models to new tasks from only a handful of examples.