Federated In-Context Learning: Iterative Refinement for Improved Answer Quality

For question-answering (QA) tasks, in-context learning (ICL) enables language models (LMs) to generate responses without modifying their parameters by leveraging examples provided in the input. However, the effectiveness of ICL heavily depends on the availability of high-quality examples, which are often scarce due to data privacy constraints, annotation costs, and distribution disparities. A natural solution is to utilize examples stored on client devices, but existing approaches either require transmitting model parameters—incurring significant communication overhead—or fail to fully exploit local datasets, limiting their effectiveness. To address these challenges, we propose Federated In-Context Learning (Fed-ICL), a general framework that enhances ICL through an iterative, collaborative process. Fed-ICL progressively refines responses by leveraging multi-round interactions between clients and a central server, improving answer quality without the need to transmit model parameters. We establish theoretical guarantees for the convergence of Fed-ICL and conduct extensive experiments on standard QA benchmarks, demonstrating that our proposed approach achieves strong performance while maintaining low communication costs.

Language models can answer questions by learning from example inputs, a process called in-context learning (ICL). While powerful, ICL depends on access to high-quality examples, which are often scarce due to privacy concerns, labeling costs, and data distribution differences. A natural solution is to use examples stored locally on users’ devices. However, existing approaches either require transmitting large model files—leading to high communication costs—or fail to fully utilize local data. We propose Federated In-Context Learning (Fed-ICL), a new framework that enables user devices to collaborate with a central server through multiple rounds of interaction. Without sharing raw data or model parameters, each round helps refine the model’s responses using locally available examples. This design preserves privacy and keeps communication efficient. Fed-ICL is backed by theoretical convergence guarantees and shows strong empirical performance on widely used question-answering benchmarks. It demonstrates a practical path toward more effective and privacy-preserving language model applications, especially in decentralized environments.