Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork

In-Context Reinforcement Learning (ICRL) has enabled foundation agents to adapt instantaneously to novel tasks, yet its efficacy in Ad-Hoc Teamwork (AHT)—where coordination with unknown partners is required—remains unexplored. To rigorously evaluate this, we introduce a large-scale benchmark ICRL4AHT, built upon a high-throughput JAX implementation of Overcooked-V2. Our benchmark includes a large, diverse teammate suite spanning both RL and heuristic policies, enabling controlled train-test shifts, and provides a reproducible end-to-end pipeline for teammate generation, learning-history collection, dataset construction, and online multi-episode evaluation. We evaluate representative history-conditioned ICRL algorithms, including Algorithm Distillation (AD) and Decision-Pretrained Transformer (DPT), across millions of transitions. Results reveal notable limitations: contrary to their success in single-agent domains, these baselines fail to exhibit robust test-time adaptation in multi-agent settings. Specifically, these methods frequently underperform random baselines across both unseen teammate and unseen layout tracks, with no clear in-context improvement over long horizons. These findings highlight the challenges of strategic inference under partial observability within the OvercookedV2 AHT protocol, establishing our benchmark as a critical testbed for next-generation coordination algorithms.

AI agents are increasingly expected to work with unfamiliar partners, such as people, robots, or other AI systems. This paper asks whether current agents can quickly understand a new teammate’s behavior and adapt while working together. We introduce ICRL4AHT, a benchmark based on a cooperative cooking game where two agents must prepare and deliver recipes together. The benchmark includes many teammate types and kitchen layouts, allowing us to test whether an agent can handle new partners and new situations. Our experiments show that current methods struggle with this challenge. Even after observing a teammate for many episodes, the agents often do not improve, and sometimes perform worse than a random agent. These results suggest that today’s agents still have difficulty understanding and adapting to unfamiliar teammates, highlighting an important direction for future research in cooperative AI.