MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning

Benchmarks play a central role in reinforcement learning (RL) research, yet their computational constraints often shape what is studied. Despite the motivation of lifelong learning, most continual RL papers consider only 3–10 sequential tasks, as CPU-bound environments make longer sequences impractical. Meanwhile, continual learning in cooperative multi-agent settings remains largely unexplored. To address these gaps, we introduce MEAL (Multi-agent Environments for Adaptive Learning), the first benchmark for continual multi-agent RL. By leveraging JAX and GPU acceleration, MEAL enables training on sequences of 100 tasks on a single GPU in a few hours. We find that long task sequences reveal failure modes that do not appear at smaller scales.

Artificial agents that learn by trial and error are usually trained on one task and then frozen. But real-world jobs keep changing, and a useful agent should keep picking up new tasks over a long lifetime without forgetting old ones. Studying this is slow and costly, so most research tries only a handful of tasks in a row, and almost none looks at agents that must work together. We built MEAL, a testing ground that tackles both gaps. It is based on a cooperative cooking game where players coordinate to prepare and deliver dishes. MEAL generates an endless variety of kitchens of increasing difficulty, and because it runs on modern graphics hardware, it can train agents on a hundred kitchens in a few hours on a single machine, far more than was previously practical. Using MEAL, we found that short experiments can mislead: methods that look strong over ten tasks can fall apart over a hundred. We also found that keeping a team coordinated is a separate, harder problem than remembering individual skills, and it worsens as teams grow and teammates change. We hope MEAL gives researchers an affordable way to study how cooperative agents keep learning and working together over the long run.