Automata-Conditioned Cooperative Multi-Agent Reinforcement Learning

We study learning multi-task, multi-agent policies for cooperative, temporal objectives, under centralized training, decentralized execution. In this setting, using automata to represent tasks assigned to agents enables breaking down a team-level objective into simpler, smaller sub-tasks. However, existing approaches remain sample-inefficient and are limited to the single-task case, requiring retraining policies for each new task. In this work, we present Automata-Conditioned Cooperative Multi-Agent Reinforcement Learning (ACC-MARL), a framework for learning task-conditioned, decentralized team policies. We identify challenges to the feasibility of ACC-MARL, propose solutions, and prove that our approach is optimal. We further show that learned value functions can be used to assign tasks optimally at test time. Experiments demonstrate emergent task-aware, multi-step coordination among agents, such as pressing a button to unlock a door, holding the door, and short-circuiting tasks.

Teaching a team of AI agents to cooperate on complex, multi-step tasks is difficult, especially when the team must adapt to new tasks on the fly. We present a framework where each agent learns to reason over both its own tasks and its teammates' tasks, allowing the team to coordinate intelligently and cooperate to complete their goals. Agents learn generalizable coordination skills without being explicitly programmed to do so.