Recorded-PBT: Simple and Efficient Approach for Safe and Robust Policy using Population-based Training

Autonomous driving policies trained with self-play reinforcement learning (RL) generalize well to unseen scenarios, but zero-shot coordination with unseen partners remains underexplored, especially for safety-critical long-tail behaviors. Population-based training (PBT), which trains an ego policy against a diverse population of pre-trained agents, is a promising alternative, but conventional PBT still allows partners to adapt online during simulation, potentially encouraging reliance on their responses. To mitigate this, we propose recorded-play PBT (recorded-PBT), which replays pre-recorded partner trajectories during ego-policy learning. By replacing online adaptation with fixed partner behavior (non-reactive), recorded-PBT promotes more conservative and robust coordination while improving training efficiency. In both a simple matrix game and a complex multi-agent driving simulator, recorded-PBT achieves lower collision rates and higher success rates than conventional PBT against non-reactive partners, including replayed human trajectories, while remaining competitive under reactive evaluation.