Probabilistic Performance Guarantees for Multi-Task Reinforcement Learning

Multi-task reinforcement learning trains generalist policies that can execute multiple tasks. While recent years have seen significant progress, existing approaches rarely provide formal performance guarantees, which are indispensable when deploying policies in safety-critical settings. We present an approach for computing high-confidence guarantees on the performance of a multi-task policy on tasks not seen during training. Concretely, we introduce a new generalisation bound that composes (i) per-task lower confidence bounds from finitely many rollouts with (ii) task-level generalisation from finitely many sampled tasks, yielding a high-confidence guarantee for new tasks drawn from the same arbitrary and unknown distribution. Across state-of-the-art multi-task RL methods, we show that the guarantees are theoretically sound and informative at realistic sample sizes.

When we train AI agents to handle many different tasks, we usually just hope they'll perform well on new ones — but we can't say how well with any mathematical certainty. This matters a lot in high-stakes situations (e.g. autonomous vehicles) where "probably fine" isn't good enough. We introduce a method to give a rigorous, provable guarantee: if you train an AI agent on a sample of tasks and then test it on a new task from the same family, we can attach a precise confidence level to how well it will perform on this unseen task. The key insight is combining two sources of statistical uncertainty: the limited number of trials per task, and the limited number of tasks seen during training. As AI agents are increasingly used to perform general tasks in safety-critical settings, our findings will be useful to certify that these systems behave as we want them to.