PolicyGuard: Towards Test-time and Step-level Adversary Defense for Reinforcement Learning Agent

While real-world applications of reinforcement learning (RL) are becoming increasingly popular, the security of RL systems deserve more attention and exploration. In particular, recent work has revealed that RL agents are vulnerable to backdoor attacks, where a victim agent behaves normally under standard conditions but executes malicious actions when a specific trigger is activated. Existing backdoor defenses for RL either require access to the agent’s internal parameters, operate only at the model or trajectory level, or are limited to specific attack types. To ensure the security of RL agents, we propose PolicyGuard, a test-time step-level backdoor defense which leverages Gaussian Process (GP) posterior variance and adapts pseudo trajectories to enable uncertainty computation for individual time step. Besides, we also provide theoretical foundations to explain the efficacy of GP posterior variance. Extensive experiments across seven RL games demonstrate that PolicyGuard achieves state-of-the-art detection performance in most cases, with average AUROC of 0.856 for perturbation-based attacks and 0.859 for adversary-agent attacks.

AI systems can learn to perform complex tasks — like playing video games or controlling robots — through trial and error, a process known as reinforcement learning. However, an attacker can secretly manipulate this learning process so that the AI behaves normally most of the time but acts dangerously when it encounters a specific hidden signal, much like a sleeper agent. Detecting such hidden manipulations is challenging because the AI appears to work perfectly under normal conditions. We propose PolicyGuard, a monitoring system that watches the AI's behavior in real time and flags suspicious actions as they happen. Our key idea is to first learn what "normal" behavior looks like by observing the AI in a safe setting, then continuously check whether the AI's actions during deployment fall within the expected range. When the AI encounters a hidden trigger and starts behaving abnormally, our system detects the deviation and raises an alarm — without needing to know what the trigger looks like or how the attack was carried out. In our experiments, PolicyGuard reliably detects a wide variety of such hidden attacks across different tasks, while rarely raising false alarms during normal operation.