Beyond Rewards in RL for Cyber Defence

Recent years have seen an explosion of interest in autonomous cyber defence agents trained to defend computer networks using deep reinforcement learning. These agents are typically trained in cyber gym environments using dense, highly engineered reward functions which combine many penalties and incentives for a range of (un)desirable states and costly actions. Dense rewards help alleviate the challenge of exploring complex environments but risk biasing agents towards suboptimal and potentially riskier solutions, a critical issue in complex cyber environments. We thoroughly evaluate the impact of reward function structure on learning and policy behavioural characteristics using a variety of sparse and dense reward functions, two well-established cyber gyms, a range of network sizes, and both policy gradient and value-based RL algorithms. Our evaluation is enabled by a novel ground truth evaluation approach which allows directly comparing between different reward functions, illuminating the nuanced inter-relationships between rewards, action space and the risks of suboptimal policies in cyber environments. Our results show that sparse rewards, provided they are goal aligned and can be encountered frequently, uniquely offer both enhanced training reliability and more effective cyber defence agents with lower-risk policies. Surprisingly, sparse rewards can also yield policies that are better aligned with cyber defender goals and make sparing use of costly defensive actions without explicit reward-based numerical penalties.

Deep reinforcement learning is increasingly being applied to autonomous cyber defence, where agents learn to protect computer networks against attackers. Almost all existing work trains these agents with hand-engineered reward functions that combine many penalties and incentives for different actions and network states. We ask whether this is actually a good idea. To answer this rigorously, we first identify a methodological problem: standard cyber simulators evaluate agents using the same reward signal they were trained on, which both depends on arbitrary numerical choices and silently ignores compromises that occur and are resolved within a single time step. We introduce a ground truth scoring mechanism that counts compromised nodes directly and is independent of any reward function, then use it to systematically compare a range of sparse and dense rewards across two widely used cyber gyms, two RL algorithms, network sizes from 2 to 50 nodes, and different orderings of attacker and defender actions. We find that the simplest sparse rewards, a positive signal only when no nodes are compromised, consistently produce agents that are more effective, lower risk, and far more reproducibly trained than agents trained with dense rewards, and that this advantage grows with the size and realism of the environment. The implication is that a substantial fraction of current evaluation practice in autonomous cyber defence overstates agent capability, and that simpler, goal-aligned reward design is both more honest and more performant.