Cyber attacks are increasing in volume, frequency, and complexity. In response, the security community is looking toward fully automating cyber defense systems using machine learning. However, so far the resultant effects on the coevolutionary dynamics of attackers and defenders have not been examined. In this paper, we argue, and provide empirical evidence, that increased automation on both sides accelerates the coevolutionary cycle. This begs the question of whether there exist any natural fixed points in the resultant attacker-defender game, and how these are characterised. Working within the threat model of Locked Shields, Europe's largest cyber defense exercise, we study blackbox adversarial attacks on network classifiers. Given already existing attack capabilities, we question the utility of optimal evasion attack frameworks based on minimal evasion distances. Instead, we propose a novel reinforcement learning setting that can be used to efficiently generate arbitrary adversarial perturbations and empirically demonstrate its utility. We then argue that attacker-defender fixed points are themselves general-sum games with complex phase transitions, and introduce, and empirically analyse, a temporally extended multi-agent reinforcement learning framework in which the resultant dynamics can be studied. Lastly, we investigate a specific plausible attacker-defender co-evolutionary fixed point and argue that continual learning techniques are indispensable for finding both optimal attacker and defender strategies in such settings.