Humans frequently compete with one another. They sometimes develop adversarial systems. And, they often cooperate to take on a common foe. My interest in Artificial Adversarial Intelligence has led me to cyber security where I employ an array of machine learning approaches. These allow cyber activity to be simulated and modeled. They also unlock text-based data sources that record, organize, and (partially) link cyber threats, targets, and threat mitigations. I will describe some of my projects in Artificial Adversarial Intelligence and present a roadmap for future research.

In this paper we explore cyber security defence, through the unification of a novel cyber security simulator with models for (causal) decision-making through optimisation. Particular attention is paid to a recently published approach: dynamic causal Bayesian optimisation (DCBO). We propose that DCBO can act as a blue agent when provided with a view of a simulated network and a causal model of how a red agent spreads within that network. To investigate how DCBO can perform optimal interventions on host nodes, in order to reduce the cost of intrusions caused by the red agent. Through this we demonstrate a complete cyber-simulation system, which we use to generate observational data for DCBO and provide numerical quantitative results which lay the foundations for future work in this space.

We study automated intrusion prevention using reinforcement learning. Following a novel approach, we formulate the interaction between an attacker and a defender as an optimal stopping game and let attack and defense strategies evolve through reinforcement learning and self-play. The game-theoretic perspective allows us to find defender strategies that are effective against dynamic attackers. The optimal stopping formulation gives us insight into the structure of optimal strategies, which we show to have threshold properties. To obtain the optimal defender strategies, we introduce T-FP, a fictitious self-play algorithm that learns Nash equilibria through stochastic approximation. We show that T-FP outperforms a state-of-the-art algorithm for our use case. Our overall method for learning and evaluating strategies includes two systems: a simulation system where defender strategies are incrementally learned and an emulation system where statistics are produced that drive simulation runs and where learned strategies are evaluated. We conclude that this approach can produce effective defender strategies for a practical IT infrastructure.

Adversarial attacks in reinforcement learning (RL) often assume highly-privileged access to the learning agent’s parameters, environment or data. Instead, this paper proposes a novel adversarial setting called a Cheap Talk MDP in which an Adversary has a minimal range of influence over the Victim. Parameterised as a deterministic policy that only conditions on the current state, an Adversary can merely append information to a Victim’s observation. To motivate the minimum-viability, we prove that in this setting the Adversary cannot occlude the ground truth, influence the underlying dynamics of the environment, introduce non-stationarity, add stochasticity, see the Victim’s actions, or access their parameters. Additionally, we present a novel meta-learning algorithm to train the Adversary, called adversarial cheap talk (ACT). Using ACT, we demonstrate that the resulting Adversary still manages to influence the Victim’s training and test performance despite these restrictive assumptions. Affecting train-time performance reveals a new attack vector and provides insight into the success and failure modes of existing RL algorithms. More specifically, we show that an ACT Adversary is capable of harming performance by interfering with the learner’s function approximation and helping the Victim’s performance by appending useful features. Finally, we demonstrate that an ACT Adversary can append information during train-time to directly and arbitrarily control the Victim at test-time in a zero-shot manner.

Recent high-profile cyber attacks have made it clear that the traditional signature-based defensive tools at the network perimeter are no longer sufficient for protecting the ever-expanding attack surface of enterprise-level computer networks. Advanced Persistent Threats (APTs) are gaining unauthorized access to networks, and performing complex, multi-stage attack campaigns, often only being detected long after accomplishing their mission. Unfortunately, due to the sensitivity of enterprise network data, there is a lack of realistic and complete, enterprise-grade data available to the research community for the purposes of building better algorithms and tools capable of modeling attacks and defending enterprise networks. The existing cybersecurity datasets often contain little to no attack data, or are unrealistically simple attacks which are not representative of full APT-level attack campaigns. In this work we generate a compact yet realistic attack-focused dataset in a simulated enterprise computer network using tools and procedures common to both the attackers and defenders. We orchestrate, document, and carry out an APT-level compromise of the domain which covers multiple tactics, techniques, and procedures across the full life-cycle of an attack. We perform full network-level monitoring typical of enterprise network defenders to capture a high fidelity and complete representation of the attack. We evaluate our dataset against the existing datasets available to the community for breadth, completeness, and utility.

Recent exploration of the multi-agent backdoor attack demonstrated the backfiring effect, a natural defense against backdoor attacks where backdoored inputs are randomly classified. This yields a side-effect of low accuracy w.r.t. clean labels, which motivates this paper's work on the construction of multi-agent backdoor defenses that maximize accuracy w.r.t. clean labels and minimize that of poison labels. Founded upon agent dynamics and low-loss subspace construction, we contribute three defenses that yield improved multi-agent backdoor robustness.

Recently, advances in deep learning have been observed in various fields, including computer vision, natural language processing, and cybersecurity. Machine learning (ML) has demonstrated its ability as a potential tool for anomaly detection-based intrusion detection systems to build secure computer networks. Increasingly, ML approaches are widely adopted than heuristic approaches for cybersecurity because they learn directly from data. Data is critical for the development of ML systems, and becomes potential targets for attackers. Basically, evasion attacks, also known as adversarial attacks, and data poisoning or contamination, are among the most common techniques used to fool ML models through data. This paper evaluates the robustness of six recent deep learning algorithms for intrusion detection on contaminated data. Our experiments suggest that the state-of-the-art algorithms used in this study are sensitive to data contamination and reveal the importance of self-defense against data perturbation when developing novel models, especially for intrusion detection systems.

Cyber attacks are increasing in volume, frequency, and complexity. In response, the security community is looking toward fully automating cyber defense systems using machine learning. However, so far the resultant effects on the coevolutionary dynamics of attackers and defenders have not been examined. In this paper, we argue, and provide empirical evidence, that increased automation on both sides accelerates the coevolutionary cycle. This begs the question of whether there exist any natural fixed points in the resultant attacker-defender game, and how these are characterised. Working within the threat model of Locked Shields, Europe's largest cyber defense exercise, we study blackbox adversarial attacks on network classifiers. Given already existing attack capabilities, we question the utility of optimal evasion attack frameworks based on minimal evasion distances. Instead, we propose a novel reinforcement learning setting that can be used to efficiently generate arbitrary adversarial perturbations and empirically demonstrate its utility. We then argue that attacker-defender fixed points are themselves general-sum games with complex phase transitions, and introduce, and empirically analyse, a temporally extended multi-agent reinforcement learning framework in which the resultant dynamics can be studied. Lastly, we investigate a specific plausible attacker-defender co-evolutionary fixed point and argue that continual learning techniques are indispensable for finding both optimal attacker and defender strategies in such settings.

The adoption of autonomous cyber defence agents within real-world contexts requires them to be able to cope with differences between their training and target environments, bridging the simulation to real gap, in order to provide robust, generalised defensive responses. Whilst the simulation to real gap has been studied in-depth across domains such as robotics to date there has been minimal research considering generalisability in the context of cyber defence agents and how differences in observation space could enhance agent generalisability when placed into environments that differ from the training environment. Within this paper, we propose a method of enhancing agent generalisability and performance within unseen environments by integrating a graph embedded network representation into the agents observation space. We then compare agent performance with and without a graph embedded network representation based observation space within a series of randomised cyber defence simulations. We find that there is a trade off between the effectiveness of the graph embedding representation and the complexity of the graph, in terms of node count and number of edges.

It is our position that when hypergraphs are used to capture multi-way local relations of data, their resulting topological features describe global behavior. These features capture complex correlations that can then serve as high fidelity inputs to autoencoder-driven anomaly detection pipelines. We propose two such potential pipelines for cybersecurity data, one that uses an autoencoder directly to determine network intrusions, and one that de-noises input data for a persistent homology system, PHANTOM. We provide heuristic justification for the use of the methods described therein for an intrusion detection pipeline for cyber data. We conclude by showing a small example over synthetic cyber attack data.

Perceptual hashes map images with identical semantic content to the same $n$-bit hash value, while mapping semantically-different images to different hashes. These algorithms carry important applications in cybersecurity such as copyright infringement detection, content fingerprinting, and surveillance. Apple's NeuralHash is one such system that aims to detect the presence of illegal content on users' devices without compromising consumer privacy. We make the surprising discovery that NeuralHash is approximately linear, which inspires the development of novel black-box attacks that can (i) evade detection of "illegal" images, (ii) generate near-collisions, and (iii) leak information about hashed images, all without access to model parameters. These vulnerabilities pose serious threats to NeuralHash's security goals; to address them, we propose a simple fix using classical cryptographic standards.

Self-play reinforcement learning has achieved state-of-the-art, and often superhuman, performance in a variety of zero-sum games. Yet prior work has found that policies that are highly capable against regular opponents can fail catastrophically against adversarial policies: an opponent trained explicitly against the victim. Prior defenses using adversarial training were able to make the victim robust to a specific adversary, but the victim remained vulnerable to new adversaries. We conjecture this limitation was due to insufficient diversity of adversaries seen during training. We propose a defense using population-based training to pit the victim against a range of opponents. We evaluate this defense's robustness against new adversaries in two low-dimensional environments. We find that our defense significantly increases robustness against adversaries in both environments and show that robustness is correlated with the size of the opponent population.

We demonstrate how machine learning could serve cyber knowledge-base curators, threat hunters, and security analysts. We present a a machine learning~(ML) based workflow that addresses the overwhelming quantity of text entries that would have to be read and assimilated by hunters and analysts in order to infer a plausible relationship between two entries from different threat, vulnerability, and mitigation sources. The workflow uses language embedding models and classifiers to automatically label exhaustively-collected pairs of entries as linked or not. It also includes humans who in a curator role guide how many pairs are labeled as linked, ``candidates''. The curator also ranks the candidates; experts participate by independently and manually categorically assessing the ML-derived candidates; and the curator reprises by specifying rules that state how the collective expert categorizations determine a final label for each candidate pair.

Cyber vulnerability management is a critical function performed by a cybersecurity operations center (CSOC) that helps protect organizations against cyber-attacks on their computer and network systems. Adversaries hold an asymmetric advantage over the CSOC, as the number of deficiencies in these systems is increasing at a significantly higher rate compared to the rate at which the security teams are expanding to mitigate them in a resource-constrained CSOC environment. The current approaches employed at the CSOCs and recently published in the literature are deterministic and one-time decision-making methods, which do not consider future uncertainties when prioritizing and selecting vulnerabilities for mitigation. In addition, these approaches are constrained by the sub-optimal distribution of resources, providing no flexibility to the security team to adjust their response to fluctuations in vulnerability arrivals and thereby, weakening their security posture. We propose a novel artificial intelligence-enabled framework, Deep VULMAN, which consists of a deep reinforcement learning agent and an integer programming method to fill this gap in the cyber vulnerability management process. Our sequential decision-making framework, first, determines the near-optimal amount of resources to be allocated for mitigation under uncertainty, given an observed state of the system, and then determines the optimal set of prioritized vulnerability instances selected for mitigation. Our proposed framework outperforms the current methods in prioritizing the selection of important organization-specific vulnerabilities on real-world vulnerability data observed over a one-year period.

High power charging fosters the adoption of electric vehicles as it ameliorates recharge time concerns. The high power application combined with network communications among the vehicle, charging infrastructure, and electric supply potentially scales and intensifies risks posed by cyberattacks. We introduce and evaluate a hybrid Long Short-Term Memory (LSTM) autoencoder and One-Class Support Vector Machine (OCSVM) self-supervised model to identify novel patterns of encrypted vehicle-charger communications. Due to communication consistency, novel patterns may indicate misuse. The autoencoder is trained using only examples of normal classes. The OCSVM input is then derived from the autoencoder's compressed representation. We use a Log4j vulnerability to demonstrate that our approach can detect misuse without access to the communication contents.

Cyber Threat Intelligence (CTI) is information de-scribing threat vectors, vulnerabilities, and attacksand is often used as training data for AI-based cy-ber defense systems such as Cybersecurity Knowl-edge Graphs (CKG). There is a large need todevelop community-accessible datasets to trainexisting AI-based cybersecurity pipelines to effi-ciently and accurately extract meaningful insightsfrom CTI. We have created an initial unstructuredCTI corpus from a variety of open sources thatwe are using to train and test cybersecurity entitymodels using the spaCy framework and exploringself-learning methods to automatically recognizecybersecurity entities. We also describe methodsto apply cybersecurity domain entity linking withexisting world knowledge from Wikidata. Ourfuture work will survey and test spaCy NLP tools,and create methods for continuous integration ofnew information extracted from text.

A walk-through of the process security researchers go through to find modern kernel exploits, and a discussion of potential ways to improve bug finding and categorization with AI. We present CVE-2022-29968, an original Linux kernel exploit we developed, and discuss the current challenges researchers face with respect to exploit categorization and automated discovery.

Recent high-profile cyber attacks have made it clear that the traditional signature-based defensive tools at the network perimeter are no longer sufficient for protecting the ever-expanding attack surface of enterprise-level computer networks. Advanced Persistent Threats (APTs) are gaining unauthorized access to networks, and performing complex, multi-stage attack campaigns, often only being detected long after accomplishing their mission. Unfortunately, due to the sensitivity of enterprise network data, there is a lack of realistic and complete, enterprise-grade data available to the research community for the purposes of building better algorithms and tools capable of modeling attacks and defending enterprise networks. The existing cybersecurity datasets often contain little to no attack data, or are unrealistically simple attacks which are not representative of full APT-level attack campaigns. In this work we generate a compact yet realistic attack-focused dataset in a simulated enterprise computer network using tools and procedures common to both the attackers and defenders. We orchestrate, document, and carry out an APT-level compromise of the domain which covers multiple tactics, techniques, and procedures across the full life-cycle of an attack. We perform full network-level monitoring typical of enterprise network defenders to capture a high fidelity and complete representation of the attack. We evaluate our dataset against the existing datasets available to the community for breadth, completeness, and utility.

Recent exploration of the multi-agent backdoor attack demonstrated the backfiring effect, a natural defense against backdoor attacks where backdoored inputs are randomly classified. This yields a side-effect of low accuracy w.r.t. clean labels, which motivates this paper's work on the construction of multi-agent backdoor defenses that maximize accuracy w.r.t. clean labels and minimize that of poison labels. Founded upon agent dynamics and low-loss subspace construction, we contribute three defenses that yield improved multi-agent backdoor robustness.

Recently, advances in deep learning have been observed in various fields, including computer vision, natural language processing, and cybersecurity. Machine learning (ML) has demonstrated its ability as a potential tool for anomaly detection-based intrusion detection systems to build secure computer networks. Increasingly, ML approaches are widely adopted than heuristic approaches for cybersecurity because they learn directly from data. Data is critical for the development of ML systems, and becomes potential targets for attackers. Basically, evasion attacks, also known as adversarial attacks, and data poisoning or contamination, are among the most common techniques used to fool ML models through data. This paper evaluates the robustness of six recent deep learning algorithms for intrusion detection on contaminated data. Our experiments suggest that the state-of-the-art algorithms used in this study are sensitive to data contamination and reveal the importance of self-defense against data perturbation when developing novel models, especially for intrusion detection systems.

Cyber attacks are increasing in volume, frequency, and complexity. In response, the security community is looking toward fully automating cyber defense systems using machine learning. However, so far the resultant effects on the coevolutionary dynamics of attackers and defenders have not been examined. In this paper, we argue, and provide empirical evidence, that increased automation on both sides accelerates the coevolutionary cycle. This begs the question of whether there exist any natural fixed points in the resultant attacker-defender game, and how these are characterised. Working within the threat model of Locked Shields, Europe's largest cyber defense exercise, we study blackbox adversarial attacks on network classifiers. Given already existing attack capabilities, we question the utility of optimal evasion attack frameworks based on minimal evasion distances. Instead, we propose a novel reinforcement learning setting that can be used to efficiently generate arbitrary adversarial perturbations and empirically demonstrate its utility. We then argue that attacker-defender fixed points are themselves general-sum games with complex phase transitions, and introduce, and empirically analyse, a temporally extended multi-agent reinforcement learning framework in which the resultant dynamics can be studied. Lastly, we investigate a specific plausible attacker-defender co-evolutionary fixed point and argue that continual learning techniques are indispensable for finding both optimal attacker and defender strategies in such settings.

The adoption of autonomous cyber defence agents within real-world contexts requires them to be able to cope with differences between their training and target environments, bridging the simulation to real gap, in order to provide robust, generalised defensive responses. Whilst the simulation to real gap has been studied in-depth across domains such as robotics to date there has been minimal research considering generalisability in the context of cyber defence agents and how differences in observation space could enhance agent generalisability when placed into environments that differ from the training environment. Within this paper, we propose a method of enhancing agent generalisability and performance within unseen environments by integrating a graph embedded network representation into the agents observation space. We then compare agent performance with and without a graph embedded network representation based observation space within a series of randomised cyber defence simulations. We find that there is a trade off between the effectiveness of the graph embedding representation and the complexity of the graph, in terms of node count and number of edges.

It is our position that when hypergraphs are used to capture multi-way local relations of data, their resulting topological features describe global behavior. These features capture complex correlations that can then serve as high fidelity inputs to autoencoder-driven anomaly detection pipelines. We propose two such potential pipelines for cybersecurity data, one that uses an autoencoder directly to determine network intrusions, and one that de-noises input data for a persistent homology system, PHANTOM. We provide heuristic justification for the use of the methods described therein for an intrusion detection pipeline for cyber data. We conclude by showing a small example over synthetic cyber attack data.

Perceptual hashes map images with identical semantic content to the same $n$-bit hash value, while mapping semantically-different images to different hashes. These algorithms carry important applications in cybersecurity such as copyright infringement detection, content fingerprinting, and surveillance. Apple's NeuralHash is one such system that aims to detect the presence of illegal content on users' devices without compromising consumer privacy. We make the surprising discovery that NeuralHash is approximately linear, which inspires the development of novel black-box attacks that can (i) evade detection of "illegal" images, (ii) generate near-collisions, and (iii) leak information about hashed images, all without access to model parameters. These vulnerabilities pose serious threats to NeuralHash's security goals; to address them, we propose a simple fix using classical cryptographic standards.

Self-play reinforcement learning has achieved state-of-the-art, and often superhuman, performance in a variety of zero-sum games. Yet prior work has found that policies that are highly capable against regular opponents can fail catastrophically against adversarial policies: an opponent trained explicitly against the victim. Prior defenses using adversarial training were able to make the victim robust to a specific adversary, but the victim remained vulnerable to new adversaries. We conjecture this limitation was due to insufficient diversity of adversaries seen during training. We propose a defense using population-based training to pit the victim against a range of opponents. We evaluate this defense's robustness against new adversaries in two low-dimensional environments. We find that our defense significantly increases robustness against adversaries in both environments and show that robustness is correlated with the size of the opponent population.

We demonstrate how machine learning could serve cyber knowledge-base curators, threat hunters, and security analysts. We present a a machine learning~(ML) based workflow that addresses the overwhelming quantity of text entries that would have to be read and assimilated by hunters and analysts in order to infer a plausible relationship between two entries from different threat, vulnerability, and mitigation sources. The workflow uses language embedding models and classifiers to automatically label exhaustively-collected pairs of entries as linked or not. It also includes humans who in a curator role guide how many pairs are labeled as linked, ``candidates''. The curator also ranks the candidates; experts participate by independently and manually categorically assessing the ML-derived candidates; and the curator reprises by specifying rules that state how the collective expert categorizations determine a final label for each candidate pair.

Cyber vulnerability management is a critical function performed by a cybersecurity operations center (CSOC) that helps protect organizations against cyber-attacks on their computer and network systems. Adversaries hold an asymmetric advantage over the CSOC, as the number of deficiencies in these systems is increasing at a significantly higher rate compared to the rate at which the security teams are expanding to mitigate them in a resource-constrained CSOC environment. The current approaches employed at the CSOCs and recently published in the literature are deterministic and one-time decision-making methods, which do not consider future uncertainties when prioritizing and selecting vulnerabilities for mitigation. In addition, these approaches are constrained by the sub-optimal distribution of resources, providing no flexibility to the security team to adjust their response to fluctuations in vulnerability arrivals and thereby, weakening their security posture. We propose a novel artificial intelligence-enabled framework, Deep VULMAN, which consists of a deep reinforcement learning agent and an integer programming method to fill this gap in the cyber vulnerability management process. Our sequential decision-making framework, first, determines the near-optimal amount of resources to be allocated for mitigation under uncertainty, given an observed state of the system, and then determines the optimal set of prioritized vulnerability instances selected for mitigation. Our proposed framework outperforms the current methods in prioritizing the selection of important organization-specific vulnerabilities on real-world vulnerability data observed over a one-year period.