ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation

We present ExCyTIn-Bench, the first benchmark to Evaluate an LLM agent X on the task of Cyber Threat Investigation through security questions derived from investigation graphs. Real‑world security analysts must sift through a large number of heterogeneous security logs, follow multi‑hop chains of evidence to investigate threats. With the developments of LLMs, building LLM-based agents for automatic threat investigation is a promising direction. We construct a benchmark from a controlled Azure tenant including a SQL environment covering 57 log tables from Microsoft Sentinel and related services, and 7542 generated questions. We leverage security logs extracted with expert-crafted detection logic to build threat investigation graphs, and then generate questions with LLMs using paired nodes on the graph, taking the start node as background context and the end node as answer. Anchoring each question to these explicit nodes and edges not only provides automatic, explainable ground truth answers but also makes the pipeline reusable and readily extensible to new logs. Our comprehensive experiments on the test set with different models confirm the difficulty of the task: the best model so far can achieve a reward of 0.606, leaving much headroom for future research.

Cybersecurity analysts often need to investigate possible attacks by searching through many security logs, connecting clues across different systems, and deciding what happened. This paper introduces ExCyTIn-Bench, a new test for measuring how well AI assistants can perform this kind of cyber threat investigation. The benchmark is built from a realistic Microsoft Azure security environment with many types of logs and several simulated attack scenarios based on real-world incidents. Instead of asking simple cybersecurity knowledge questions, ExCyTIn-Bench gives an AI assistant a security question and requires it to search the log database, follow chains of evidence, and produce the correct answer. The questions are generated from investigation graphs, which makes the answers explainable and allows the benchmark to give partial credit when an assistant finds useful intermediate clues. Experiments with many current AI models show that the task is still difficult: even the best tested model leaves substantial room for improvement. Overall, ExCyTIn-Bench provides a reusable way to evaluate and improve AI assistants for realistic cyber-defense work.