Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Hunting

As cyber threats continue to grow in scale and sophistication, blue team defenders increasingly require advanced tools to proactively detect and mitigate risks. Large Language Models (LLMs) offer promising capabilities for enhancing threat analysis. However, their effectiveness in real-world blue team threat-hunting scenarios remains insufficiently explored. This paper presents CYBERTEAM, a benchmark designed to guide LLMs in blue teaming practice. CYBERTEAM constructs a standardized workflow in two stages. First, it models realistic threat-hunting workflows by capturing the dependencies among analytical tasks from threat attribution to incident response. Next, each task is addressed through a set of operational modules tailored to its specific analytical requirements. This transforms threat hunting into a structured sequence of reasoning steps, with each step grounded in a discrete operation and ordered according to task-specific dependencies. Guided by this framework, LLMs are directed to perform threat-hunting tasks through modularized steps. Overall, CYBERTEAM integrates 30 tasks and 9 operational modules to guide LLMs through standardized threat analysis. We evaluate both leading LLMs and state-of-the-art cybersecurity agents, comparing CYBERTEAM against open-ended reasoning strategies. Our results highlight the improvements enabled by standardized design, while also revealing the limitations of open-ended reasoning in real-world threat hunting.

Recently, AI chatbots powered by large language models (like the ones behind ChatGPT or Claude) have shown promise as helpers for security work. But most research so far has only tested them on small, isolated tasks, like identifying a single piece of malware. Real threat hunting isn't like that. It's a chain of connected steps, where the answer to one question shapes the next. You can't recommend how to fix a break-in until you understand how the intruder got in. We built CyberTeam, a testing ground that mirrors how real blue teams actually work. It does two things. First, it lays out threat hunting as a connected sequence of 30 tasks, from figuring out who the attacker is, to understanding their behavior, to ranking how serious the threat is, to suggesting fixes. Second, instead of just asking the AI to "figure it out," we give it a toolkit of nine focused operations, i.e., things like pulling out key names and addresses from a report, looking up relevant security advisories, or summarizing long incident logs. The AI decides which tools to use at each step, like a detective choosing the right method for each clue. We tested leading AI models using this structured approach and compared it to the usual way of just asking the AI to think things through on its own. The structured approach worked noticeably better, especially on complex tasks where one step depends on another. It also made the AI's reasoning easier for human analysts to check and trust. We also found that AI models held up reasonably well when reports contained small typos, but struggled when the meaning of the text was subtly twisted, which suggests that accurate, careful threat reports really matter. Our hope is that CyberTeam helps researchers and security teams build AI assistants that are more reliable, transparent, and genuinely useful for defending against the growing tide of cyberattacks.