Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All

Repairing system crashes discovered by kernel fuzzers like Syzkaller is a critical yet underexplored challenge in software engineering. While recent works have introduced Large Language Model (LLM) based agents for Linux kernel crash-resolution, their evaluation benchmarks are usually static and thus, do not capture the evolving nature of the Linux kernel, and suffer from potential data contamination due to LLM knowledge cutoffs. To address the above problem, we present (i) Live-kBench, an evaluation framework for self-evolving benchmarks that continuously scrapes and evaluates agents on freshly discovered kernel bugs, and (ii) kEnv, an agent-agnostic standardized crash-resolution environment for kernel compilation, execution, and feedback. This design decouples agent workflows from heavy-weight execution, enabling fair and scalable comparison across diverse agent frameworks under identical conditions. To this end, we curate an inaugural dataset of 534 Linux kernel bugs and empirically demonstrate a significant performance gap, with agents achieving up to 25% higher equivalent patch rate on bugs fixed before the LLM knowledge cutoff. Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt (plausible patches); however only ~20% of generated patches closely match developer fixes. Additionally, exposing crash resolution feedback improves crash resolution rate by 29%. Live-kBench provides the community with an evaluation infrastructure for self-evolving benchmarks that is both time and attribute sensitive; complete with a public dashboard to track agent progress on Linux kernel bugs.

The Linux kernel is the software foundation that powers most of the world's servers, phones, and cloud infrastructure. Automated tools called "fuzzers" constantly discover crashes in the kernel — thousands to date — but fixing them still largely depends on human developers, and each fix can take days of expert effort. Recently, AI coding agents have shown promise in automatically repairing such crashes, but evaluating them fairly is difficult: existing test sets are static and don't reflect the kernel's constant evolution, while older fixes risk leaking into AI training data. We built Live-kBench, a benchmark that continuously collects fresh kernel crashes from public bug trackers and tests AI agents on bugs they could not have seen before. We also created kEnv, a standardized testing environment that lets any AI agent attempt repairs under identical conditions. Using 534 real kernel bugs, we found a significant performance gap: AI agents score up to 25% better on older bugs than on newer ones. While current agents can suppress about 74% of crashes, only around 20% of their fixes truly match what a human developer would write. Live-kBench provides the community with a continuously updated, fair way to track real progress.