Ekka: Automated Diagnosis of Silent Errors in LLM Inference

LLM serving frameworks are quickly evolving with a complex software stack and a vast number of optimizations. The rapid development process can introduce silent errors where output quality silently degrades without any explicit error signals. Diagnosing silent errors is notoriously difficult due to the substantial semantic gap between the high-level symptoms and the low-level root causes. We observe that diagnosis of silent errors can be effectively framed as a differential debugging problem by leveraging the existence of semantically correct reference implementations. We propose Ekka, an automated diagnosis system that identifies root causes by systematically aligning and comparing intermediate execution states between a target and a reference framework. We constructed a benchmark of real-world silent errors from popular serving frameworks, where Ekka shows 80\% pass@$1$ diagnosis accuracy and 88\% pass@$5$ diagnosis accuracy, outperforming state-of-the-art systems. Ekka also diagnoses 4 new silent errors from serving frameworks, all of which have been confirmed by the developers.

Large language models are often deployed using optimized serving systems that make them faster and cheaper to run. However, these systems are complex, and sometimes they introduce silent errors: the model still produces an answer, but the answer becomes worse, inconsistent, or completely wrong, without any crash or warning. These silent errors are especially hard to debug because the visible symptom may be a bad final answer, while the real cause may be buried deep inside the model implementation or a low-level computation kernel. We built Ekka, an automated debugging system for finding the source of these silent errors. Ekka compares a buggy serving system with a trusted reference implementation, such as HuggingFace. Instead of only checking the final answer, it compares the model’s internal computation step by step, aligns equivalent parts across different systems, and identifies where the computation first starts to go wrong. On real bugs from popular serving systems, Ekka diagnoses the cause more accurately than state-of-the-art coding agents, while costing about $30 per case. It also diagnosed four new silent errors that were confirmed by developers. This can help make large language model deployment more reliable.