Beyond Text-to-SQL: Can LLMs Really Debug Enterprise ETL SQL?

SQL is central to enterprise data engineering, yet generating fully correct SQL code in a single attempt remains difficult—even for experienced developers and advanced Text-to-SQL LLMs—often requiring multiple debugging iterations. We introduce Squirrel Benchmark, the first benchmark for enterprise-level SQL reasoning and debugging. Our benchmark is built upon two key innovations: (1) an automated construction workflow that employs reverse engineering to systematically inject realistic bugs into large-scale SQL code, enabling scalable and diverse benchmark generation; and (2) an execution-free evaluation framework tailored for enterprise settings, providing fast, accurate, and resource-efficient assessment. Squirrel comprises 469 Squirrel-Syntax queries featuring syntax errors with explicit error messages, and 516 Squirrel-Semantic queries targeting semantic errors where code fails to meet user intent. The queries are highly complex, averaging over 140 lines and featuring deep, wide abstract syntax trees (average width >11, depth >8.7). Evaluation of nearly 30 LLMs reveals a substantial performance gap: the best-performing model, Claude-4-Sonnet, achieves only 36.46% accuracy on Squirrel-Syntax and 32.17% on Squirrel-Semantic, while most models score below 20%. We further explore four solution strategies, identify key challenges, and outline promising directions for enterprise SQL debugging with LLMs.

We introduced the Squirrel Benchmark, the first large-scale, enterprise-level dataset for evaluating Large Language Models' (LLMs) ability to debug complex ETL SQL, finding that even state-of-the-art models like Claude-4-Sonnet achieve only 32-36% Graph Match scores, highlighting a substantial gap in current LLM capabilities for real-world data engineering tasks.