{"id":"d39c6935-b419-47d6-aa54-b2bdc16c2d7e","arxiv_id":"2601.18119","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On 985 synthesized 140+-line enterprise ETL SQL debugging tasks, the best of ~30 LLMs (Claude-4-Sonnet) scores 36.5% on syntax and 32.2% on semantic repair, and most models stay below 20%.","lead":"This paper introduces Squirrel, a benchmark of 985 long, enterprise-style SQL debugging tasks, and finds that even the best LLM tested repairs only about a third of them. It also contributes a reverse-engineering pipeline for injecting realistic SQL bugs and an execution-free evaluation method for judging repairs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GM metric is unvalidated: all headline accuracy numbers rest on exact graph isomorphism over Calcite plans, with no check against executing repairs on real Hive/Spark data.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern: the execution-free Graph Match metric is unvalidated. I agree with that diagnosis. Every headline result in the paper is a GM number. EM is explicitly a strict lower bound, and MB only measures whether a prediction moves closer to the reference; neither supports the central claim of model failure rates. Only GM supports '36.46% / 32.17% / most models below 20%.' The construction pipeline — seed SQL from production logs, LLM-based scenario transfer, taxonomy-driven minimal bug injection, attack-defense filtering — is described in enough detail to be credible, and the qualitative finding that long nested ETL SQL is hard for LLMs is plausible and directionally consistent across many models. But the quantitative claim is only as strong as the metric. The paper asserts a 'strong correlation with real-world debugging outcomes' without reporting any execution-based agreement study, and Appendix F explicitly acknowledges reliance on reference solutions as a limitation. The proposed test would settle the concern: if GM agrees with execution on a stratified sample, the benchmark's central claim is supported; if not, the numbers and possibly the rankings are unreliable. Other issues noted by the reader — benchmark name inconsistency, the 33.17 vs 32.17 discrepancy, the contradicted 'consistently lower' claim, and unavailable code/data — are real but secondary and easily fixed. Because the core concern is addressable and not fatal to the benchmark's usefulness, the verdict should remain CONDITIONAL, requiring GM validation and correction of the internal inconsistencies before the headline numbers are trusted.","tokens_in":27056,"tokens_out":4391,"duration_ms":52051,"concrete_test":"Sample ~100 syntax and ~100 semantic tasks with their reference and model predictions from at least three models spanning the performance range (e.g., Claude-4-Sonnet, DeepSeek-V3, Qwen-2.5-Coder-7B). For each sampled task, execute buggy SQL, reference SQL, and predicted SQL on a real Spark/Hive engine against the provided test schemas and small but discriminating datasets; judge correctness by executability (syntax tasks) and by order-insensitive result-set equality between predicted and reference outputs across multiple random test databases (semantic tasks). Compare these execution-based labels to the GM labels, computing agreement, false-positive rate, and false-negative rate, then recompute GM scores on the sample. If GM disagrees with execution on more than 5% of semantic cases, or if Claude-4-Sonnet's sample GM shifts by more than ~3 points, the Table 2 numbers and the 'most model","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline results (36.46% / 32.17% GM, most models below 20%) are produced entirely by Graph Match, defined as exact graph isomorphism over Apache Calcite's optimized logical plan (Eq. 6, Appendix D.1.2). For the central claim to hold, GM must be a trustworthy oracle for Hive/Spark ETL repairs: every semantically correct repair must map to an isomorphic plan, and every incorrect repair to a non-isomorphic plan. That is a strong requirement. Calcite's rule-based optimizer is a conservative approximation of SQL equivalence; it does not canonicalize all equivalent rewrites, and it is especially untested for the dialect constructs this benchmark is built on (LATERAL VIEW / explode, collect_set, INSERT OVERWRITE ... PARTITION, Hive functions like conv/md5/substr). Because each task is scored against a single reference plan, a valid repair that removes duplicates or restructures a join differently can be scored 0 (false negative), while a minimal edit that keeps plan shape but changes semantics (e.g., wrong constant, wrong ordering) can be scored 1 (false positive). The paper asserts 'strong correlation with real-world debugging outcomes' (Sec. 1, 7) but reports no comparison of GM against executing repaired SQL on test data. Appendix F concedes the framework 'is inherently limited by its reliance on reference solutions.' Until GM is validated, the benchmark's main empirical conclusion is unsupported — not because the conclusion is implausible, but because the yardstick's accuracy is unmeasured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Squirrel Benchmark, a set of 469 syntax-debugging and 516 semantic-debugging tasks built from synthetic Hive/Spark ETL scripts of 140+ lines, together with an execution-free evaluation framework based on Exact Match, Graph Match (graph isomorphism over Apache Calcite optimized plans), and Modify Better scores. The authors evaluate roughly 30 LLMs, report that Claude-4-Sonnet achieves only 36.46% GM on Squirrel-Syntax and 32.17% GM on Squirrel-Semantic, and present three SFT baselines plus an agentic baseline. The paper claims this is the first enterprise-level SQL debugging benchmark and that the benchmark correlates strongly with real-world debugging outcomes.","tokens_in":27326,"tokens_out":2989,"duration_ms":34012,"significance":"If the benchmark and metric are valid, this is a useful contribution to an underserved area: enterprise ETL SQL debugging is qualitatively different from short text-to-SQL generation, and a reproducible, complex, dialect-specific benchmark with fine-grained bug taxonomy would be valuable to the community. The automatic construction pipeline and the three-level taxonomy derived from 268 production-log errors are concrete assets, and the broad model comparison provides a starting point. However, the headline empirical claim — that even state-of-the-art models mostly fail — rests entirely on the Graph Match metric, which is not validated against execution, and on a construction loop in which the same model family generates, corrupts, and filters the tasks. Those two load-bearing supports must be strengthened before the benchmark can serve as a reliable reference.","major_comments":[{"comment":"The central correctness metric GM is defined as exact graph isomorphism over Apache Calcite optimized logical plans, but no evidence is provided that this is a sound oracle for Hive/Spark ETL repairs. Calcite's rule-based optimizer is a conservative approximation of SQL equivalence, and it is not established that it canonicalizes the dialect constructs used here (LATERAL VIEW/explode, collect_set, INSERT OVERWRITE ... PARTITION, Hive functions). A valid repair that restructures a join or removes duplicates may produce a non-isomorphic plan and be scored 0, while a minimal edit that changes semantics but preserves plan shape may be scored 1. Appendix F concedes the framework 'is inherently limited by its reliance on reference solutions.' Because all headline numbers (36.46%, 32.17%, 'most models below 20%') are GM scores, the main empirical conclusion is unsupported until GM is calibrated","section":"Appendix D.1.2, Eq. (6); Appendix F"},{"comment":"The benchmark construction loop is partly self-referential. Seed SQL, solution SQL, bug injection, and semantic task descriptions are all generated by Claude-4-Sonnet at temperature 0; the 'attack–defense' step in §3.4 then discards instances that a majority of evaluated models can already solve, retaining the hard tail. This does not measure a naturally occurring difficulty distribution; it measures performance on a set adversarially selected to separate models. It also creates a potential in-family advantage for Claude-4-Sonnet, since the reference answers and the task phrasing come from that model family. The paper should quantify how many instances were discarded at each stage, report results before and after filtering, and ideally validate the benchmark on an independently authored or execution-verified subset to show that the retained tasks are representative rather than synthetic","section":"§3.3, §3.4, §5.1"},{"comment":"The claim that 'performance on Squirrel-Semantic is consistently lower than on Squirrel-Syntax' is directly contradicted by Table 2. Gemini-2.5-Pro has GM 21.54 on Squirrel-Syntax and 23.06 on Squirrel-Semantic; O3-mini has GM 19.83 on Squirrel-Syntax and 28.68 on Squirrel-Semantic. Several other models (e.g., Qwen-2.5-Coder-32B) also have higher semantic GM than syntactic GM. The sentence should be revised to describe the observed trend rather than a universal ordering, or the discrepancy should be explained.","section":"§5.1, Table 2"},{"comment":"The paper repeatedly describes the validation pipeline as 'execution-based,' but Appendix D.1.3 states explicitly that no SQL is run against a live engine and that TQS is a static-analysis tool. Equation (4) claims 'guaranteed execution correctness' from a check that only validates syntax and schema-level semantics. This overstates what the construction pipeline establishes: a query can pass TQS while returning incorrect results on data. The terminology should be corrected, and the guarantee in Eq. (4) softened to 'passes static validation,' otherwise readers will overestimate the reliability of the ground-truth references that the entire benchmark inherits.","section":"§3.1, Eq. (4); Appendix D.1.3"}],"minor_comments":[{"comment":"The abstract refers to the benchmark as 'OurBench' while the body uses 'Squirrel Benchmark'; also the abstract and §1 report Claude-4-Sonnet's Squirrel-Semantic score as both 32.17% (abstract, Table 2) and 33.17% (§1). Please reconcile.","section":"Abstract and §1"},{"comment":"Rows are mislabeled 'Squrriel-Syntax' and 'Squrriel-Semantic' (missing 'r').","section":"Table 1"},{"comment":"The complexity threshold τ and weights α, β are never specified. A sensitivity analysis or at least the chosen values should be reported, since the threshold controls which seed SQL enters the corpus.","section":"Eq. (2), §3.1"},{"comment":"'Majority of models succeed' and 'only a few models succeed' are not quantified; the number of models used in the attack and the counts of discarded/retained instances should be given for reproducibility.","section":"§3.4"},{"comment":"The phrase 'strictly non-executive' is confusing; consider replacing with 'static analysis' or 'non-executing.'","section":"Appendix D.1.3"}],"recommendation":"major_revision","confidential_remarks":"The central direction is plausible and the dataset, once released and validated, could be influential. However, the paper's strongest empirical claims rest on an unvalidated graph-isomorphism oracle and a partly circular construction loop. Both issues are addressable with additional experiments and disclosure (execution-based validation on a sample, filtering counts, pre/post attack–defense results, and an independent subset), so I would not reject the paper outright, but the current version is not yet ready for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution with one load-bearing soft spot. The benchmark is new in scale and construction: 985 tasks averaging over 140 lines in Hive/Spark dialect, a three-level bug taxonomy from 268 production logs, a reverse-engineering pipeline for minimal bug injection, and a 30-model evaluation showing most models under 20% GM. The pipeline is described well enough to be rebuilt, and the SFT and agent baselines are a useful addition. The paper deserves a serious referee.\n\nThe central direction—LLMs are weak at long, nested enterprise ETL debugging—is plausible and survives scrutiny. The numbers are directionally consistent across model families. But the yardstick is unvalidated. The GM score is exact graph isomorphism over Calcite-optimized logical plans, scored against a single reference per task. The paper claims it is 'accurate' and 'strongly correlated with real-world debugging outcomes' without reporting a single comparison against actually executing repaired SQL on test data. That matters because Calcite's optimizer is a conservative approximation of SQL equivalence, and the benchmark is built on dialect constructs (LATERAL VIEW/explode, collect_set, INSERT OVERWRITE ... PARTITION) where canonicalization is least tested. A valid repair that changes plan shape can be scored 0, and a minimal edit that keeps plan shape but changes semantics can be scored 1. Appendix F concedes reliance on reference solutions, but the paper never quantifies the risk. This is addressable—validate on a sample against execution and report agreement—but until then the headline 36.46% and 'most models below 20%' are not fully supported.\n\nThere are also smaller inconsistencies that should have been caught: the abstract names the benchmark 'OurBench' while the body says 'Squirrel'; the introduction gives Claude 33.17% on Squirrel-Semantic while Table 2 and the abstract say 32.17%; and §5.1's claim that semantic performance is 'consistently lower' across all models is contradicted by Table 2 for Gemini-2.5-Pro (23.06 vs 21.54) and O3-mini (28.68 vs 19.83). The attack–defense filtering also removes easy instances, so reporting pre-filter numbers would make the difficulty claim more transparent.\n\nThe benchmark, code, and TQS are not public, so none of this can be independently checked today. That is a real limitation, not a fatal one.\n\nBottom line: the research direction is sound, the pipeline is buildable, and the paper is honest about its main risks. The metric validation is the gating issue. I'd send it to peer review and tell the authors to validate GM against execution and release the data before the numbers are treated as established.","headline":"A genuinely useful benchmark for enterprise SQL debugging, but the headline numbers rest on an unvalidated execution-free equivalence oracle; worth reviewing seriously, with the metric validation as the gating issue.","tokens_in":28005,"tokens_out":1706,"would_cite":false,"duration_ms":17888,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Enterprise SQL debugging is unsolved for LLMs: on a new benchmark of 985 long, real-world-style ETL repair tasks, the best model fixes 36% of syntax errors and 32% of semantic errors, and most models stay under 20%.","keywords":["SQL debugging","ETL","LLM evaluation","benchmark construction","syntax errors","semantic errors","graph isomorphism","Hive/Spark SQL"],"falsifier":"Take a random sample of Squirrel-Semantic tasks, execute the reference and the best model's repairs on small test tables seeded with edge-case rows (NULLs, duplicate keys, no-match joins), and measure the agreement between Graph Match's correct/incorrect decision and actual output equivalence; any material disagreement rate invalidates the reported percentages.","tokens_in":26765,"feed_emoji":"🐿️","tokens_out":4532,"duration_ms":50964,"temperature":0.7,"pith_summary":"Enterprise SQL debugging is more than an extension of text-to-SQL. This paper introduces Squirrel Benchmark, a set of 985 repair tasks built from production-style ETL scripts averaging more than 140 lines, with an automated pipeline that injects realistic bugs and an execution-free scoring method based on structural graph equivalence. The paper's central finding is that current LLMs largely fail at this task: the best model, Claude-4-Sonnet, repairs 36.46% of syntax-bugged queries and 32.17% of semantic-bugged queries, while most of the nearly 30 tested models fall below 20%. A sympathetic reader would take this as evidence that robust, multi-step SQL debugging in enterprise settings is an open capability gap that current benchmarks were not measuring.","feed_headline":"Best LLM fixes only 36% of enterprise SQL bugs","feed_subtitle":"New 985-task benchmark shows most models stay under 20% on long, nested ETL scripts.","key_machinery":"Two mechanisms carry the argument. First, reverse-engineering bug construction: correct SQL seeds are sampled from production logs, a taxonomy built from 268 real error cases guides minimal-change bug injection, and an attack–defense filtering pass (multiple models attempt the tasks; hard cases get human review) sets final difficulty. Second, execution-free evaluation: instead of running repaired SQL on test data, the paper compiles predicted and reference queries into canonical logical plans and scores a repair correct when the two graphs are isomorphic (Graph Match), alongside exact string match and edit-distance progress. The graph isomorphism check is the load-bearing component: it deter","core_discovery":"The paper claims that LLMs currently cannot reliably debug long enterprise ETL SQL, and that this failure is unseen by standard text-to-SQL benchmarks. It constructs a benchmark where each task is a 140+-line Hive/Spark SQL script with either an explicit syntax error or a silent semantic error, and where a repair is judged correct only if its logical query plan is isomorphic to the reference plan. Across nearly 30 models, including reasoning and SQL-specialized systems, the best score is 36.46% (syntax) and 32.17% (semantic), with most models below 20%; code-generation and text-to-SQL specialists drop to single digits. The paper also reports that tuned fine-tuning and an agentic loop improve","pith_inferences":["The construction pipeline is dialect-agnostic, so the same reverse-engineering workflow could produce sibling benchmarks for other SQL dialects; if scores stay low, the difficulty is general, not an artifact of Hive/Spark.","The single-reference Graph Match metric may penalize valid alternative repairs in the semantic tasks; building a small execution-validated subset would show whether models are failing or the tester is.","If execution-free graph scoring is validated, the same evaluation pattern could extend to other expensive-to-run code-repair domains (shell pipelines, data-prep scripts), where running code in production is costly or unsafe.","Because the benchmark's construction and much of its evaluation relied on one model family, a head-to-head with benchmarks built by other generators would reveal whether the reported ceiling is a model-family blind spot."],"forward_implications":["If the benchmark numbers hold, LLM-based SQL debugging tools cannot be trusted for unattended repair of enterprise ETL scripts; human review in the loop remains necessary.","Both syntax and semantic repair rates stay below 40% even for the strongest model, so gains from the next generation of reasoning models should show up first as higher Graph Match on these 985 tasks.","The finding that a 7B model fine-tuned on minimal-edit debugging pairs beats a 32B base model (33.18% vs 20.26% Graph Match on syntax) points to synthetic bug-injection data as a practical lever before scale.","Reasoning models' high Modify-Better scores indicate they move scripts closer to the fix but rarely land it in one attempt, suggesting iterative agent loops, rather than single-shot generation, are the right interface for SQL repair."],"fun_headline_variants":["LLMs top out at 36% on long SQL debugging","Benchmark: LLMs debug long SQL at only 36%","Enterprise SQL bug fixing: LLMs hit 36% ceiling","Most LLMs under 20% on tough SQL debugging tasks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's headline numbers assume the execution-free Graph Match test, which compares normalized query plans by graph isomorphism, correctly decides semantic correctness without ever running the repaired SQL on data that would expose the bug.","fun_headline_variants_meta":{"raw":{"variants":["LLMs top out at 36% on long SQL debugging","Benchmark: LLMs debug long SQL at only 36%","Enterprise SQL bug fixing: LLMs hit 36% ceiling","Most LLMs under 20% on tough SQL debugging tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000631,"raw_usage":{"total_tokens":2764,"prompt_tokens":769,"completion_tokens":1995,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":1935}},"tokens_in":513,"tokens_out":1995,"duration_ms":16063,"temperature":1.0,"reasoning_tokens":1935,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T08:07:23.594889+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of Squirrel-Semantic tasks, execute the reference and the best model's repairs on small test tables seeded with edge-case rows (NULLs, duplicate keys, no-match joins), and measure the agreement between Graph Match's correct/incorrect decision and actual output equivalence; any material disagreement rate invalidates the reported percentages.","supporting_citations":[],"review_version":1}