{"id":"639677eb-fd75-4b59-a1f1-54f6e43e3b02","arxiv_id":"2505.18744","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors built a 4,038-question text-to-SQL benchmark with chain-of-thought annotations and found that state-of-the-art models reach only 33.20% execution accuracy.","lead":"LogicCat is a new benchmark that asks AI systems to write SQL queries from natural language questions requiring math, physics, common sense, and hypothetical reasoning. It tests whether large language models can chain multiple reasoning steps into a correct database query.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold SQL semantic correctness is not established; Table 1's 95.05% VES conflicts with the claim that all gold queries are executable, so the reported 33.20% EX may not be a valid measure of reasoning.","rationale":"The reader's weakest assumption is the most load-bearing condition for the central claim. The paper's headline difficulty numbers are computed by comparing model-generated SQL results against gold SQL results, so if a meaningful fraction of gold queries are semantically wrong, the 33.20% figure does not measure reasoning. The internal VES contradiction in Section 3.1 strengthens this concern beyond a generic quality worry. I considered the alternative concern that the prompt template in Figure 4 supplies the required formulas, which would undermine the 'physical knowledge' claim; while plausible, this affects the interpretation of what is being tested rather than the validity of the reported accuracies, so it is secondary. The reader's conditional verdict remains appropriate: the dataset is likely valuable and the direction of the difficulty claim is probably correct, but the paper must either provide semantic verification of the gold SQL or temper the claim until it is supplied.","tokens_in":14997,"tokens_out":9033,"duration_ms":80427,"concrete_test":"Conduct a semantic audit on a random sample of 100 LogicCat questions stratified across the four reasoning categories: two SQL experts, blind to the released gold and to each other, independently write SQL from the question and schema only; execute the released gold and both expert queries and compare result sets. If the released gold's result differs from both experts on more than 10% of samples, semantic correctness is not established and EX should be recomputed on a verified subset. Before this, run the released evaluation script on all 4,038 gold queries to resolve the contradiction between the 'all queries executable' claim and the 95.05% VES in Table 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The experimental claim that LogicCat caps state-of-the-art models at 33.20% execution accuracy depends on the 4,038 gold SQL queries being correct answers to their questions. That is not established. Section 3.1 states that 'automated scripts confirmed the executability of all queries,' but Table 1 reports only 95.05% VES after revision; if VES is computed on the gold queries, about 5% are not valid executable SQL, a direct internal contradiction. Moreover, the Table 1 caption defines VES as 'syntactically correct SQL, regardless of result correctness,' so even 100% VES would not certify that a gold query returns the answer to the intended question. The described expert review in Section 3.1 is qualitative, and no semantic agreement metric is reported. Because EX is defined as matching the gold result (Section 4.3), any semantically wrong gold query makes the reported upper bound uninterpretable as a measure of model reasoning. A secondary concern is that Figure 4's caption states the prompt provides 'a high-level reasoning plan including the necessary formula'; if the prompt indeed supplies formulas, the Physical Knowledge category tests formula application rather than knowledge retrieval, changing the benchmark's difficulty interpretation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LogicCat, a new text-to-SQL benchmark containing 4,038 English questions paired with chain-of-thought annotations across 45 databases, organized around four reasoning categories: physical knowledge, mathematical logic, commonsense, and hypothetical reasoning. The authors describe a multi-stage construction and quality-control process with expert review, report dataset complexity statistics, and evaluate a wide range of general-purpose, coder, and specialized text-to-SQL models in a zero-shot setting. Their headline result is that the best method, Chess-SQL with GPT-4o, reaches only 33.20% execution accuracy, and that removing chain-of-thought annotations degrades performance substantially across all tested models.","tokens_in":15158,"tokens_out":3370,"duration_ms":28748,"significance":"If the gold SQL queries are semantically correct and the evaluation protocol is free of leakage, LogicCat would be a useful and timely resource: it targets a genuine gap in text-to-SQL evaluation, provides reasoning-category breakdowns, includes a large annotation effort, and ships with released code and data. The broad model coverage and the chain-of-thought ablation are strengths, as is the explicit reporting of per-category and per-difficulty results. However, the central difficulty claim depends on the correctness of the gold SQL and on a fair comparison to existing benchmarks; both are currently not established to the standard needed for a benchmark paper.","major_comments":[{"comment":"The quality-control narrative is internally inconsistent: the text states that 'automated scripts confirmed the executability of all queries,' yet Table 1 reports a post-revision VES of only 95.05%, and the table caption defines VES as 'syntactically correct SQL, regardless of result correctness.' Since Section 4.3 defines EX as matching the gold result, any gold query that is not semantically correct—not merely non-executable—makes the reported 33.20% upper bound uninterpretable as a measure of model reasoning. The paper must either report a semantic-agreement metric (e.g., independent gold SQL generation with execution-match agreement) or otherwise demonstrate that the gold queries answer their questions, not merely that they parse.","section":"Section 3.1, Table 1"},{"comment":"The prompt description is ambiguous and potentially leaks the answer structure: the Figure 4 caption says the template provides 'a high-level reasoning plan including the necessary formula,' and the shown assistant output contains both the SQL and stepwise reasoning. If the formula or the chain-of-thought steps are supplied in the model input during the zero-shot evaluation, then the Physical and Mathematical categories test formula application and prompt following rather than knowledge retrieval or自主 reasoning, which materially changes the difficulty interpretation. The authors should specify exactly what is in the model input for both the with-CoT and without-CoT conditions, and if formulas are given, add an unassisted evaluation where the model must retrieve the formula itself.","section":"Figure 4, Section 4.1"},{"comment":"The CoT ablation contains an unexplained discrepancy: Table 5 lists 'Deepseek-Coder' with 4.74 EX without CoT and 16.11 EX with CoT, but Table 4 reports Deepseek-Coder-V2 at 23.70 overall EX in what appears to be the with-CoT setting. If these are different models or different evaluation conditions, the table should say so; if they are the same, the numbers contradict each other. This inconsistency undermines the specific claim that CoT produces a 16.11-point gain for this model.","section":"Section 5.4, Table 5 vs Table 4"},{"comment":"The claim that LogicCat 'substantially increases the task difficulty' relative to Spider, BIRD, and Archer is not supported by controlled evidence. Table 3 reports aggregate dataset statistics, but no same-model, same-prompt baseline is provided on Archer or Spider under identical prompting and evaluation conditions. Because the benchmark was constructed with explicit instructions to include arithmetic, physics, and hypothetical reasoning, the observed low accuracy is partly by construction. Adding a small controlled comparison on Archer and Spider with the same models and prompts would let the reader separate benchmark difficulty from dataset-content effects.","section":"Section 1, Section 5.1, Table 3"}],"minor_comments":[{"comment":"The number of chain-of-thought annotations is given as 12,114 in the Abstract and 12,144 in the contributions list; these should be reconciled.","section":"Abstract and Section 1"},{"comment":"The annotation effort is reported as '800-person-hour' and later as 'approximately 500 person-hours' plus 200 person-hours for expert review; the total should be stated consistently.","section":"Section 3.1"},{"comment":"The text refers to 'the left chart in Figure 1' for per-category performance, but Figure 1 is the benchmark overview; the intended reference is likely Figure 5 or Table 4.","section":"Section 5.2, Figure 5"},{"comment":"There are typographical errors in the caption ('broekn' instead of 'broken') and in Section 5.4 ('wtih' instead of 'with', 'Din-SQL' for DIN-SQL); a careful proofreading pass is needed.","section":"Table 4 caption"},{"comment":"The model label 'Deepseek-Coder' is ambiguous; use the exact identifier from Table 4 (Deepseek-Coder-V2) to avoid confusion.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The benchmark could be a valuable community resource if the gold SQL semantic correctness is rigorously demonstrated and the prompting condition is clarified. The internal consistency issues in Section 3.1 and Table 5 are fixable, but they currently block acceptance because the headline accuracy number depends on them. I would encourage the editor to ask for a revised version with those validations rather than rejecting outright, as the dataset fills a real niche."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LogicCat is worth taking seriously, but the headline number should come with a caveat. The new thing here is real: a 4,038-example benchmark that pushes beyond Archer by adding physics knowledge, more databases, and explicit chain-of-thought annotations. The model evaluation is broad, the error analysis is informative, and the dataset and code are released. That is reproducible evidence and should be credited.\n\nThe soft spots are concentrated in data quality and the difficulty argument. The stress-test note is right: Section 3.1 says automated scripts confirmed executability of all queries, yet Table 1 reports 95.05% VES after revision. Those cannot both be true if VES is computed on gold SQL. More importantly, VES is a syntactic check; it says nothing about whether the gold query returns the answer to the question. EX is defined as matching gold results, so with no semantic agreement metric on the gold set, the 33.20% ceiling is not a clean measure of reasoning. I would not call this fatal — expert review clearly happened, and the direction of the result is probably right — but it is load-bearing for a benchmark paper.\n\nThe prompt concern also lands. Figure 4 shows the template supplies a high-level reasoning plan including the necessary formula, so the Physical Knowledge category measures formula application more than formula retrieval. That is a legitimate design choice, but it weakens the claim that the benchmark tests domain knowledge. And the \"substantially increases difficulty\" claim would be sharper with a same-model, same-prompt baseline on Archer and Spider; comparing to published numbers leaves prompt effects uncontrolled.\n\nMinor issues: the \"first\" claim is overstated given Archer already covers arithmetic, commonsense, and hypothetical reasoning, and the paper has small internal inconsistencies (12,114 vs 12,144 CoT annotations; 500 vs 800 person-hours). These are easy fixes.\n\nWho is this for? People building or evaluating text-to-SQL systems, especially those interested in multi-step reasoning. The dataset is likely to get community use. What it needs before being treated as a definitive stress test is a semantic correctness audit on a sample of gold queries and a controlled baseline run. That is exactly what a good referee should ask for.\n\nRecommendation: send it to peer review. With major revisions addressing gold correctness and the prompt-disclosure issue, this could be a solid benchmark contribution.","headline":"LogicCat is a genuinely useful reasoning-heavy text-to-SQL benchmark, but its headline numbers rest on gold SQL whose semantic correctness is not established, so the difficulty claim is indicative rather than definitive.","tokens_in":15764,"tokens_out":2132,"would_cite":true,"duration_ms":20881,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces LogicCat, a text-to-SQL benchmark whose questions require physics, arithmetic, commonsense, and hypothetical reasoning, and reports that the strongest evaluated system reaches only 33.20% execution accuracy.","keywords":["text-to-SQL","chain-of-thought","benchmark dataset","complex reasoning","execution accuracy","physical knowledge reasoning","commonsense reasoning","hypothetical reasoning"],"falsifier":"Independently recompute the answers to a random sample of LogicCat questions — solving the physics and arithmetic by hand — and compare them with the outputs of the gold SQL; a mismatch rate above a few percent would mean the execution-accuracy numbers are not a clean measure of model reasoning.","tokens_in":14760,"feed_emoji":"🐱","tokens_out":13900,"duration_ms":91389,"temperature":0.7,"pith_summary":"The paper argues that existing text-to-SQL benchmarks miss the multi-step reasoning that real data analysis requires, and that this blind spot is measurable. It introduces LogicCat, a benchmark of 4,038 English questions paired with 12,114 chain-of-thought reasoning steps across 45 databases, built so that answering questions requires physics formulas, arithmetic, commonsense, and hypothetical scenarios. In zero-shot evaluation, the best system reaches 33.20% execution accuracy, and accuracy on hard questions falls to at most 14.96%. The paper's ablation shows that removing the chain-of-thought annotations lowers execution accuracy on every tested model, with the largest drops above 19 points. The point of the benchmark is to give systems that claim to handle real-world analytical SQL a test where the reasoning demand is explicit.","feed_headline":"New benchmark caps top text-to-SQL models at 33.20% accuracy","feed_subtitle":"LogicCat adds physics, math, commonsense, and hypothetical questions; dropping chain-of-thought hurts every model.","key_machinery":"The carrying mechanism is the benchmark design itself. Each LogicCat question is paired with a gold SQL query and a chain-of-thought annotation that decomposes the reasoning into numbered steps, and the questions are constructed to force four reasoning types — physical knowledge (35.0% of questions), mathematical logic (78%), commonsense (59%), and ideal hypothetical scenarios (25%) — often in combination. Difficulty tiers are defined by SQL token length and the number of symbolic arithmetic operators, and the dataset averages 4.42 arithmetic operators and 3.1 joins per query. Evaluation uses VES, which only checks that a predicted query executes, and execution accuracy, which requires exact match of the executed result; the gap between the two is what lets the paper attribute failures to reasoning rather than syntax. The quality gate is an executability check on the gold SQL, which rose to a 95.05% valid-execution rate after expert review.","core_discovery":"LogicCat is presented as the first text-to-SQL benchmark designed around chain-of-thought parsing. The central discovery is a performance ceiling: the strongest evaluated approach reaches only 33.20% execution accuracy overall, and hard questions cap near 14.96%, while syntactically valid execution rates reach about 70% for the best models. The gold queries come with 12,114 explicit reasoning steps, and those steps are load-bearing: an ablation shows that removing chain-of-thought annotations lowers execution accuracy in every tested case, with the largest drops above 19 points. The paper reads this as evidence that current models can parse schemas but cannot yet chain external knowledge, unit conversions, and hypothetical assumptions into correct SQL.","pith_inferences":["Inference: a natural next step is to measure how often a model's stated chain-of-thought steps are actually consistent with its final SQL, turning the annotations into a new 'reasoning fidelity' metric.","Inference: the paper does not report human performance, so an independent sample of human accuracy on a subset of questions would clarify how much of the 33.20% ceiling reflects model limits rather than ambiguity in the questions themselves.","Inference: the large gap between valid-execution and exact-match scores suggests errors concentrate after syntax generation, so a targeted error taxonomy could guide training data that fixes wrong-query and incorrect-knowledge failures first.","Inference: perturbing database values, adding distractor columns, or changing unit systems would test whether models generalize the underlying reasoning or rely on surface patterns in the schema."],"forward_implications":["If LogicCat's difficulty numbers hold, text-to-SQL systems aimed at real analytics must be tested on chained computations and domain knowledge, not just retrieval over business schemas.","Because removing the chain-of-thought annotations consistently degrades accuracy, stepwise reasoning is a load-bearing part of the benchmark and a practical requirement for deployment.","The category breakdown — commonsense easiest, physics and math hardest — gives a concrete target: improving formula application and unit handling should move overall execution accuracy more than general SQL tuning.","Releasing the dataset lets any future system be measured against the same 33.20% ceiling, making progress on complex reasoning a trackable quantity."],"supporting_citations":[{"why":"The standard cross-domain text-to-SQL dataset that LogicCat argues deliberately excludes mathematical queries; supplies the main baseline for cross-domain difficulty.","marker":"Yu et al. 2018a"},{"why":"The human-labeled benchmark with arithmetic, commonsense, and hypothetical reasoning that LogicCat extends and enlarges.","marker":"Zheng, Lapata, and Pan 2024"},{"why":"The large-scale real-world benchmark whose high reported execution accuracy motivates LogicCat's harder reasoning questions.","marker":"Li et al. 2024a"},{"why":"Survey that supplies the VES and execution-accuracy metrics used throughout the evaluation.","marker":"Qin et al. 2022"},{"why":"The chain-of-thought prompting method whose value LogicCat tests by ablation.","marker":"Wei et al. 2022"},{"why":"Prior text-to-SQL chain-of-thought prompting work that LogicCat's annotations build on.","marker":"Tai et al. 2023"},{"why":"One of the state-of-the-art text-to-SQL pipelines evaluated on LogicCat.","marker":"Pourreza et al. 2025a"},{"why":"The enterprise-level text-to-SQL benchmark used in LogicCat's complexity comparison.","marker":"Lei et al. 2025"}],"fun_headline_variants":["LogicCat: SQL benchmark where best models hit 33.2%","Why text-to-SQL fails: new benchmark shows 33% ceiling","Chain-of-thought is key in new SQL benchmark, top score 33.2%","New benchmark: even best models fail 2 out of 3 SQL queries","Text-to-SQL gets tougher: LogicCat cuts top accuracy to 33%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy numbers are only meaningful if each gold SQL query actually returns the answer to the question it accompanies; the paper verifies that the gold queries are executable (95.05% VES) but does not verify that their results are semantically correct answers.","fun_headline_variants_meta":{"raw":{"variants":["LogicCat: SQL benchmark where best models hit 33.2%","Why text-to-SQL fails: new benchmark shows 33% ceiling","Chain-of-thought is key in new SQL benchmark, top score 33.2%","New benchmark: even best models fail 2 out of 3 SQL queries","Text-to-SQL gets tougher: LogicCat cuts top accuracy to 33%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000247,"raw_usage":{"total_tokens":1536,"prompt_tokens":933,"completion_tokens":603,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":500}},"tokens_in":549,"tokens_out":603,"duration_ms":4969,"temperature":1.0,"reasoning_tokens":500,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:25:30.174837+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently recompute the answers to a random sample of LogicCat questions — solving the physics and arithmetic by hand — and compare them with the outputs of the gold SQL; a mismatch rate above a few percent would mean the execution-accuracy numbers are not a clean measure of model reasoning.","supporting_citations":[],"review_version":1}