{"id":"3124bab8-2f4a-4800-b7eb-2acf18aceac7","arxiv_id":"2508.12769","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CRED-SQL substantially improves Text-to-SQL on large-schema benchmarks by down-weighting common schema columns during retrieval and generating SQL through a natural language execution description.","lead":"CRED-SQL is a new framework for turning natural language questions into SQL over very large database schemas. It combines cluster-based schema retrieval with a natural language intermediate representation, reporting large accuracy gains over CRUSH on two large-schema benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EDL fine-tuning may include the Spider/Bird dev questions used for evaluation, which would invalidate the reported execution-accuracy SOTA numbers unless the authors confirm train-only splits.","rationale":"The reader's weakest assumption identifies exactly the condition on which the paper's central empirical claim depends: the separation between EDL training data and evaluation data. The paper's own wording makes overlap plausible, but not certain, because the phrase \"based on the training and validation sets\" could in principle mean that the datasets include both splits and that only the training portion is used for supervised fine-tuning. However, Section 3.3 calls the examples \"training instances\" and Section 4.6 evaluates on the validation set, so the burden is on the authors to demonstrate that no dev questions were seen during fine-tuning. I agree with the reader that this is the most load-bearing concern. I do not escalate the verdict to REJECT because the CLSR-only ablation (72.5% on SpiderUnion, Table 6) suggests that even without EDL the retrieval component may support a strong system, and the contamination claim is empirically checkable from the released code. I also note that the paper claims higher Text-to-EDL accuracy than GPT-4o but never reports Text-to-EDL accuracy, and that the EDL-to-SQL results in Table 5 use gold EDLs and therefore do not validate the Text-to-EDL stage. These omissions reinforce the need for a clean split check but do not replace it. The concrete test proposed here would settle the central validity question directly.","tokens_in":18873,"tokens_out":7924,"duration_ms":84660,"concrete_test":"Inspect the released repository (github.com/smduan/CRED-SQL) or contact the authors for the exact question IDs used in LoRA fine-tuning for Spider-EDL and Bird-EDL. Check whether any Spider development-set or Bird development-set question ID appears in the fine-tuning files. If any do, retrain both modules from scratch using only the original Spider/Bird training splits, rerun Tables 2, 4, 5, and 11, and report the execution-accuracy deltas. A material drop (e.g., more than a few points on SpiderUnion) would confirm the concern; identical or near-identical numbers after train-only fine-tuning would resolve it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 states that Spider-EDL and Bird-EDL are constructed \"based on the training and validation sets\" of the original benchmarks and describes every example as a \"training instance.\" Section 4.6 then evaluates all EDL models \"on the Spider validation set,\" and Section 4.4 evaluates on SpiderUnion and BirdUnion dev sets, which are exactly the Spider and Bird development questions. If the validation split appears in the LoRA fine-tuning data for either the Text-to-EDL or EDL-to-SQL module, Tables 2, 4, 5, and 11 measure memorization rather than generalization. The paper never explicitly states that fine-tuning uses only the training partition, and no split files or question-ID checks are provided. This condition is load-bearing because the headline SOTA execution-accuracy numbers are produced by the EDL pipeline; the CLSR retrieval results in Table 3 are independent of this concern, but the end-to-end CRED-SQL numbers are not. The paper's own ablation (Table 6) shows CLSR alone with direct NLQ-to-SQL reaches 72.5% on SpiderUnion versus 73.4% with EDL, so the retrieval contribution may survive, but the EDL-specific improvements and the exact claimed numbers would be invalidated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CRED-SQL, a Text-to-SQL framework for large unified schemas. It consists of CLSR (cluster-based schema retrieval that down-weights semantically redundant columns) and EDL (Execution Description Language, a natural-language intermediate representation that decomposes NLQ-to-SQL into Text-to-EDL and EDL-to-SQL). The authors construct Spider-EDL and Bird-EDL datasets from Spider and Bird, fine-tune open LLMs on these datasets, and evaluate on SpiderUnion and BirdUnion, reporting state-of-the-art execution accuracy (73.4% on SpiderUnion with Qwen2.5-Coder-32B and 62.91% on BirdUnion with CRED-SQL+MAC-SQL). They also report large table-recall gains for CLSR over CRUSH, conduct ablations, and release code.","tokens_in":19165,"tokens_out":6062,"duration_ms":59747,"significance":"If the reported results are validated, the CLSR retrieval method is a substantial contribution: table recall@3 improves from roughly 0.31 to 0.81 on SpiderUnion with Qwen2.5-Coder-32B, and the end-to-end gains over CRUSH-based baselines are large. The EDL representation is an interesting direction, but the paper's own ablations show only a 0.9-point gain on SpiderUnion and a degradation on Bird dev for the main model, so the incremental value of EDL is not clearly established. The lack of an explicit train/dev split statement for the EDL fine-tuning data is a serious validity risk for all EDL execution-accuracy numbers; until that is resolved, the claimed SOTA results are conditional.","major_comments":[{"comment":"The paper must clarify whether the Spider-EDL and Bird-EDL fine-tuning data include the validation (development) questions that are later used for evaluation. Section 3.3 states these datasets are constructed 'based on the training and validation sets of the original Spider and Bird benchmarks,' and Section 4.6 evaluates EDL models 'on the Spider validation set' (and similarly on Bird dev and BirdUnion/SpiderUnion dev in Section 4.4). If the dev questions appear in the fine-tuning data for either the Text-to-EDL or EDL-to-SQL module, then the execution-accuracy numbers in Tables 2, 4, 5, and 11 measure memorization rather than generalization. The authors should explicitly state that only the training partition was used, and provide split files or question-ID overlap checks.","section":"Section 3.3, Section 4.4, Section 4.6, Tables 2, 4, 5, 11"},{"comment":"The definition of Recall is mathematically inverted. The text defines Recall as |R(g)| / |R(g) ∩ R(q)|, where R(g) is the gold set and R(q) is the selected set; this expression is typically greater than 1 and is not a recall. The correct form should be |R(q) ∩ R(g)| / |R(g)|. Since Table 3 uses this metric to support the central retrieval claim, the formula should be corrected and the reported values verified against the corrected definition.","section":"Section 4.2, Eq. for Recall"},{"comment":"The claimed benefit of EDL is inconsistent across the paper. Table 6 shows that on SpiderUnion, replacing NLQ→SQL with NLQ→EDL→SQL improves execution accuracy only from 72.5% to 73.4% (0.9 points). Table 11 shows that on the Bird dev set with the original schema, the strongest open model (Qwen2.5-Coder-32B) drops from 58.41% (NLQ→SQL) to 53.19% (NLQ→EDL→SQL). The paper should discuss this degradation explicitly and provide a BirdUnion ablation with CLSR replaced, to support the general claim that EDL improves accuracy rather than only helping on SpiderUnion by a small margin.","section":"Table 6 and Table 11"}],"minor_comments":[{"comment":"The headline comparison in Table 2 changes both retrieval (CRUSH vs CLSR) and generation (NLQ→SQL/QPL vs NLQ→EDL→SQL) simultaneously. The paper does provide an ablation in Table 6, but the main text should more clearly state that most of the end-to-end gain comes from CLSR, and avoid attributing the full gap between CRUSH and CRED-SQL to EDL.","section":"Section 4.4"},{"comment":"The pseudo-code contains undefined or inconsistently named variables (e.g., 'max_cluster_cat', 'clu_size' vs 'c_s', and 'cluster_categories_list' is updated but not used consistently). Please rewrite the algorithm so it is unambiguous and reproducible.","section":"Appendix A.2, Algorithm 1"},{"comment":"Section 4.6 says the QPL comparison 'is limited to Spider' because there is no Bird-QPL dataset, but Appendix A.3.3 reports QPL results on Bird dev. Please reconcile these statements: either the Bird-QPL dataset was manually constructed, as described in A.3.3, and the earlier sentence should be qualified, or the appendix should be removed from the comparison.","section":"Section 4.6 and Appendix A.3.3"},{"comment":"There are several typos and minor wording issues, including 'Abalation Study' in Table 6, 'Execute Description Language' in the Section 4.6 title, and 'close-source' instead of 'closed-source' in Section 4.3. A careful proofread would improve presentation.","section":"Throughout"},{"comment":"Reference [1] is the paper itself; this self-citation should be removed and replaced with a normal citation of the arXiv version if needed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the train/dev split for the EDL fine-tuning data. If the authors cannot confirm that the Spider and Bird validation questions were excluded from Spider-EDL and Bird-EDL training, the EDL-related execution-accuracy numbers are not interpretable, and a rejection would be warranted. The CLSR retrieval results in Table 3 appear independent of this concern and are the strongest part of the paper. I recommend asking for an explicit split statement and, if necessary, re-running the EDL experiments with a clean training split before final acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Leading with the punchline: the cluster-based retrieval (CLSR) is a genuine improvement. Recall at 3 on SpiderUnion is 0.81 versus 0.31 for CRUSH, and the trend holds across both benchmarks. That is the kind of number that makes people pay attention. The EDL part is much more modest; the paper's own ablation shows it adds 0.9 points on SpiderUnion, and the headline Table 2 changes both retrieval and generation at once, so the 'SOTA' framing mostly travels on the retrieval win, not on the intermediate language.\n\nWhat's new and good: the idea of down-weighting columns in large semantic clusters is simple, sensible, and appears to work. The two new EDL datasets could be useful to the community. The paper is honest about response time overhead and the manual dataset construction. No circular fitting in the method; the benchmarks are external.\n\nSoft spots, in order of severity. First, data-split ambiguity. Section 3.3 says Spider-EDL and Bird-EDL are built from the training and validation sets, and nowhere does the paper explicitly state that LoRA fine-tuning uses only the training partition. Since all EDL evaluations are on the dev sets, the end-to-end numbers are at risk if validation leaked into fine-tuning. This needs a clear sentence plus split files in the repo. Second, the main comparison is confounded: Table 2 changes both CLSR and EDL relative to the CRUSH+SQL baselines. The ablation partly fixes this, but the framing oversells EDL. Third, retrieval baselines are thin. Only CRUSH is compared; CHESS and MURRE are cited but not evaluated, so 'state of the art' is under-supported. Minor: the recall formula in Section 4.2 is inverted, |R(g)|/|R(g)∩R(q)|, which is a typo but should be fixed.\n\nWho is this for? People working on large-schema text-to-SQL, especially schema linking. It deserves a serious referee, but the split question has to be answered first. I'd engage with it, ask for a train/dev statement and a broader baseline set, and treat the CLSR recall result as the main claim to verify.","headline":"CLSR retrieval is a real advance; EDL's added value is small and a train/dev split ambiguity must be resolved before the SOTA numbers can be believed.","tokens_in":19715,"tokens_out":4363,"would_cite":true,"duration_ms":41122,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CRED-SQL claims that cluster-based schema retrieval plus a natural-language execution description lifts execution accuracy on large-scale text-to-SQL benchmarks, reaching 73.4% on SpiderUnion and 62.91% on BirdUnion.","keywords":["text-to-SQL","schema retrieval","Execution Description Language","semantic clustering","large-scale databases","SpiderUnion","BirdUnion","intermediate representation"],"falsifier":"Compare the question text in the Spider-EDL and Bird-EDL training files against the Spider and Bird development sets. If any development question appears in the fine-tuning data, rerun the Text-to-EDL evaluation with those examples removed; a drop in execution accuracy or table recall on the cleaned set would overturn the paper's EDL-related claims.","tokens_in":18684,"feed_emoji":"🗄️","tokens_out":5811,"duration_ms":52666,"temperature":0.7,"pith_summary":"CRED-SQL claims that two coordinated fixes remove most of the semantic gap that ruins text-to-SQL on large schemas: cluster-aware schema retrieval that down-weights generic, lookalike columns, and a natural-language 'execution description' that sits between the question and the SQL. On the large-schema benchmarks SpiderUnion and BirdUnion, the pipeline reports 73.4% and 62.91% execution accuracy, beating direct SQL generation and SQL-like intermediate representations such as QPL. The paper also reports that cluster retrieval alone raises table recall at rank 3 from about 0.31 to 0.81 on SpiderUnion versus CRUSH. If true, the result means open-source models can outperform closed-source ones on large database text-to-SQL when the right retrieval and intermediate representation are used.","feed_headline":"Cluster retrieval lifts text-to-SQL to 73.4% on SpiderUnion","feed_subtitle":"A cluster-based schema search plus a natural-language execution description beats direct GPT-4o SQL on large database benchmarks.","key_machinery":"The load-bearing mechanisms are CLSR and EDL. CLSR builds two indexes: one for table descriptions, one for columns, where columns are grouped by semantic similarity into clusters and each column carries its cluster size; retrieval scores a table as a weighted sum in which rare, distinctive columns count more than members of large generic clusters. EDL is a natural-language execution plan: each step is a line like 'Reserve rows of #2 where age is less than 30', using 16 operators covering scans, joins, filters, grouping, set operations, arithmetic, dates, casting, and window ranking. It carries the argument by decomposing Text-to-SQL into NLQ-to-EDL and EDL-to-SQL, with a fine-tuned Qwen2.5-Coder-32B for the first stage and any LLM for the second, so the LLM reasons about intended execution rather than about SQL syntax.","core_discovery":"The central discovery is that both failure modes of large-scale text-to-SQL, schema mismatch and semantic drift, can be attacked by restructuring the two pipeline stages around semantics rather than symbols. CLSR clusters all columns by embedding similarity, then weights each column's retrieval score by the inverse of its cluster size, so ubiquitous fields like 'name' or 'date' contribute less and rare, identifying attributes dominate ranking. EDL expresses a SQL query as a numbered, tree-shaped sequence of natural-language operations drawn from 16 fixed operators, and the task is split into NLQ-to-EDL and EDL-to-SQL. On SpiderUnion, Qwen2.5-Coder-32B fine-tuned for the EDL stage reaches 73.4% execution accuracy, and with MAC-SQL reaches 62.91% on BirdUnion; on the Spider development set, gold EDL converts to SQL at 99.5% with GPT-4o and 99.3% with Qwen2.5-Coder-32B.","pith_inferences":["The cluster-size down-weighting heuristic should transfer to other retrieval problems where schema or entity names are ambiguous; a direct test would be applying CLSR scoring to open-domain table retrieval and measuring recall at rank 3 against CRUSH-style baselines.","The threefold latency overhead suggests a practical variant could distill the Text-to-EDL and EDL-to-SQL stages into a single fine-tuned model or train EDL generation jointly with schema selection; the paper does not explore this.","If the released EDL training data contains any development-set questions, the reported gains would shrink on a clean holdout; a checkpoint trained strictly on the training split would settle whether the improvement is memorization or generalization."],"forward_implications":["Replacing CRUSH retrieval with CLSR improves execution accuracy on SpiderUnion dev by 23.2 points in the ablation, showing schema retrieval is the dominant bottleneck in large-schema settings.","EDL can be swapped into existing pipelines such as DIN-SQL, MAC-SQL, and DAIL-SQL without architectural changes and generally beats both direct SQL generation and QPL as an intermediate representation.","Fine-tuned Qwen2.5-Coder-32B with CRED-SQL outperforms GPT-4o on SpiderUnion by 4.3 points and on BirdUnion by 0.85 points when paired with MAC-SQL.","Gold EDL translates to executable SQL at roughly 99% execution accuracy on Spider dev, so the EDL-to-SQL stage is nearly lossless and most remaining error sits in schema retrieval and NLQ-to-EDL.","The two-stage design approximately triples average response time per query, a reported cost that matters for real-world deployment."],"supporting_citations":[{"why":"Constructs the SpiderUnion and BirdUnion large-schema setting and supplies the CRUSH schema-retrieval baseline that CLSR is compared against.","marker":"[11]"},{"why":"Source of Spider, from which SpiderUnion and Spider-EDL are built and whose development set measures EDL-to-SQL accuracy.","marker":"[23]"},{"why":"Source of Bird, from which BirdUnion and Bird-EDL are built and whose development set is used for evaluation.","marker":"[15]"},{"why":"DIN-SQL baseline whose pipeline is re-run with QPL and EDL substitutions.","marker":"[17]"},{"why":"MAC-SQL baseline that provides the best BirdUnion result when combined with CRED-SQL.","marker":"[21]"},{"why":"DAIL-SQL baseline, including the few-shot retrieval setup that QPL cannot satisfy on Bird.","marker":"[4]"},{"why":"GPT-4o, the closed-source model used as the generator and as the EDL-to-SQL few-shot converter.","marker":"[9]"},{"why":"Qwen2.5-Coder-32B, the open-source model fine-tuned on Spider-EDL and Bird-EDL that achieves the top reported scores.","marker":"[8]"},{"why":"QPL, the SQL-like intermediate representation that EDL is compared against and shown to outperform.","marker":"[2]"}],"fun_headline_variants":["Cluster retrieval + EDL lifts text-to-SQL to 73.4%","Semantic-first two-stage pipeline beats direct SQL on large DBs","Rare attribute columns now decide text-to-SQL retrieval","Cluster-weighted schema retrieval cuts semantic drift","EDL reformulation yields near-perfect SQL conversion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported execution-accuracy gains assume the fine-tuned models were trained only on the training partitions of Spider and Bird, never on the development questions used for evaluation, since the EDL dataset is built from both training and validation sets and the paper never states the split explicitly.","fun_headline_variants_meta":{"raw":{"variants":["Cluster retrieval + EDL lifts text-to-SQL to 73.4%","Semantic-first two-stage pipeline beats direct SQL on large DBs","Rare attribute columns now decide text-to-SQL retrieval","Cluster-weighted schema retrieval cuts semantic drift","EDL reformulation yields near-perfect SQL conversion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000997,"raw_usage":{"total_tokens":4249,"prompt_tokens":997,"completion_tokens":3252,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":3169}},"tokens_in":613,"tokens_out":3252,"duration_ms":24740,"temperature":1.0,"reasoning_tokens":3169,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:19:13.798258+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the question text in the Spider-EDL and Bird-EDL training files against the Spider and Bird development sets. If any development question appears in the fine-tuning data, rerun the Text-to-EDL evaluation with those examples removed; a drop in execution accuracy or table recall on the cleaned set would overturn the paper's EDL-related claims.","supporting_citations":[{"cited_title":"Kothyari, D","cited_arxiv_id":null,"evidence_quote":"Constructs the SpiderUnion and BirdUnion large-schema setting and supplies the CRUSH schema-retrieval baseline that CLSR is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of Bird, from which BirdUnion and Bird-EDL are built and whose development set is used for evaluation."},{"cited_title":"Pourreza and D","cited_arxiv_id":null,"evidence_quote":"DIN-SQL baseline whose pipeline is re-run with QPL and EDL substitutions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DAIL-SQL baseline, including the few-shot retrieval setup that QPL cannot satisfy on Bird."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"QPL, the SQL-like intermediate representation that EDL is compared against and shown to outperform."}],"review_version":2}