REVIEW 3 major objections 5 minor 28 references
CRED-SQL: Enhancing Real-world Large Scale Database Text-to-SQL Parsing through Cluster Retrieval and Execution Description
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read CRED-SQL claims that cluster-based schema retrieval plus a natural-language execution description lifts execution accuracy on large-scale text-to-SQL benchmarks, reaching 73.4% on SpiderUnion and 62.91% on BirdUnion.
desk verdict CLSR retrieval is a real advance; EDL's added value is small and a train/dev split ambiguity must be resolved before the SOTA numbers can be believed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanisms are CLSR and EDL. CLSR builds two indexes: one for table descriptions, one for columns, where columns are grouped by semantic similarity into clusters and each column carries its cluster size; retrieval scores a table as a weighted sum in which rare, distinctive columns count more than members of large generic clusters. EDL is a natural-language execution plan: each step is a line like 'Reserve rows of #2 where age is less than 30', using 16 operators covering scans, joins, filters, grouping, set operations, arithmetic, dates, casting, and window ranking. It carries the argument by decomposing Text-to-SQL into NLQ-to-EDL and EDL-to-SQL, with a fine-tuned Qwen2.5-Coder-32B for the first stage and any LLM for the second, so the LLM reasons about intended execution rather than about SQL syntax.
What would settle it
Compare the question text in the Spider-EDL and Bird-EDL training files against the Spider and Bird development sets. If any development question appears in the fine-tuning data, rerun the Text-to-EDL evaluation with those examples removed; a drop in execution accuracy or table recall on the cleaned set would overturn the paper's EDL-related claims.
Extended reading notes
Core claim
The central discovery is that both failure modes of large-scale text-to-SQL, schema mismatch and semantic drift, can be attacked by restructuring the two pipeline stages around semantics rather than symbols. CLSR clusters all columns by embedding similarity, then weights each column's retrieval score by the inverse of its cluster size, so ubiquitous fields like 'name' or 'date' contribute less and rare, identifying attributes dominate ranking. EDL expresses a SQL query as a numbered, tree-shaped sequence of natural-language operations drawn from 16 fixed operators, and the task is split into NLQ-to-EDL and EDL-to-SQL. On SpiderUnion, Qwen2.5-Coder-32B fine-tuned for the EDL stage reaches 73.4% execution accuracy, and with MAC-SQL reaches 62.91% on BirdUnion; on the Spider development set, gold EDL converts to SQL at 99.5% with GPT-4o and 99.3% with Qwen2.5-Coder-32B.
Load-bearing premise
The reported execution-accuracy gains assume the fine-tuned models were trained only on the training partitions of Spider and Bird, never on the development questions used for evaluation, since the EDL dataset is built from both training and validation sets and the paper never states the split explicitly.
Editorial extensions
If this is right
- Replacing CRUSH retrieval with CLSR improves execution accuracy on SpiderUnion dev by 23.2 points in the ablation, showing schema retrieval is the dominant bottleneck in large-schema settings.
- EDL can be swapped into existing pipelines such as DIN-SQL, MAC-SQL, and DAIL-SQL without architectural changes and generally beats both direct SQL generation and QPL as an intermediate representation.
- Fine-tuned Qwen2.5-Coder-32B with CRED-SQL outperforms GPT-4o on SpiderUnion by 4.3 points and on BirdUnion by 0.85 points when paired with MAC-SQL.
- Gold EDL translates to executable SQL at roughly 99% execution accuracy on Spider dev, so the EDL-to-SQL stage is nearly lossless and most remaining error sits in schema retrieval and NLQ-to-EDL.
- The two-stage design approximately triples average response time per query, a reported cost that matters for real-world deployment.
Reading between the lines
- The cluster-size down-weighting heuristic should transfer to other retrieval problems where schema or entity names are ambiguous; a direct test would be applying CLSR scoring to open-domain table retrieval and measuring recall at rank 3 against CRUSH-style baselines.
- The threefold latency overhead suggests a practical variant could distill the Text-to-EDL and EDL-to-SQL stages into a single fine-tuned model or train EDL generation jointly with schema selection; the paper does not explore this.
- If the released EDL training data contains any development-set questions, the reported gains would shrink on a clean holdout; a checkpoint trained strictly on the training split would settle whether the improvement is memorization or generalization.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CRED-SQL, a Text-to-SQL framework for large unified schemas. It consists of CLSR (cluster-based schema retrieval that down-weights semantically redundant columns) and EDL (Execution Description Language, a natural-language intermediate representation that decomposes NLQ-to-SQL into Text-to-EDL and EDL-to-SQL). The authors construct Spider-EDL and Bird-EDL datasets from Spider and Bird, fine-tune open LLMs on these datasets, and evaluate on SpiderUnion and BirdUnion, reporting state-of-the-art execution accuracy (73.4% on SpiderUnion with Qwen2.5-Coder-32B and 62.91% on BirdUnion with CRED-SQL+MAC-SQL). They also report large table-recall gains for CLSR over CRUSH, conduct ablations, and release code.
Significance. If the reported results are validated, the CLSR retrieval method is a substantial contribution: table recall@3 improves from roughly 0.31 to 0.81 on SpiderUnion with Qwen2.5-Coder-32B, and the end-to-end gains over CRUSH-based baselines are large. The EDL representation is an interesting direction, but the paper's own ablations show only a 0.9-point gain on SpiderUnion and a degradation on Bird dev for the main model, so the incremental value of EDL is not clearly established. The lack of an explicit train/dev split statement for the EDL fine-tuning data is a serious validity risk for all EDL execution-accuracy numbers; until that is resolved, the claimed SOTA results are conditional.
major comments (3)
- [Section 3.3, Section 4.4, Section 4.6, Tables 2, 4, 5, 11] The paper must clarify whether the Spider-EDL and Bird-EDL fine-tuning data include the validation (development) questions that are later used for evaluation. Section 3.3 states these datasets are constructed 'based on the training and validation sets of the original Spider and Bird benchmarks,' and Section 4.6 evaluates EDL models 'on the Spider validation set' (and similarly on Bird dev and BirdUnion/SpiderUnion dev in Section 4.4). If the dev questions appear in the fine-tuning data for either the Text-to-EDL or EDL-to-SQL module, then the execution-accuracy numbers in Tables 2, 4, 5, and 11 measure memorization rather than generalization. The authors should explicitly state that only the training partition was used, and provide split files or question-ID overlap checks.
- [Section 4.2, Eq. for Recall] The definition of Recall is mathematically inverted. The text defines Recall as |R(g)| / |R(g) ∩ R(q)|, where R(g) is the gold set and R(q) is the selected set; this expression is typically greater than 1 and is not a recall. The correct form should be |R(q) ∩ R(g)| / |R(g)|. Since Table 3 uses this metric to support the central retrieval claim, the formula should be corrected and the reported values verified against the corrected definition.
- [Table 6 and Table 11] The claimed benefit of EDL is inconsistent across the paper. Table 6 shows that on SpiderUnion, replacing NLQ→SQL with NLQ→EDL→SQL improves execution accuracy only from 72.5% to 73.4% (0.9 points). Table 11 shows that on the Bird dev set with the original schema, the strongest open model (Qwen2.5-Coder-32B) drops from 58.41% (NLQ→SQL) to 53.19% (NLQ→EDL→SQL). The paper should discuss this degradation explicitly and provide a BirdUnion ablation with CLSR replaced, to support the general claim that EDL improves accuracy rather than only helping on SpiderUnion by a small margin.
minor comments (5)
- [Section 4.4] The headline comparison in Table 2 changes both retrieval (CRUSH vs CLSR) and generation (NLQ→SQL/QPL vs NLQ→EDL→SQL) simultaneously. The paper does provide an ablation in Table 6, but the main text should more clearly state that most of the end-to-end gain comes from CLSR, and avoid attributing the full gap between CRUSH and CRED-SQL to EDL.
- [Appendix A.2, Algorithm 1] The pseudo-code contains undefined or inconsistently named variables (e.g., 'max_cluster_cat', 'clu_size' vs 'c_s', and 'cluster_categories_list' is updated but not used consistently). Please rewrite the algorithm so it is unambiguous and reproducible.
- [Section 4.6 and Appendix A.3.3] Section 4.6 says the QPL comparison 'is limited to Spider' because there is no Bird-QPL dataset, but Appendix A.3.3 reports QPL results on Bird dev. Please reconcile these statements: either the Bird-QPL dataset was manually constructed, as described in A.3.3, and the earlier sentence should be qualified, or the appendix should be removed from the comparison.
- [Throughout] There are several typos and minor wording issues, including 'Abalation Study' in Table 6, 'Execute Description Language' in the Section 4.6 title, and 'close-source' instead of 'closed-source' in Section 4.3. A careful proofread would improve presentation.
- [References] Reference [1] is the paper itself; this self-citation should be removed and replaced with a normal citation of the arXiv version if needed.
Circularity Check
No circularity found: CLSR and EDL are evaluated on external benchmarks with independent baselines; the ambiguous train/validation split description is a data-integrity risk, not a demonstrated circular reduction.
full rationale
CRED-SQL's central derivation is not circular by construction. CLSR is an unsupervised clustering and weighting scheme (Eq. 8: Score(T_j) = sum Score(C_ij)W_i + Score_table(T_j), with W_i = 1/|G_k|); no parameter is fitted to the SpiderUnion/BirdUnion dev labels, and Table 3 reports retrieval recall against CRUSH on the same external benchmark. The EDL stage is a supervised reformulation: EDL is produced from gold SQL via GPT-4o with execution-equivalence verification, and the Text-to-EDL and EDL-to-SQL modules are fine-tuned on those pairs; the final accuracy numbers are compared against DIN-SQL, MAC-SQL, DAIL-SQL, and QPL baselines rather than derived from the paper's own assumptions. The only passage that could raise a circularity concern is Section 3.3, which says Spider-EDL and Bird-EDL are 'based on the training and validation sets' while Sections 4.1 and 4.6 evaluate on the Spider and Bird development sets; if the validation questions were included in the LoRA fine-tuning data, Tables 2, 4, 5, and 11 would be memorization scores rather than predictions. However, the paper's own Table 5 text distinguishes 'fine-tuned directly on the same training data' from evaluation 'on the Spider validation set,' suggesting a train/eval split, and no explicit statement establishes that validation examples were used for training. That ambiguity is a data-integrity/correctness risk to be checked against the released code and split files, not a demonstrated circular step, so it does not affect the circularity score. The self-reference [1] points to the paper's own appendix and is not load-bearing.
Assumptions & free parameters
free parameters (4)
- similarity threshold s1 =
not reported for main runs; robustness tested over 0.4-0.8
- top-N BM25 candidates in clustering =
not reported
- number of retrieved tables for SQL generation =
10 (default)
- LoRA rank and target modules =
not reported
assumptions (4)
- domain assumption Semantically similar columns that appear in many tables are less discriminative and should be down-weighted via W_i = 1/|G_k|.
- domain assumption The pretrained embedding model used for table/column vectors captures semantic similarity relevant to schema linking.
- domain assumption LLM-based schema selection is sufficiently reliable without fine-tuning.
- domain assumption The EDL datasets used for fine-tuning do not include the evaluation dev set.
invented entities (1)
-
Execution Description Language (EDL)
Cite this review
Pith. "Pith review of CRED-SQL: Enhancing Real-world Large Scale Database Text-to-SQL Parsing through Cluster Retrieval and Execution Description." pith.science (2026). https://pith.science/paper/VTEWFITC
@misc{pith2026250812769,
author = {Pith},
title = {Pith review of: CRED-SQL: Enhancing Real-world Large Scale Database Text-to-SQL Parsing through Cluster Retrieval and Execution Description},
year = {2026},
howpublished = {\url{https://pith.science/paper/VTEWFITC}},
note = {Machine review of arXiv:2508.12769}
}
read the original abstract
Recent advances in large language models (LLMs) have significantly improved the accuracy of Text-to-SQL systems. However, a critical challenge remains: the semantic mismatch between natural language questions (NLQs) and their corresponding SQL queries. This issue is exacerbated in large-scale databases, where semantically similar attributes hinder schema linking and semantic drift during SQL generation, ultimately reducing model accuracy. To address these challenges, we introduce CRED-SQL, a framework designed for large-scale databases that integrates Cluster Retrieval and Execution Description. CRED-SQL first performs cluster-based large-scale schema retrieval to pinpoint the tables and columns most relevant to a given NLQ, alleviating schema mismatch. It then introduces an intermediate natural language representation-Execution Description Language (EDL)-to bridge the gap between NLQs and SQL. This reformulation decomposes the task into two stages: Text-to-EDL and EDL-to-SQL, leveraging LLMs' strong general reasoning capabilities while reducing semantic deviation. Extensive experiments on two large-scale, cross-domain benchmarks-SpiderUnion and BirdUnion-demonstrate that CRED-SQL achieves new state-of-the-art (SOTA) performance, validating its effectiveness and scalability. Our code is available at https://github.com/smduan/CRED-SQL.git
Reference graph
Works this paper leans on
-
[1]
S. Duan, Z. Wang, C. Liu, Z. Zhu, Y . Zhang, P. Han, L. Yan, and Z. Penge. Cred-sql: Enhancing real-world large scale database text- to-sql parsing through cluster retrieval and execution description, 2025. URL https://arxiv.org/abs/2508.12769
work page Pith review arXiv 2025
-
[2]
B. Eyal, M. Mahabi, O. Haroche, A. Bachar, and M. Elhadad. Semantic decomposition of question and sql for text-to-sql parsing. In The 2023 Conference on Empirical Methods in Natural Language Processing
work page 2023
-
[3]
Y . Gan, X. Chen, J. Xie, M. Purver, J. R. Woodward, J. Drake, and Q. Zhang. Natural sql: Making sql easier to infer from natural language specifications. arXiv preprint arXiv:2109.05153, 2021
arXiv 2021
-
[4]
D. Gao, H. Wang, Y . Li, X. Sun, Y . Qian, B. Ding, and J. Zhou. Text- to-sql empowered by large language models: A benchmark evaluation. Proceedings of the VLDB Endowment, 17(5):1132–1145, 2024
work page 2024
-
[5]
Z. Gu, J. Fan, N. Tang, L. Cao, B. Jia, S. Madden, and X. Du. Few- shot text-to-sql translation using structure and content prompt learning. Proceedings of the ACM on Management of Data, 1(2):1–28, 2023
work page 2023
-
[6]
D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024
arXiv 2024
-
[7]
J. Guo, Z. Zhan, Y . Gao, Y . Xiao, J.-G. Lou, T. Liu, and D. Zhang. To- wards complex text-to-sql in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4524–4535, 2019
work page 2019
-
[8]
B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024
arXiv 2024
Show all 28 references
-
[9]
Hurst, A
A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024
2024 arXiv
-
[10]
Katsogiannis-Meimarakis and G
G. Katsogiannis-Meimarakis and G. Koutrika. A survey on deep learn- ing approaches for text-to-sql. The VLDB Journal , 32(4):905–936, 2023
2023
-
[11]
Kothyari, D
M. Kothyari, D. Dhingra, S. Sarawagi, and S. Chakrabarti. Crush4sql: Collective retrieval using schema hallucination for text2sql. InProceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14054–14066, 2023
2023
-
[12]
Lee and D.-K
J.-O. Lee and D.-K. Baik. Semql: a semantic query language for multi- database systems. In Proceedings of the eighth international conference on Information and knowledge management, pages 259–266, 1999
1999
-
[13]
H. Li, J. Zhang, C. Li, and H. Chen. Resdsql: Decoupling schema link- ing and skeleton parsing for text-to-sql. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13067–13075, 2023
2023
-
[14]
H. Li, J. Zhang, H. Liu, J. Fan, X. Zhang, J. Zhu, R. Wei, H. Pan, C. Li, and H. Chen. Codes: Towards building open-source language models for text-to-sql. Proceedings of the ACM on Management of Data, 2(3): 1–28, 2024
2024
-
[15]
J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural In- formation Processing Systems, 36, 2024
2024
-
[16]
Muennighoff
N. Muennighoff. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904, 2022
2022 arXiv
-
[17]
Pourreza and D
M. Pourreza and D. Rafiei. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. Advances in Neural Information Pro- cessing Systems, 36, 2024
2024
-
[18]
Robertson, H
S. Robertson, H. Zaragoza, et al. The probabilistic relevance frame- work: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009
2009
-
[19]
Talaei, M
S. Talaei, M. Pourreza, Y .-C. Chang, A. Mirhoseini, and A. Saberi. Chess: Contextual harnessing for efficient sql synthesis. arXiv preprint arXiv:2405.16755, 2024
2024 arXiv
-
[20]
Touvron, T
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023
2023 arXiv
-
[21]
B. Wang, C. Ren, J. Yang, X. Liang, J. Bai, Q.-W. Zhang, Z. Yan, and Z. Li. Mac-sql: Multi-agent collaboration for text-to-sql. arXiv preprint arXiv:2312.11242, 2023
2023 arXiv
-
[22]
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022
2022
-
[23]
T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural L...
2018
-
[24]
Zhang, Y
C. Zhang, Y . Mao, Y . Fan, Y . Mi, Y . Gao, L. Chen, D. Lou, and J. Lin. Finsql: Model-agnostic llms-based text-to-sql framework for financial analysis. In Companion of the 2024 International Conference on Man- agement of Data, pages 93–105, 2024
2024
-
[25]
Zhang, D
X. Zhang, D. Wang, L. Dou, Q. Zhu, and W. Che. Multi-hop table retrieval for open-domain text-to-sql.arXiv preprint arXiv:2402.10666, 2024
2024 arXiv
-
[26]
Zhang, H
Y . Zhang, H. Liu, J. Lv, X. Xiao, J. Zhu, X. Liu, J. Su, X. Li, Q. Wu, F. Wang, et al. Qdmr: a quantitative method for identification of differ- entially methylated regions by entropy. Nucleic acids research, 39(9): e58–e58, 2011. A Appendix A.1 Limitations and Future Works A...
2011
-
[27]
All columns in the database schema are vectorized, and the top N attributes most relevant to the current attribute are retrieved using the BM25 algorithm [18]
-
[28]
Find the number of cities in each district whose population is greater than the average population of cities?
A subset of attributes is selected based on a pre-set threshold. The current attribute is then classified into the cluster with the highest frequency within this set, and the cluster size is updated accord- ingly. A.3 More Results A.3.1 More results of CLSR on BirdUnion dev da...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.