Pith. sign in

REVIEW 3 major objections 5 minor 65 references

A dedicated set of error tokens lets an LLM detect and fix its own SQL mistakes, cutting correctness gaps by up to 18.5%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 19:01 UTC pith:HEFFACXL

load-bearing objection Dedicated error-token detection is a real contribution, and the gains look real; but the paper's own numbers contradict its 'without corrupting correct SQLs' claim, and missing code and error bars keep it a conditional accept. the 3 major comments →

arxiv 2603.03742 v2 pith:HEFFACXL submitted 2026-03-04 cs.CL cs.DB

ErrorLLM: Modeling SQL Errors for Text-to-SQL Refinement

Zijin Hong , Hao Chen , Zheng Yuan , Qinggang Zhang , Luyao Zhuang , Qing Liao , Feiran Huang , Yangqiu Song
show 1 more author
Xiao Huang
This is my paper
classification cs.CL cs.DB
keywords text-to-SQLSQL refinementerror tokenserror detectionlarge language modelssemantic parsingexecution accuracySQL error taxonomy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the bottleneck in text-to-SQL refinement is not correction machinery but error detection. It proposes ErrorLLM, which gives a language model a dedicated token for each of twelve SQL error types and trains it to emit those tokens from structural representations of the question, schema, and SQL abstract syntax tree. With detected errors in hand, a localization and refinement pipeline fixes only what is actually wrong, avoiding the 'corruption' of correct SQL that plagues self-correction. On the BIRD and Spider benchmarks, ErrorLLM reports the largest gains over backbone generation—BIRD execution accuracy 55.87% to 66.23%, Spider 75.44% to 86.94%—and is the only tested method that improves a stronger backbone instead of degrading it. The paper concludes that detection quality directly determines refinement effectiveness.

Core claim

ErrorLLM's central claim is that explicit, typed error detection is the foundation of effective refinement. The authors show that only a small fraction (about 3%) of incorrect SQLs raise execution errors, so self-debugging has little to work with, while unguided self-correction frequently corrupts already-correct SQLs. By extending the LLM's vocabulary with error tokens, training on a mix of rule-injected and LLM-annotated erroneous SQLs, and constraining decoding to token sequences, ErrorLLM reaches detection F1 of 78.12 on GPT-4o-generated SQL. This detection quality then drives the refinement stage: error tokens are localized to AST nodes and schema elements, a priority-ordered context is

What carries the argument

The load-bearing mechanism is the error-token vocabulary: reserved tokens [Err]1…[Err]12 and [Err]∅, each mapped to a specific error type in a 12-category taxonomy (attribute/table/value/condition/function/clause/modifier errors, etc.). The tokens are initialized from the embeddings of semantically related words and fine-tuned with LoRA on a corpus of 9,497 samples synthesized by AST-level perturbation operators and an LLM-assisted injection process. Detection runs in two stages: deterministic inverse-rule checks for high-precision surface errors, then the fine-tuned LLM with constrained decoding that can only output error tokens, given the question-schema structure, the SQL AST, and executi

Load-bearing premise

The entire framework assumes that the 12-item error taxonomy and the synthetically generated training errors—rule-based perturbations plus GPT-3.5 annotations—faithfully cover the real errors produced by LLMs on BIRD and Spider, so that the trained error tokens genuinely represent the errors encountered at inference time.

What would settle it

A direct falsifier: construct a held-out set of SQL errors that are predominantly subquery-related and operator-related (the two categories ErrorLLM scores zero on). If ErrorLLM's detection F1 on that set is close to zero while the headline 78.12 D-F1 holds on the original distribution, the claim that the framework's detection quality is sufficient for general refinement is false for precisely the error classes the taxonomy omits.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If detection quality is the binding constraint, then the framework's accuracy ceiling is raised by expanding the error taxonomy (the paper reserves token slots for this).
  • The approach transfers to stronger backbones: because it only refines SQLs for which errors are truly detected, it avoids the corruption that makes other refinement methods degrade strong generated SQL.
  • The decomposition ΔEX = (|TP|·FR − |FP|·CR)/A gives a practical recipe for engineering refinement systems: maximize true-positive coverage while keeping false-positive count near zero.
  • ErrorLLM's per-sample detection cost is tiny (constrained decoding emits roughly 1.45 tokens on average), making explicit detection affordable as a standard module in SQL pipelines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One could stress-test the 'detection determines refinement' thesis by artificially corrupting the detector's output (e.g., adding random noise to the token set) and observing whether the EX gain scales linearly with D-F1; the paper's ablations show correlation, not causality.
  • The zero type-specific accuracy on subquery- and operator-related errors is a red flag for workloads dominated by nested queries; a testable extension would be to add those two categories to the taxonomy and measure the new D-F1 and EX.
  • The priority-ordering heuristic (structural errors first) is plausible but unverified in isolation; a randomized-order ablation would clarify whether ordering itself contributes or whether the guidance templates carry all the signal.
  • The paper's corruption-rate analysis suggests a practical refinement system should treat every false positive as a potential regression; an adaptive threshold on detection confidence could trade a little recall for near-zero corruption.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ErrorLLM, a framework for refining LLM-generated SQL. It defines a 12-type error taxonomy, represents the question, schema, and SQL as a question-schema graph plus an abstract syntax tree, and trains a CodeS-7B model with LoRA to predict dedicated error tokens. Detection is two-stage: deterministic static rules plus an LLM-based semantic detector with constrained decoding. Refinement is then performed by an error-guided pipeline (LocLLM plus RefLLM) that localizes errors and applies priority-ordered corrections. The main experiments on BIRD and Spider report large execution-accuracy improvements over GPT-4o outputs (55.87→66.23 and 75.44→86.94) and smaller gains over the stronger OpenSearch-SQL backbone (+1.27% and +2.39%), with an ablation indicating that semantic error-token detection is the core driver.

Significance. If the results hold, the explicit error-token design and the detection/refinement decomposition in Eq. (27) are useful contributions: they provide a clear mechanistic explanation of why refinement helps or hurts, and the reported gains over two very different backbones are practically meaningful. The paper is also honest in several places: it discloses zero type-specific accuracy on subquery- and operator-related errors, and it includes a useful ablation separating detection from refinement components. However, the empirical claims currently rest on single runs with no variance, no released code, and a conclusion that contradicts the paper's own metric decomposition. The central idea is defensible, but the presentation and several load-bearing claims need work before publication.

major comments (3)
  1. [§5 Conclusion and §4.3, Eq. (27)] The statement that ErrorLLM improves 'without corrupting originally correct SQLs' is contradicted by the paper's own numbers. Using A=1,534, ΔEX=+10.36, |TP|=516, FR=53.88%, and Precision=80.12%, one obtains |FP|≈128. Substituting into Eq. (27) gives CR≈93%. Thus roughly 119 of the 128 false-positive SQLs are corrupted by the refinement step. This is the same compliance/hallucination behavior the paper attributes to self-correction baselines, and it undermines the §4.2 claim that ErrorLLM 'avoids the false-positive corrections that other methods struggle with.' The net gain is real, but it comes from higher recall and precision, not from a refinement stage that preserves correct SQLs. This needs to be acknowledged and the conclusions rephrased.
  2. [§3.1 and §4.4 (Figure 5)] The training labels for ErrorLLM are synthesized with the same rule-based perturbation operators that are later inverted into detection rules, supplemented by GPT-3.5 annotations. This creates a circularity risk: high D-F1 may reflect the model's ability to recognize the synthetic perturbation distribution rather than genuine semantic errors in LLM outputs. The paper's own RQ3 shows zero TSA on subquery-related and operator-related errors, and no analysis of how often real BIRD/Spider errors fall outside the 12-type taxonomy. Please report the distribution of error types actually found in the dev-set predictions that are not covered by the taxonomy, or temper the claim that detection quality 'directly determines' refinement effectiveness.
  3. [§4.1 and §4.2 (Table 1)] All experiments are single runs with no variance or significance testing. Some of the headline claims rely on small differences, e.g., +1.27% and +2.39% on OpenSearch-SQL, and ±0.00% on both the Hard and Extra Spider splits. Given the stochasticity of GPT-4o and DeepSeek-V3 sampling, these differences could be run-to-run noise. Please report multiple seeds or at least confidence intervals for the main comparisons, and ensure that the 'only method that generally improves' claim is supported by more than a single point estimate.
minor comments (5)
  1. [§4.1] Typo: 'eroor detection benchmark' should be 'error detection benchmark'.
  2. [Figure 4] The axis labels 'Depth' and 'Effectiveness' are not self-explanatory. The figure would be clearer if the axes were explicitly labeled 'Number of True Positive Detections (TP)' and 'Fixed Rate (%),' as in the caption.
  3. [Table 7] The table formatting is garbled, especially the ErrorLLM (SL) row. Please ensure the input/output token counts are presented consistently and clarify whether these are per-sample averages.
  4. [Eq. (11)] The set W_{λ_i} is described only as 'semantically related words derived from the description of error type λ_i'; please provide a precise construction rule or a concrete example for each of the 12 types, since the initialization quality is an ablation variable.
  5. [§4.3] The paragraph discussing self-correction says 'its overall ΔEX on BIRD is only ↑2.34%.' This number is from Table 1 but the derivation is not shown; including the CR implied by Eq. (27) for that baseline would make the comparison more transparent and would also make the high CR of ErrorLLM visible.

Circularity Check

0 steps flagged

No load-bearing circularity; empirical claims are benchmark-anchored, though the 'without corrupting' claim is internally inconsistent with Eq. 27.

full rationale

ErrorLLM's derivation is not circular in any load-bearing sense. The detection model is trained on synthetic data (Eqs. 12–13), but its D-F1 (78.12) and the refinement gains (BIRD 55.87→66.23, Spider 75.44→86.94) are measured on real GPT-4o outputs and on the external expert-annotated NL2SQL-Bugs benchmark, not on the synthesis distribution. The static rules are explicitly defined as inverses of the perturbation operators ('we construct the corresponding detection rules by inversion'), so they are guaranteed to recognize the injected synthetic errors; however, the rule-only recall on real BIRD SQLs is only 6.50%, so the rule component does not by itself produce the claimed improvement. The central claim that detection quality determines refinement effectiveness is supported by an ablation (Table 3) and an accounting identity (Eq. 27), not by a fit masquerading as a prediction. Self-citations (e.g., [60], [17], [57]) appear but only support structural representation choices and are not the evidence for the quantitative claims, so they are not load-bearing. The paper explicitly discloses a taxonomy limitation (zero TSA on subquery- and operator-related errors in RQ3). One important non-circularity caveat: the conclusion's phrase 'without corrupting originally correct SQLs' is inconsistent with the paper's own Eq. 27; solving with the reported BIRD/GPT-4o values (A=1534, ΔEX=+10.36, |TP|=516, FR=53.88%, Precision=80.12%) gives |FP|≈128 and CR≈93%, implying most false positives are corrupted. This is an internal consistency/correctness problem, not a circular derivation, and does not raise the circularity score.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 1 invented entities

The central claim rests on the chosen taxonomy, the synthetic data-generation scheme, and the structural encodings. The free parameters are mostly hand-set thresholds and training choices; none are fitted to the test benchmarks in a way that would turn the evaluation into a fit. The only invented entity is the error-token vocabulary, which has external falsifiable handles through the NL2SQL-Bugs benchmark and detection F1.

free parameters (6)
  • fuzzy value-link threshold γ = not reported
    Eq. 41: a cell value is accepted as a bridge value when FuzzyRatio >= γ; hand-set and not swept.
  • RGAT link threshold δ = not reported
    Eq. 48: question-span to schema edges require σ(score) >= δ; hand-set threshold.
  • Error token capacity N = 32
    Eq. 9: 32 reserved slots, 12 active error types; capacity is a design choice to allow future taxonomy growth.
  • LoRA rank/alpha = 32 / 64
    Table 6: LoRA hyperparameters chosen by hand for fine-tuning CodeS-7B.
  • Priority function φ = not specified
    Eq. 23 and Algorithm 2: orders error types for refinement; the concrete mapping and rationale are not disclosed.
  • Training data balance = 49% no-error / 51% error
    Table 5: the selected ratio shapes the detector's precision/recall trade-off and is not optimized or swept.
axioms (6)
  • domain assumption Execution-equivalence to gold SQL defines correctness (Eq. 3)
    All primary metrics (EX, FR, CR) treat two SQLs as equivalent iff Exec returns the same result; semantic errors that do not change execution are invisible to the oracle.
  • ad hoc to paper 12-type error taxonomy covers the relevant space of text-to-SQL errors
    Rule-based perturbation and error tokens are built on this taxonomy (Table 9); the paper admits it lacks subquery and operator error types, where it scores 0 TSA.
  • domain assumption Synthesized training data from BIRD train is representative of real LLM errors
    ErrorLLM is trained on rule-perturbed and GPT-3.5-annotated samples (Section 3.1), then evaluated on BIRD dev and Spider; transfer depends on distributional similarity.
  • domain assumption GPT-3.5 accepted annotations are faithful error labels
    LLM-assisted labels are kept only when the assistant's refined SQL executes like gold (Eq. 13); the annotation semantics are otherwise unverified.
  • domain assumption LoRA/frozen-embedding tuning preserves the base model's SQL abilities
    Eq. 16 assumes the original linguistic and SQL capabilities remain while only adapters and error-token embeddings change, relying on prior LoRA findings.
  • standard math RGAT/QSS/AST structural encoding faithfully represents question-schema-SQL relations
    The framework's inputs depend on RAT-SQL-style relational graph attention and AST alignment (Appendix B.1), adopted without a new proof of sufficiency.
invented entities (1)
  • Dedicated error tokens [Err]_1..[Err]_12 and [Err]_∅ plus reserved slots independent evidence
    purpose: Map each categorized SQL error type to a token so the fine-tuned LLM can detect errors by constrained token prediction.
    Their usefulness is measured externally on NL2SQL-Bugs (TSA) and D-F1 on GPT-4o-generated SQLs; this is not merely an internal label.

pith-pipeline@v1.3.0-alltime-deepseek · 30938 in / 15482 out tokens · 140795 ms · 2026-08-02T19:01:03.674927+00:00 · methodology

0 comments
read the original abstract

Despite the remarkable performance of large language models (LLMs) in text-to-SQL (SQL generation), correctly producing SQL queries remains challenging during initial generation. The SQL refinement task is subsequently introduced to correct syntactic and semantic errors in generated SQL queries. However, existing paradigms face two major limitations: (i) self-debugging becomes increasingly ineffective as modern LLMs rarely produce explicit execution errors that can trigger debugging signals; (ii) self-correction exhibits low detection precision due to the lack of explicit error modeling grounded in the question and schema, and suffers from severe hallucination that frequently corrupts correct SQLs. In this paper, we propose ErrorLLM, a framework that explicitly models text-to-SQL Errors within a dedicated LLM for text-to-SQL refinement. Specifically, we represent the user question and database schema as structural features, employ static detection to identify execution failures and surface mismatches, and extend ErrorLLM's semantic space with dedicated error tokens that capture categorized implicit semantic error types. Through a well-designed training strategy, we explicitly model these errors with structural representations, enabling the LLM to detect complex implicit errors by predicting dedicated error tokens. Guided by the detected errors, we perform error-guided refinement on the SQL structure by prompting LLMs. Extensive experiments demonstrate that ErrorLLM achieves the most significant improvements over backbone initial generation. Further analysis reveals that detection quality directly determines refinement effectiveness, and ErrorLLM addresses both sides by high detection F1 score while maintain refinement effectiveness.

Figures

Figures reproduced from arXiv: 2603.03742 by Feiran Huang, Hao Chen, Luyao Zhuang, Qinggang Zhang, Qing Liao, Xiao Huang, Yangqiu Song, Zheng Yuan, Zijin Hong.

Figure 1
Figure 1. Figure 1: Different paradigms of text-to-SQL refinement. Self [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An illustration of the modeling the SQL error in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Results of text-to-SQL execution accuracy (EX) (%) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of text-to-SQL refinement methods by [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: SQL error detection results of ErrorLLM and pro￾prietary LLM baselines on NL2SQL-Bugs [33], evaluated per error category using type-specific accuracy (TSA). As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Perturbation operator for Value Error. The exact [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Raw guideline template 𝑔𝜆7 [ · ] for Value Error with all slots unfilled. Curly-brace placeholders are filled by LocLLM conditioned on the predicted SQL, AST, and QSS. completed guideline 𝑔𝜆𝑚 [∗] that is subsequently used by RefLLM [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 13 linked inside Pith

  1. [1]

    Arian Askari, Christian Poelitz, and Xinye Tang. 2025. Magic: Generating self- correction guideline for in-context text-to-sql. InConference on Artificial Intelli- gence (AAAI)

    2025

  2. [2]

    Hasan Alp Caferoğlu and Özgür Ulusoy. 2024. E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL.arXiv preprint arXiv:2409.16751(2024)

    Pith/arXiv arXiv 2024

  3. [3]

    Ruisheng Cao, Lu Chen, Jieyu Li, Hanchong Zhang, Hongshen Xu, Wangyou Zhang, and Kai Yu. 2023. A Heterogeneous Graph to Abstract Syntax Tree Framework for Text-to-SQL.IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)(2023)

    2023

  4. [4]

    Ruisheng Cao, Hanchong Zhang, Hongshen Xu, Jieyu Li, Da Ma, Lu Chen, and Kai Yu. 2023. ASTormer: An AST Structure-aware Transformer Decoder for Text-to-SQL.arXiv preprint arXiv:2310.18662(2023)

    Pith/arXiv arXiv 2023

  5. [5]

    Zhenbiao Cao, Yuanlei Zheng, Zhihao Fan, Xiaojin Zhang, and Wei Chen. 2024. RSL-SQL: Robust Schema Linking in Text-to-SQL Generation.arXiv preprint arXiv:2411.00073(2024)

    Pith/arXiv arXiv 2024

  6. [6]

    Jipeng Cen, Jiaxin Liu, Zhixu Li, and Jingjing Wang. 2025. SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent Collaboration. InConference on Artificial Intelligence (AAAI)

    2025

  7. [7]

    Shijie Chen, Ziru Chen, Huan Sun, and Yu Su. 2023. Error Detection for Text- to-SQL Semantic Parsing. InFindings of Empirical Methods in Natural Language Processing (EMNLP)

    2023

  8. [8]

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024. Teaching Large Language Models to Self-Debug. InInternational Conference on Learning Representations (ICLR)

    2024

  9. [9]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)

    Pith/arXiv arXiv 2025

  10. [10]

    Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2021. Structure-Grounded Pretraining for Text-to-SQL. InNorth American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)

    2021

  11. [11]

    Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, Jinshu Lin, Dongfang Lou, et al. 2023. C3: Zero-shot text-to-sql with chatgpt.arXiv preprint arXiv:2307.07306(2023)

    Pith/arXiv arXiv 2023

  12. [12]

    Woodward, Jinxia Xie, and Pengsheng Huang

    Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, and Pengsheng Huang. 2021. Towards Robustness of Text-to-SQL Mod- els against Synonym Substitution. InAssociation for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL-IJCNLP)

    2021

  13. [13]

    Yujian Gan, Xinyun Chen, and Matthew Purver. 2021. Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization. InEmpirical Methods in Natural Language Processing (EMNLP)

    2021

  14. [14]

    Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2024. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. InInternational Conference on Very Large Data Bases (VLDB)

    2024

  15. [15]

    Yingqi Gao, Yifu Liu, Xiaoxia Li, Xiaorong Shi, Yin Zhu, Yiming Wang, Shiqi Li, Wei Li, Yuntao Hong, Zhiling Luo, et al. 2024. Xiyan-sql: A multi-generator ensemble framework for text-to-sql.arXiv preprint arXiv:2411.08599(2024)

    Pith/arXiv arXiv 2024

  16. [16]

    Zijin Hong, Zheng Yuan, Hao Chen, Qinggang Zhang, Feiran Huang, and Xiao Huang. 2024. Knowledge-to-SQL: Enhancing SQL Generation with Data Expert LLM. InFindings of Association for Computational Linguistics (ACL)

    2024

  17. [17]

    Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2025. Next-generation database interfaces: A survey of llm-based text-to-sql.IEEE Transactions on Knowledge and Data Engineering (TKDE)(2025)

    2025

  18. [18]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models.. InInternational Conference on Learning Representations (ICLR)

    2022

  19. [19]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large language models cannot self-correct reasoning yet. InInternational Conference on Learning Representations (ICLR)

    2024

  20. [20]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024)

    Pith/arXiv arXiv 2024

  21. [21]

    Dongjun Lee, Choongwon Park, Jaehyuk Kim, and Heesoo Park. 2025. MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For Text-to-SQL Generation. InInternational Conference on Computational Linguistics (COLING)

    2025

  22. [22]

    Boyan Li, Chong Chen, Zhujun Xue, Yinan Mei, and Yuyu Luo. 2025. DeepEye- SQL: A software-engineering-inspired text-to-sql framework.arXiv preprint arXiv:2510.17586(2025)

    Pith/arXiv arXiv 2025

  23. [23]

    Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. 2024. The Dawn of Natural Language to SQL: Are We Fully Ready?. InInternational Con- ference on Very Large Data Bases (VLDB)

    2024

  24. [24]

    Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, Hong Chen, and Cuiping Li. 2025. OmniSQL: Synthesizing High-Quality Text-to-SQL Data at Scale. In International Conference on Very Large Data Bases (VLDB)

    2025

  25. [25]

    Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. 2024. CodeS: Towards Building Open-source Language Models for Text-to-SQL. InConference on Management of Data (SIGMOD)

    2024

  26. [26]

    Jinyang Li, Binyuan Hui, GE QU, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. InAdvances in Neural Information Proces...

    2023

  27. [27]

    Muyao Li, Zihao Wang, Kaichen He, Xiaojian Ma, and Yitao Liang. 2025. JARVIS- VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse. InFindings of Association for Computational Linguis- tics (ACL)

    2025

  28. [28]

    Shuaimin Li, Xuanang Chen, Yuanfeng Song, Yunze Song, Chen Jason Zhang, Fei Hao, and Lei Chen. 2025. prompt4vis: prompting large language models with example mining for tabular data visualization.The VLDB Journal(2025)

    2025

  29. [29]

    Zhishuai Li, Xiang Wang, Jingjing Zhao, Sun Yang, Guoqing Du, Xiaoru Hu, Bin Zhang, Yuxiao Ye, Ziyue Li, Hangyu Mao, et al. 2025. Pet-sql: A prompt-enhanced two-round refinement of text-to-sql with cross-consistency. InInternational Conference on Database Systems for Advanced Applications (DASFAA)

    2025

  30. [30]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437(2024)

    Pith/arXiv arXiv 2024

  31. [31]

    Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S. Yu. 2023. A comprehen- sive evaluation of ChatGPT’s zero-shot Text-to-SQL capability.arXiv preprint arXiv:2303.13547(2023)

    Pith/arXiv arXiv 2023

  32. [32]

    Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuxin Zhang, Ju Fan, Guoliang Li, Nan Tang, and Yuyu Luo. 2025. A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?IEEE Transactions on Knowledge and Data Engineering (TKDE)(2025)

    2025

  33. [33]

    Xinyu Liu, Shuyu Shen, Boyan Li, Nan Tang, and Yuyu Luo. 2025. NL2SQL- BUGs: A Benchmark for Detecting Semantic Errors in NL2SQL Translation. In Conference on Knowledge Discovery and Data Mining (KDD)

    2025

  34. [34]

    Peixian Ma, Xialie Zhuang, Chengjin Xu, Xuhui Jiang, Ran Chen, and Jian Guo

  35. [35]

    Zheng Ning, Yuan Tian, Zheng Zhang, Tianyi Zhang, and Toby Jia-Jun Li. 2024. Insights into Natural Language Database Query Errors: from Attention Misalign- ment to User Handling Strategies.ACM Transactions on Interactive Intelligent Systems (TIIS)(2024)

    2024

  36. [36]

    Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O Arik

  37. [37]

    Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed In- Context Learning of Text-to-SQL with Self-Correction. InAdvances in Neural Information Processing Systems (NeurIPS)

    2023

  38. [38]

    Mohammadreza Pourreza and Davood Rafiei. 2024. DTS-SQL: Decomposed Text- to-SQL with Small Large Language Models. InFindings of Empirical Methods in Natural Language Processing (EMNLP)

    2024

  39. [39]

    Ge Qu, Jinyang Li, Bowen Li, Bowen Qin, Nan Huo, Chenhao Ma, and Reynold Cheng. 2024. Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation. InFindings of Association for Computational Linguistics (ACL)

    2024

  40. [40]

    Ge Qu, Jinyang Li, Bowen Qin, Xiaolong Li, Nan Huo, Chenhao Ma, and Reynold Cheng. 2025. SHARE: An SLM-based Hierarchical Action CorREction Assistant for Text-to-SQL. InAssociation for Computational Linguistics (ACL)

    2025

  41. [41]

    Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. InConference on Human Factors in Computing Systems (CHI)

    2021

  42. [42]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. InAssociation for Computational Linguistics (ACL)

    2016

  43. [43]

    Jeff Shute, Shannon Bales, Matthew Brown, Jean-Daniel Browne, Brandon Dol- phin, Romit Kudtarkar, Andrey Litvinov, Jingchi Ma, John Morcos, Michael Shen, et al. 2024. SQL has problems. We can fix them: Pipe syntax in SQL. In International Conference on Very Large Data Bases (VLDB)

    2024

  44. [44]

    Chang-Yu Tai, Ziru Chen, Tianshu Zhang, Xiang Deng, and Huan Sun. 2023. Exploring Chain of Thought Style Prompting for Text-to-SQL. InEmpirical Methods in Natural Language Processing (EMNLP)

    2023

  45. [45]

    Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. 2024. Chess: Contextual harnessing for efficient sql synthesis. ErrorLLM: Modeling SQL Errors for Text-to-SQL Refinement arXiv preprint arXiv:2405.16755(2024)

    Pith/arXiv arXiv 2024

  46. [46]

    Dayton G Thorpe, Andrew J Duberstein, and Ian A Kinsey. 2024. Dubo-SQL: Diverse Retrieval-Augmented Generation and Fine Tuning for Text-to-SQL.arXiv preprint arXiv:2404.12560(2024)

    Pith/arXiv arXiv 2024

  47. [47]

    Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Linzheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun, and Zhoujun Li. 2024. MAC- SQL: A Multi-Agent Collaborative Framework for Text-to-SQL. InInternational Conference on Computational Linguistics (COLING)

    2024

  48. [48]

    Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers. InAssociation for Computational Linguistics (ACL)

    2020

  49. [49]

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. InInternational Conference on Learning Representations (ICLR)

    2021

  50. [50]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al . 2022. Chain-of-thought prompting elicits rea- soning in large language models. InAdvances in Neural Information Processing Systems (NeurIPS)

    2022

  51. [51]

    This is My SQL, Are You With Me?

    Hanchen Xia, Feng Jiang, Naihao Deng, Cunxiang Wang, Guojiang Zhao, Rada Mi- halcea, and Yue Zhang. 2024. "This is My SQL, Are You With Me?" A Consensus- Based Multi-Agent System for Text-to-SQL Tasks. InACL 2025 Fourth Table Representation Learning Workshop (ACL)

    2024

  52. [52]

    Xiangjin Xie, Guangwei Xu, Lingyan Zhao, and Ruijie Guo. 2025. OpenSearch- SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Align- ment. InConference on Management of Data (SIGMOD)

    2025

  53. [53]

    Wenbo Xu, Haifeng Zhu, Liang Yan, Chuanyi Liu, Peiyi Han, Shaoming Duan, and Jeff Z. Pan. 2025. TS-SQL: Test-driven Self-refinement for Text-to-SQL. In Findings of Empirical Methods in Natural Language Processing (EMNLP)

    2025

  54. [54]

    Bo Yang, Yinfen Xia, Weisong Sun, and Yang Liu. 2025. Hallucination Detection for LLM-based Text-to-SQL Generation via Two-Stage Metamorphic Testing. ACM Transactions on Software Engineering and Methodology (TOSEM)(2025)

    2025

  55. [55]

    Hongyu Yao, Zijin Hong, Hao Chen, Zhiqing Li, Qijie Shen, Zuobin Ying, Qihua Feng, Huan Gong, and Feiran Huang. 2026. Macro Graph of Experts for Billion- Scale Multi-Task Recommendation. InConference on Knowledge Discovery and Data Mining (KDD)

    2026

  56. [56]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. InEmpirical Methods in Natural Language Processing (EMNLP)

    2018

  57. [57]

    Zheng Yuan, Hao Chen, Zijin Hong, Qinggang Zhang, Feiran Huang, Qing Li, and Xiao Huang. 2025. Knapsack Optimization-based Schema Linking for LLM-based Text-to-SQL Generation.arXiv preprint arXiv:2502.12911(2025)

    Pith/arXiv arXiv 2025

  58. [58]

    Hanchong Zhang, Ruisheng Cao, Lu Chen, Hongshen Xu, and Kai Yu. 2023. ACT- SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain- of-Thought. InFindings of Empirical Methods in Natural Language Processing (EMNLP)

    2023

  59. [59]

    Qinggang Zhang, Shengyuan Chen, Yuanchen Bei, Zheng Yuan, Huachi Zhou, Zijin Hong, Junnan Dong, Hao Chen, Yi Chang, and Xiao Huang. 2025. A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models.arXiv preprint arXiv:2501.13958(2025)

    arXiv 2025

  60. [60]

    Qinggang Zhang, Junnan Dong, Hao Chen, Wentao Li, Feiran Huang, and Xiao Huang. 2024. Structure Guided Large Language Model for SQL Generation. In International Conference on Machine Learning (ICML)

    2024

  61. [61]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models. InAssociation for Computational Linguistics (ACL): System Demonstrations. Zijin Hong et al. A Related Work LLM-based Text-to-SQL.LLMs have enabled two paradigms for text-to-SQL: in-co...

    2024

  62. [64]

    a natural-language description [26], and 2) the m-schema [15] representation1 up with example values{𝜖(𝑐𝑖 𝑗)} fetched from the database. Formally, the node set is defined as below: VS ={𝑣 𝑡𝑖|𝑡 𝑖∈T}∪{𝑣 𝑐𝑖 𝑗|𝑐 𝑖𝑗 ∈C, 𝑡𝑖∈T}.(29) (ii) Edge setES.We define three deterministic edge types, all in- stantiated from the database metadata (full schema). 1) Ahas_colu...

  63. [65]

    Find the names of students who have completed the Database course

    Schema–schema relations (R𝑠𝑠 ).We define structural relation types withinVS based onES. For two table nodes, the relation captures whether they are the same table, connected by a foreign key in either direction, connected bidirectionally, or otherwise un- related. For two column nodes, the relation distinguishes whether they are the same column, belong to...

    2023

  64. [2024]

    InInternational Conference on Learning Representations (ICLR)

    Chase-sql: Multi-path reasoning and preference optimized candidate selection in text-to-sql. InInternational Conference on Learning Representations (ICLR)

  65. [2025]

    InAdvances in Neural Information Processing Systems (NeurIPS)

    Sql-r1: Training natural language to sql reasoning model by reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS)