REVIEW 5 major objections 5 minor 15 references
SQLord: A Robust Enterprise Text-to-SQL Solution via Reverse Data Generation and Workflow Decomposition
T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read SQLord claims that reverse-generating training questions from a company's own SQL and automating workflow decomposition lets a 7B open-source model beat GPT-4-based frameworks on enterprise and Spider text-to-SQL benchmarks.
desk verdict The enterprise offline gain is unsafe until the paper shows the 6,000 evaluation pairs are disjoint from the 35,948 reverse-generated training pairs from the same two scenarios. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the reverse data generation loop. RevLLM, a model trained on developer-written <SQL, COMMENT> pairs, generates a natural-language query for each raw SQL statement; the resulting pseudo-annotated <Query, SQL> dataset is used for supervised fine-tuning of SQLLM. Around this loop sit an automated workflow generator that retrieves domain knowledge and schema via vector similarity and decomposes queries into dependency-ordered sub-tasks, and GPT-Judge, a three-mode evaluator (EXE compares execution results, QSE judges query-SQL consistency, SSE judges SQL-SQL equivalence) that can score quality even when ground truth or a database is unavailable.
What would settle it
Take a random sample of pseudo-annotated pairs from Dgen and have independent raters judge whether each question is unambiguous and whether the SQL is a correct answer; if a substantial fraction fail, the supervised fine-tuning stage is being trained on hallucinated mappings and the reported accuracy gains come from somewhere other than faithful data.
Extended reading notes
Core claim
The central claim is that unlabeled enterprise SQL, together with the comments developers already write, is sufficient raw material for a complete text-to-SQL solution. SQLord trains RevLLM on <SQL, COMMENT> pairs, uses it to mass-produce <Query, SQL> training pairs, fine-tunes SQLLM on them, and wraps generation in an automated workflow that retrieves schema and business knowledge and decomposes queries into executable sub-tasks. The paper reports that this pipeline raises execution accuracy from 51.3% for the base Qwen to 86.5% on the enterprise dataset, and that the full system outperforms GPT-4-based frameworks on both enterprise data and Spider.
Load-bearing premise
The load-bearing premise is that the natural-language questions RevLLM generates from raw SQL are faithful enough to serve as training labels; the paper states this but reports no human validation rate, no filtering criteria, and no analysis of generation failures.
Editorial extensions
If this is right
- Reverse data generation turns existing unlabeled SQL in production systems into training data, so specialized NL2SQL models can be built without hand-annotated datasets.
- A 7B open-source model fine-tuned this way can exceed the execution accuracy of GPT-4-based in-context-learning systems, suggesting domain-specific fine-tuning is more decisive than model size.
- The workflow decomposition component provides the largest single gain in the ablation, implying that complex business queries benefit from being routed through retrieval and sub-task planning.
- GPT-Judge's QSE and SSE modes allow evaluation without executable databases or ground-truth SQL, which the paper argues is the common real-world situation.
Reading between the lines
- The same reverse-generation pipeline could be applied to other logged code-and-comment corpora, such as data transformation scripts or API usage examples, wherever unlabeled code is abundant.
- A quality filter or human validation rate for RevLLM-generated questions would probably further improve SQLLM accuracy, since the paper reports no measurement of generation failures.
- QSE/SSE scores could double as a training reward or as automatic filters for selecting synthetic data, rather than only as offline evaluation metrics.
- The reported +15-16% online gains suggest the offline enterprise benchmark may understate real-world impact, but also that online deployments introduce their own variability.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SQLord, an enterprise text-to-SQL framework built on three ideas: reverse data generation (training a RevLLM on <SQL, comment> pairs to synthesize <query, SQL> pseudo-labels for fine-tuning), automated workflow decomposition via retrieval-augmented dynamic sub-task generation, and a GPT-Judge evaluation framework with EXE, QSE, and SSE modes. Offline experiments report 86.5% execution accuracy on a 6,000-pair enterprise dataset (vs. 72.9% for DAIL-SQL with GPT-4) and 87.4% on Spider using Qwen2-7B as the base model. An ablation attributes gains to reverse generation and workflow generation, and an online evaluation reports 15.6–16.7 percentage-point improvements in two real scenarios.
Significance. The combination of synthetic reverse-generated training data, workflow decomposition, and an LLM judge is practically relevant: if the reported numbers are trustworthy, the paper would demonstrate that a 7B open model with domain-specific synthetic data can outperform GPT-4-based pipelines on enterprise text-to-SQL. The reverse-generation idea is reasonable, and the three-mode evaluation framework addresses a real operational need. However, the paper provides no data release, no overlap analysis between the generated training corpus and the test set, no human validation of RevLLM outputs, and no human agreement study for GPT-Judge. These omissions leave the headline gains unverifiable and create a concrete risk of train/test leakage and LLM self-consistency. The work is a strong candidate for an industrial-systems paper, but the evidence as written does not yet support the central claims.
major comments (5)
- [Section 3 (Dataset) and Section 2.1] The paper never establishes that the 6,000 manually annotated evaluation pairs are disjoint from the 35,948 pseudo-annotated pairs used to fine-tune SQLLM. Both are drawn from the same two enterprise scenarios (Customs Import-Export Assistant and Intelligent Product Selection), and Section 2.1 says the raw SQL set S comes from routine development without any temporal or query-based separation from the evaluation set. If the evaluation SQLs or near-equivalents appear in Dgen, SQLLM has been trained on the gold answers to the test queries while the GPT-4 and DAIL-SQL baselines have not, and the +13.6 EXE gain in Table 1 would be a memorization artifact rather than evidence of generalization. The authors need to report an explicit overlap analysis (e.g., exact or near-duplicate SQL matching after normalization) and describe how the evaluation set was held out.
- [Section 2.1, Eq. (1)] The pseudo-annotation step q_i = RevLLM(s_i; θ) is the load-bearing premise of the SFT pipeline, yet the paper reports no human validation rate, no filtering criteria, and no analysis of generation failures for the 35,948 generated <Query, SQL> pairs. If RevLLM produces questions that admit multiple readings, or if the original SQL is not a correct answer to the generated question, the fine-tuning objective in Eq. (2) teaches SQLLM a hallucinated question–SQL mapping. The authors should provide a human-annotated sample (e.g., 200–500 pairs) with validity/accuracy measurements, describe failure modes, and state any filtering or deduplication applied before training.
- [Section 3.3 and Abstract] The online evaluation in Table 4 reports only deltas (+15.6%, +16.4%, etc.) and gives no absolute accuracy values, no number of online test queries, and no definition of 'before and after the launch.' The abstract's claim that 'online accuracy consistently exceeds 90' does not correspond to any number in the paper, so it is unverifiable. The authors should report absolute EXE/QSE/SSE scores, sample sizes, and the exact comparison protocol for the online before/after measurement.
- [Section 2.3 and Section 3.1] The GPT-Judge evaluation is performed by an LLM whose judgment criteria are not specified (no prompt template, no temperature, no cutoff threshold), and no human agreement study is reported. The statement in Section 3.1 that 'the consistent differences in QSE and SSE scores (±7%) confirm GPT-Judge's reliability' is not a validity argument: a stable gap between metrics says nothing about whether the judge agrees with human intent. Given that SQLLM is trained on LLM-generated data and judged by an LLM, there is a systematic risk that the reported QSE/SSE gains reflect self-consistency between the generator and the judge. A concrete corrective is to report Cohen's kappa or agreement rate between GPT-Judge and human judges on a subset of at least 200 samples for both QSE and SSE.
- [Section 2.2 and Table 3] The workflow-decomposition component is underspecified: the paper does not state how sub-tasks are formulated from the retrieved context, how intermediate results are represented, how SQLLM is prompted per sub-task, or how the final Summary step combines results. This makes the ablation in Table 3 hard to interpret: the row labels do not make clear whether '+ Workflow Generation' is applied on top of reverse generation or on the bare Qwen model, and no Spider ablation is given. Since workflow generation is one of the two components credited with the largest gains, the authors need to specify the decomposition algorithm (including the exact role of d_q and the dependency graph) and clarify the cumulative structure of the ablation.
minor comments (5)
- [Section 3 (Dataset)] The paper should describe how the 6,000 manually annotated pairs were produced, including the number of annotators and inter-annotator agreement; without this information, the gold SQL may be seen as arbitrary rather than reproducible.
- [Section 3.1, Table 2] The Spider evaluation setup needs clarification: for SQLord on Spider, how are the knowledge base K and TableHub instantiated and retrieved for each of the many databases? This is essential for reproducing the reported 87.4% EXE.
- [Section 2.3, EXE definition] The execution comparison I(r_gen = r_gold) should specify the equivalence semantics for result sets, including row ordering, duplicate rows, null values, and floating-point comparisons, all of which are nontrivial in practice.
- [Throughout] Several typos and style issues remain, including 'Nature language', 'statments', 'challange', and 'RAG-liked'; these should be corrected.
- [References] Reference [1] lacks an author list, and several entries omit venue or publisher details; the reference list should be brought to a consistent, complete format.
Circularity Check
No significant circularity: the Spider result is an external benchmark, and the enterprise claims rest on a data pipeline that is not shown to reduce to its own inputs by construction.
full rationale
The paper's derivation chain is not circular in the restricted sense required here. Reverse data generation trains RevLLM on <SQL, COMMENT> pairs to create pseudo-annotated <Query, SQL> training data, then fine-tunes SQLLM on that data; this is an augmentation pipeline, not a self-referential derivation. The central Spider evaluation (87.4% EXE) is measured against an independent, externally maintained benchmark on which SQLLM was not trained, so the headline comparison to DAIL-SQL and GPT-4 has independent content. On the enterprise dataset, there is a legitimate data-hygiene concern: both the 6,000 manually annotated evaluation pairs and the 35,948 reverse-generated training pairs are drawn from the same two online scenarios and 'routine development,' and the paper never states that evaluation gold SQLs were excluded from the raw SQL pool S. However, this is a potential train/test overlap, not a demonstrated circularity: the paper contains no equation or statement showing that a test gold SQL appears in Dgen, and without that direct reduction the reported 86.5% EXE cannot be shown to be equivalent to the training objective by construction. The GPT-Judge QSE/SSE metrics are auxiliary evaluation signals anchored by execution accuracy, and the claim that their consistency 'confirms GPT-Judge's reliability' is not load-bearing for the central EXE results. No load-bearing self-citation or imported uniqueness theorem appears in the paper. Therefore the appropriate finding is no significant circularity, with the enterprise overlap risk noted as an external validity concern rather than a circularity defect.
Assumptions & free parameters
free parameters (2)
- Training hyperparameters (3 epochs, learning rate 1e-5, batch size 1) =
3 epochs, 1e-5, batch size 1
- Cutoff/threshold choices for the GPT-Judge =
not stated
assumptions (3)
- domain assumption A small set of <SQL, COMMENT> pairs collected from routine development is sufficient to train RevLLM to produce high-quality questions for the large corpus of raw SQL.
- domain assumption Running generated SQL on the actual database and comparing execution results is a valid measure of correctness for all 6,000 internal examples.
- domain assumption The open-source Spider benchmark, trained on by the baselines but not by SQLord's models as far as described, is a meaningful generalization test for the enterprise workflow.
Cite this review
Pith. "Pith review of SQLord: A Robust Enterprise Text-to-SQL Solution via Reverse Data Generation and Workflow Decomposition." pith.science (2026). https://pith.science/paper/CLVWSQGM
@misc{pith2026250710629,
author = {Pith},
title = {Pith review of: SQLord: A Robust Enterprise Text-to-SQL Solution via Reverse Data Generation and Workflow Decomposition},
year = {2026},
howpublished = {\url{https://pith.science/paper/CLVWSQGM}},
note = {Machine review of arXiv:2507.10629}
}
read the original abstract
Transforming natural language into SQL queries (NL2SQL) is crucial for data-driven business applications. Existing frameworks, trained on open-source datasets, struggle with complex business logic and lack domain-specific data for fine-tuning. Additionally, evaluation methods often require annotated data and executable database environments, which are scarce in real-world scenarios. To address these challenges, we propose SQLord, an enterprise-level NL2SQL framework. First, SQLord introduces a data reverse generation approach to convert raw SQL statements into annotated data for supervised fine-tuning (SFT). Second, it proposes a decomposition method for complex queries using an automated workflow generator. Additionally, SQLord features a comprehensive GPT-Judge evaluation framework, including Execution Evaluation (EXE), Query-SQL Evaluation (QSE), and SQL-SQL Evaluation (SSE), tailored to diverse scenarios. Offline tests significantly outperform state of the art baselines, and online accuracy consistently exceeds 90, highlighting SQLord's advantages and effectiveness in complex real world scenarios. SQLord has been successfully applied across multiple scenarios on the world's largest B2B e-commerce platform.
Figures
Reference graph
Works this paper leans on
- [1]
-
[2]
LILY Group at Yale University. 2018. Spider 1.0, Yale Semantic Parsing and Text-to-SQL Challenge. (2018). https://yale-lily.github.io/spider
work page 2018
-
[3]
Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, Lu Chen, Jinshu Lin, and Dongfang Lou. 2023. C3: Zero-shot Text-to-SQL with ChatGPT. CoRR abs/2307.07306 (2023)
arXiv 2023
-
[4]
Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2024. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. Proceedings of the VLDB Endowment 17, 5 (2024)
work page 2024
-
[5]
Binyuan Hui, Ruiying Geng, Lihan Wang, Bowen Qin, Yanyang Li, Bowen Li, Jian Sun, and Yongbin Li. 2022. S 2SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parsers. In Findings of the Associa- tion for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022 . Association for Computational Linguistics, 1254–1262
work page 2022
-
[6]
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 13067–13075
work page 2023
-
[7]
Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. 2023. Graphix-t5: Mixing pre-trained transformers with graph-aware layers for text-to-sql parsing. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 37. 13076–13084
2023
-
[8]
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What Makes Good In-Context Examples for GPT-3?. In Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO@ACL 2022, Dublin, Ireland and Online, May 27, 2022 . Association for Computa...
work page 2022
Show all 15 references
-
[9]
Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuyu Luo, Yuxin Zhang, Ju Fan, Guoliang Li, and Nan Tang. 2024. A Survey of NL2SQL with Large Language Models: Where are we, and where are we going? CoRR abs/2408.05109 (2024)
2024
-
[10]
Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. 2023. Enhancing text-to-SQL capabilities of large language models: A study on prompt design strategies. In Findings of the Association for Computational Linguistics: E...
2023
- [11]
-
[12]
Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed In- Context Learning of Text-to-SQL with Self-Correction. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA,...
2023
-
[13]
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022. Evaluating the Text-to-SQL Capabilities of Large Language Models.CoRR abs/2204.00498 (2022)
2022 arXiv
-
[14]
Immanuel Trummer. 2022. CodexDB: Synthesizing code for query processing from natural language instructions using GPT-3 Codex. Proceedings of the VLDB Endowment 15, 11, 2921–2928
2022
-
[15]
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, ...
2020
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.