Pith. sign in

REVIEW 5 major objections 5 minor 15 references

SQLord: A Robust Enterprise Text-to-SQL Solution via Reverse Data Generation and Workflow Decomposition

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read SQLord claims that reverse-generating training questions from a company's own SQL and automating workflow decomposition lets a 7B open-source model beat GPT-4-based frameworks on enterprise and Spider text-to-SQL benchmarks.

desk verdict The enterprise offline gain is unsafe until the paper shows the 6,000 evaluation pairs are disjoint from the 35,948 reverse-generated training pairs from the same two scenarios. read the letter →

arxiv 2507.10629 v1 pith:CLVWSQGM submitted 2025-07-14 cs.DB cs.AI

classification cs.DBcs.AI
keywords text-to-SQLreversedatagenerationsupervisedfine-tuningworkflowdecompositionGPT-JudgeQwen2enterpriseSQLsynthetictraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SQLord is a framework for enterprise text-to-SQL that turns the scarcity of labeled training data into a strength. The paper claims that by training a reverse model to generate natural-language questions from raw SQL statements, fine-tuning a small open-source model on those pseudo-labeled pairs, decomposing complex queries into dependency-ordered sub-tasks, and evaluating with a three-mode LLM judge, SQLord reaches 86.5% execution accuracy on a 6,000-query enterprise dataset—13.6 points above DAIL-SQL with GPT-4—and 87.4% on Spider using only a Qwen2-7B base model. If true, this means a small open model plus synthetic data from existing SQL can rival or beat large closed-source systems on domain-specific NL2SQL, with an evaluation method that does not require executable databases or full ground-truth SQL.

What carries the argument

The load-bearing machinery is the reverse data generation loop. RevLLM, a model trained on developer-written <SQL, COMMENT> pairs, generates a natural-language query for each raw SQL statement; the resulting pseudo-annotated <Query, SQL> dataset is used for supervised fine-tuning of SQLLM. Around this loop sit an automated workflow generator that retrieves domain knowledge and schema via vector similarity and decomposes queries into dependency-ordered sub-tasks, and GPT-Judge, a three-mode evaluator (EXE compares execution results, QSE judges query-SQL consistency, SSE judges SQL-SQL equivalence) that can score quality even when ground truth or a database is unavailable.

What would settle it

Take a random sample of pseudo-annotated pairs from Dgen and have independent raters judge whether each question is unambiguous and whether the SQL is a correct answer; if a substantial fraction fail, the supervised fine-tuning stage is being trained on hallucinated mappings and the reported accuracy gains come from somewhere other than faithful data.

Watch

Extended reading notes

Core claim

The central claim is that unlabeled enterprise SQL, together with the comments developers already write, is sufficient raw material for a complete text-to-SQL solution. SQLord trains RevLLM on <SQL, COMMENT> pairs, uses it to mass-produce <Query, SQL> training pairs, fine-tunes SQLLM on them, and wraps generation in an automated workflow that retrieves schema and business knowledge and decomposes queries into executable sub-tasks. The paper reports that this pipeline raises execution accuracy from 51.3% for the base Qwen to 86.5% on the enterprise dataset, and that the full system outperforms GPT-4-based frameworks on both enterprise data and Spider.

Load-bearing premise

The load-bearing premise is that the natural-language questions RevLLM generates from raw SQL are faithful enough to serve as training labels; the paper states this but reports no human validation rate, no filtering criteria, and no analysis of generation failures.

Editorial extensions

If this is right

  • Reverse data generation turns existing unlabeled SQL in production systems into training data, so specialized NL2SQL models can be built without hand-annotated datasets.
  • A 7B open-source model fine-tuned this way can exceed the execution accuracy of GPT-4-based in-context-learning systems, suggesting domain-specific fine-tuning is more decisive than model size.
  • The workflow decomposition component provides the largest single gain in the ablation, implying that complex business queries benefit from being routed through retrieval and sub-task planning.
  • GPT-Judge's QSE and SSE modes allow evaluation without executable databases or ground-truth SQL, which the paper argues is the common real-world situation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same reverse-generation pipeline could be applied to other logged code-and-comment corpora, such as data transformation scripts or API usage examples, wherever unlabeled code is abundant.
  • A quality filter or human validation rate for RevLLM-generated questions would probably further improve SQLLM accuracy, since the paper reports no measurement of generation failures.
  • QSE/SSE scores could double as a training reward or as automatic filters for selecting synthetic data, rather than only as offline evaluation metrics.
  • The reported +15-16% online gains suggest the offline enterprise benchmark may understate real-world impact, but also that online deployments introduce their own variability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SQLord, an enterprise text-to-SQL framework built on three ideas: reverse data generation (training a RevLLM on <SQL, comment> pairs to synthesize <query, SQL> pseudo-labels for fine-tuning), automated workflow decomposition via retrieval-augmented dynamic sub-task generation, and a GPT-Judge evaluation framework with EXE, QSE, and SSE modes. Offline experiments report 86.5% execution accuracy on a 6,000-pair enterprise dataset (vs. 72.9% for DAIL-SQL with GPT-4) and 87.4% on Spider using Qwen2-7B as the base model. An ablation attributes gains to reverse generation and workflow generation, and an online evaluation reports 15.6–16.7 percentage-point improvements in two real scenarios.

Significance. The combination of synthetic reverse-generated training data, workflow decomposition, and an LLM judge is practically relevant: if the reported numbers are trustworthy, the paper would demonstrate that a 7B open model with domain-specific synthetic data can outperform GPT-4-based pipelines on enterprise text-to-SQL. The reverse-generation idea is reasonable, and the three-mode evaluation framework addresses a real operational need. However, the paper provides no data release, no overlap analysis between the generated training corpus and the test set, no human validation of RevLLM outputs, and no human agreement study for GPT-Judge. These omissions leave the headline gains unverifiable and create a concrete risk of train/test leakage and LLM self-consistency. The work is a strong candidate for an industrial-systems paper, but the evidence as written does not yet support the central claims.

major comments (5)
  1. [Section 3 (Dataset) and Section 2.1] The paper never establishes that the 6,000 manually annotated evaluation pairs are disjoint from the 35,948 pseudo-annotated pairs used to fine-tune SQLLM. Both are drawn from the same two enterprise scenarios (Customs Import-Export Assistant and Intelligent Product Selection), and Section 2.1 says the raw SQL set S comes from routine development without any temporal or query-based separation from the evaluation set. If the evaluation SQLs or near-equivalents appear in Dgen, SQLLM has been trained on the gold answers to the test queries while the GPT-4 and DAIL-SQL baselines have not, and the +13.6 EXE gain in Table 1 would be a memorization artifact rather than evidence of generalization. The authors need to report an explicit overlap analysis (e.g., exact or near-duplicate SQL matching after normalization) and describe how the evaluation set was held out.
  2. [Section 2.1, Eq. (1)] The pseudo-annotation step q_i = RevLLM(s_i; θ) is the load-bearing premise of the SFT pipeline, yet the paper reports no human validation rate, no filtering criteria, and no analysis of generation failures for the 35,948 generated <Query, SQL> pairs. If RevLLM produces questions that admit multiple readings, or if the original SQL is not a correct answer to the generated question, the fine-tuning objective in Eq. (2) teaches SQLLM a hallucinated question–SQL mapping. The authors should provide a human-annotated sample (e.g., 200–500 pairs) with validity/accuracy measurements, describe failure modes, and state any filtering or deduplication applied before training.
  3. [Section 3.3 and Abstract] The online evaluation in Table 4 reports only deltas (+15.6%, +16.4%, etc.) and gives no absolute accuracy values, no number of online test queries, and no definition of 'before and after the launch.' The abstract's claim that 'online accuracy consistently exceeds 90' does not correspond to any number in the paper, so it is unverifiable. The authors should report absolute EXE/QSE/SSE scores, sample sizes, and the exact comparison protocol for the online before/after measurement.
  4. [Section 2.3 and Section 3.1] The GPT-Judge evaluation is performed by an LLM whose judgment criteria are not specified (no prompt template, no temperature, no cutoff threshold), and no human agreement study is reported. The statement in Section 3.1 that 'the consistent differences in QSE and SSE scores (±7%) confirm GPT-Judge's reliability' is not a validity argument: a stable gap between metrics says nothing about whether the judge agrees with human intent. Given that SQLLM is trained on LLM-generated data and judged by an LLM, there is a systematic risk that the reported QSE/SSE gains reflect self-consistency between the generator and the judge. A concrete corrective is to report Cohen's kappa or agreement rate between GPT-Judge and human judges on a subset of at least 200 samples for both QSE and SSE.
  5. [Section 2.2 and Table 3] The workflow-decomposition component is underspecified: the paper does not state how sub-tasks are formulated from the retrieved context, how intermediate results are represented, how SQLLM is prompted per sub-task, or how the final Summary step combines results. This makes the ablation in Table 3 hard to interpret: the row labels do not make clear whether '+ Workflow Generation' is applied on top of reverse generation or on the bare Qwen model, and no Spider ablation is given. Since workflow generation is one of the two components credited with the largest gains, the authors need to specify the decomposition algorithm (including the exact role of d_q and the dependency graph) and clarify the cumulative structure of the ablation.
minor comments (5)
  1. [Section 3 (Dataset)] The paper should describe how the 6,000 manually annotated pairs were produced, including the number of annotators and inter-annotator agreement; without this information, the gold SQL may be seen as arbitrary rather than reproducible.
  2. [Section 3.1, Table 2] The Spider evaluation setup needs clarification: for SQLord on Spider, how are the knowledge base K and TableHub instantiated and retrieved for each of the many databases? This is essential for reproducing the reported 87.4% EXE.
  3. [Section 2.3, EXE definition] The execution comparison I(r_gen = r_gold) should specify the equivalence semantics for result sets, including row ordering, duplicate rows, null values, and floating-point comparisons, all of which are nontrivial in practice.
  4. [Throughout] Several typos and style issues remain, including 'Nature language', 'statments', 'challange', and 'RAG-liked'; these should be corrected.
  5. [References] Reference [1] lacks an author list, and several entries omit venue or publisher details; the reference list should be brought to a consistent, complete format.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Spider result is an external benchmark, and the enterprise claims rest on a data pipeline that is not shown to reduce to its own inputs by construction.

full rationale

The paper's derivation chain is not circular in the restricted sense required here. Reverse data generation trains RevLLM on <SQL, COMMENT> pairs to create pseudo-annotated <Query, SQL> training data, then fine-tunes SQLLM on that data; this is an augmentation pipeline, not a self-referential derivation. The central Spider evaluation (87.4% EXE) is measured against an independent, externally maintained benchmark on which SQLLM was not trained, so the headline comparison to DAIL-SQL and GPT-4 has independent content. On the enterprise dataset, there is a legitimate data-hygiene concern: both the 6,000 manually annotated evaluation pairs and the 35,948 reverse-generated training pairs are drawn from the same two online scenarios and 'routine development,' and the paper never states that evaluation gold SQLs were excluded from the raw SQL pool S. However, this is a potential train/test overlap, not a demonstrated circularity: the paper contains no equation or statement showing that a test gold SQL appears in Dgen, and without that direct reduction the reported 86.5% EXE cannot be shown to be equivalent to the training objective by construction. The GPT-Judge QSE/SSE metrics are auxiliary evaluation signals anchored by execution accuracy, and the claim that their consistency 'confirms GPT-Judge's reliability' is not load-bearing for the central EXE results. No load-bearing self-citation or imported uniqueness theorem appears in the paper. Therefore the appropriate finding is no significant circularity, with the enterprise overlap risk noted as an external validity concern rather than a circularity defect.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entity. Its burden is instead in hidden assumptions about data quality, judge reliability, and the transfer from internal synthetic data to the Spider benchmark. The most significant assumption is that reverse-generated questions are faithful training labels, which is not validated with human judgments.

free parameters (2)
  • Training hyperparameters (3 epochs, learning rate 1e-5, batch size 1) = 3 epochs, 1e-5, batch size 1
    These hyperparameters are stated in Section 3 but there is no sweep or analysis showing they are optimal; they are chosen by hand.
  • Cutoff/threshold choices for the GPT-Judge = not stated
    The judge is described as returning a score, but no decision threshold, calibration, or human-agreement measurement is reported in Section 2.3 or Section 3.
assumptions (3)
  • domain assumption A small set of <SQL, COMMENT> pairs collected from routine development is sufficient to train RevLLM to produce high-quality questions for the large corpus of raw SQL.
    Stated in Section 2.1; the paper provides no quality measurement for the 35,948 generated pairs.
  • domain assumption Running generated SQL on the actual database and comparing execution results is a valid measure of correctness for all 6,000 internal examples.
    Used in Section 3; execution accuracy can be misleading when queries return empty results or when the gold SQL is itself incorrect, and no error analysis is provided.
  • domain assumption The open-source Spider benchmark, trained on by the baselines but not by SQLord's models as far as described, is a meaningful generalization test for the enterprise workflow.
    Reported in Section 3.1; the connection between the domain-specific training recipe and cross-domain Spider performance is asserted without analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SQLord: A Robust Enterprise Text-to-SQL Solution via Reverse Data Generation and Workflow Decomposition." pith.science (2026). https://pith.science/paper/CLVWSQGM

@misc{pith2026250710629,
  author       = {Pith},
  title        = {Pith review of: SQLord: A Robust Enterprise Text-to-SQL Solution via Reverse Data Generation and Workflow Decomposition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CLVWSQGM}},
  note         = {Machine review of arXiv:2507.10629}
}
read the original abstract

Transforming natural language into SQL queries (NL2SQL) is crucial for data-driven business applications. Existing frameworks, trained on open-source datasets, struggle with complex business logic and lack domain-specific data for fine-tuning. Additionally, evaluation methods often require annotated data and executable database environments, which are scarce in real-world scenarios. To address these challenges, we propose SQLord, an enterprise-level NL2SQL framework. First, SQLord introduces a data reverse generation approach to convert raw SQL statements into annotated data for supervised fine-tuning (SFT). Second, it proposes a decomposition method for complex queries using an automated workflow generator. Additionally, SQLord features a comprehensive GPT-Judge evaluation framework, including Execution Evaluation (EXE), Query-SQL Evaluation (QSE), and SQL-SQL Evaluation (SSE), tailored to diverse scenarios. Offline tests significantly outperform state of the art baselines, and online accuracy consistently exceeds 90, highlighting SQLord's advantages and effectiveness in complex real world scenarios. SQLord has been successfully applied across multiple scenarios on the world's largest B2B e-commerce platform.

Figures

Figures reproduced from arXiv: 2507.10629 by the authors.

Figure 1
Figure 1. A real-world online case of NL2SQL challenges [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The SQLord framework: (a) Reverse Data Gener [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 10 canonical work pages

  1. [1]

    Qwen2 Technical Report

    2024. Qwen2 Technical Report. (2024)

  2. [2]

    LILY Group at Yale University. 2018. Spider 1.0, Yale Semantic Parsing and Text-to-SQL Challenge. (2018). https://yale-lily.github.io/spider

  3. [3]

    Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, Lu Chen, Jinshu Lin, and Dongfang Lou. 2023. C3: Zero-shot Text-to-SQL with ChatGPT. CoRR abs/2307.07306 (2023)

  4. [4]

    Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2024. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. Proceedings of the VLDB Endowment 17, 5 (2024)

  5. [5]

    Binyuan Hui, Ruiying Geng, Lihan Wang, Bowen Qin, Yanyang Li, Bowen Li, Jian Sun, and Yongbin Li. 2022. S 2SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parsers. In Findings of the Associa- tion for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022 . Association for Computational Linguistics, 1254–1262

  6. [6]

    Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 13067–13075

  7. [7]

    Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. 2023. Graphix-t5: Mixing pre-trained transformers with graph-aware layers for text-to-sql parsing. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 37. 13076–13084

  8. [8]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What Makes Good In-Context Examples for GPT-3?. In Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO@ACL 2022, Dublin, Ireland and Online, May 27, 2022 . Association for Computa...

Show all 15 references
  1. [9]

    Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuyu Luo, Yuxin Zhang, Ju Fan, Guoliang Li, and Nan Tang. 2024. A Survey of NL2SQL with Large Language Models: Where are we, and where are we going? CoRR abs/2408.05109 (2024)

  2. [10]

    Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. 2023. Enhancing text-to-SQL capabilities of large language models: A study on prompt design strategies. In Findings of the Association for Computational Linguistics: E...

  3. [11]

    OpenAI. 2023. GPT-4 Technical Report. CoRR abs/2303.08774 (2023). https: //doi.org/10.48550/arXiv.2303.08774

  4. [12]

    Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed In- Context Learning of Text-to-SQL with Self-Correction. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA,...

  5. [13]

    Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022. Evaluating the Text-to-SQL Capabilities of Large Language Models.CoRR abs/2204.00498 (2022)

  6. [14]

    Immanuel Trummer. 2022. CodexDB: Synthesizing code for query processing from natural language instructions using GPT-3 Codex. Proceedings of the VLDB Endowment 15, 11, 2921–2928

  7. [15]

    Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.