REVIEW 3 major objections 4 minor 23 references
X-SQL: Expert Schema Linking and Understanding of Text-to-SQL with Multi-LLMs
T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A dedicated schema linker and a plain-language schema explainer lift open-source text-to-SQL to 84.9% execution accuracy on Spider-Dev.
desk verdict Worth reading for the dedicated SFT schema linker, but the headline 84.9/82.5 is a per-split best-of-16 selection, not a single system. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is X-Linking, a schema linker built by supervised fine-tuning: its training objective maximizes the probability of the correct table-name sequence T given a prompt that concatenates candidate schemas S, foreign keys K, and the user question Q. Inference adds a self-consistency step in which the candidate order is shuffled five times and the union of predicted table names becomes the final linked schema. The second mechanism is X-Admin, a role-prompted schema-understanding stage that turns abstract column names and sample values into plain-language definitions and join hints. The argument also depends on a multi-LLM assignment: different backbone models are used for linking, generation, and debugging, with a debugger different from the generator consistently helping, which the paper attributes to LLMs favoring their own outputs.
What would settle it
Reproduce X-Linking with a fully specified sampling procedure for candidate schemas and evaluate table recall and end-to-end accuracy on Spider-Test and on held-out databases not in Spider; if the gain over no linking shrinks materially or table recall drops, the reported 7.3-point improvement is not a general schema-linking capability.
Extended reading notes
Core claim
The central claim is that schema linking and schema understanding are distinct, learnable bottlenecks and that a text-to-SQL system should treat them as separate components rather than as incidental steps in prompting. X-Linking is trained with supervised fine-tuning to output a sequence of table names T given candidate schemas S, foreign keys K, and question Q; at inference the inputs are shuffled five times and the predicted table sets are unioned. X-Admin then takes the linked tables and produces natural-language descriptions of each column and join relationship, so the generator receives a compact, human-readable view instead of raw DDL. The paper reports that removing X-Linking costs 7.3 points of execution accuracy, removing X-Admin costs 1.7 points, and the complete multi-LLM system reaches 84.9% on Spider-Dev and 82.5% on Spider-Test, the highest open-source-model results it compares against.
Load-bearing premise
The load-bearing premise is that the training pairs for X-Linking—candidate schemas, foreign keys, and ground-truth table sets sampled from Spider—represent the schema-linking task broadly; the paper does not describe how those pairs were curated, so the large reported gain could be an artifact of that unstated sampling.
Editorial extensions
If this is right
- X-Linking can be dropped into existing text-to-SQL pipelines as a pre-step; the paper shows it improves SQL generation for four different generator models relative to no linking and approaches oracle-schema accuracy.
- X-Admin needs no fine-tuning and can be added to any framework, giving a gain comparable to the widely used debugging step.
- Longer context windows will increase, not decrease, the value of a dedicated linker, because more irrelevant tables will otherwise crowd the prompt.
- Assigning different LLMs to generation and debugging is a cheap way to improve open-source systems, consistent with the paper's self-preference explanation.
Reading between the lines
- Editorial inference: because the SFT training data for X-Linking is described as 'carefully curated' but not specified, the 7.3-point gain could partly reflect the sampling of candidate schemas rather than a general linking skill; a reproducible curation procedure is needed to test this.
- Editorial inference: X-Admin's mechanism of explaining schema elements in natural language likely transfers to other structured-code tasks such as API selection or dataframe transformation, though the paper does not test those settings.
- Editorial inference: the near-oracle gap in the paper's own table suggests that once schemas are linked well, remaining errors come from generation and debugging, so the next gains should come from those stages rather than from better linking.
- Editorial inference: using a different, possibly weaker, model as the debugger might be as effective and cheaper; the paper reports the benefit of different-backbone debugging but does not explore the cheapest such configuration.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes X-SQL, a modular text-to-SQL system for the Spider benchmark built on open-source LLMs. Its first component, X-Linking, is an SFT-trained schema linker (Q-LoRA on LLMs) that takes a question, candidate schemas, and foreign keys and outputs relevant table names; at inference it unions outputs over five shuffled orderings. The second component, X-Admin, prompts an LLM to explain linked columns in natural language. A third stage generates and debug-fixes SQL. The authors also assign different LLMs to the X-Admin and Debugging components. They report execution accuracies of 84.9% on Spider-Dev and 82.5% on Spider-Test, claiming state-of-the-art performance among open-source frameworks and attributing large gains to schema learning.
Significance. The X-Linking contribution is potentially valuable: Table 4 shows a controlled plug-and-play comparison in which adding X-Linking improves CodeQwen1.5-7B-Chat from 74.3 to 81.6 on Spider-Dev, and Table 5 identifies X-Linking as the largest component, with a 7.3% drop when it is removed. Unlike prior schema-linking modules, the method uses dedicated SFT and is evaluated externally on Spider with recall metrics, so the core X-Linking result is not circular. If the training-data construction is disclosed and the per-split model-selection issue is resolved, the framework and ablations would be a useful addition to the open-source text-to-SQL literature. The X-Admin idea is also clearly motivated and shows a modest but positive ablation. The paper's strengths are its controlled component ablations and its use of an external benchmark rather than a self-constructed evaluation.
major comments (3)
- [Abstract; Table 2; Table 6; Table 7; Figure 1 caption] The reported headline pair 84.9/82.5 is not achieved by any single system configuration. In Table 6, the row CQ+DC scores 84.9 on Spider-Dev but 81.7 on Spider-Test, while the row QW+LS scores 83.4 on Spider-Dev and 82.5 on Spider-Test; no row in Table 7 attains both values. The abstract and Table 2 present these numbers together as X-SQL's performance, and Figure 1's caption states that the LLM setup is based on the best Spider-Test result, which was selected after running all 16 combinations in Appendix C. This is per-split, test-set-driven model selection, so the claim that X-SQL is the leading open-source framework and the Multi-LLM gains of 1.3%/2.2% are not supported as stated. Please either report a single pre-specified configuration for both splits or explicitly present the results as an upper envelope, and re-evaluate the Multi-LLM comparison against a fixed baseline rather than per-split maxima.
- [X-Linking (Schema Linking); Appendix B] The manuscript states 'We carefully curate the dataset for SFT Schema Linking training' but never describes how the candidate table schemas S, foreign keys K, and ground-truth table sets T are constructed from Spider. Appendix B gives only Q-LoRA hyperparameters and hardware details. This omission is load-bearing because the central 7.3% ablation gain depends on the training distribution: if candidate tables are sampled from the same databases as the evaluation questions, or if the negative candidates are unrepresentative, the reported improvements may not transfer. Please specify the data construction procedure, including the source of negative examples, the number of training instances, and whether Spider-train databases overlap with the dev and test databases.
- [Experiments; Tables 5-6] No significance tests or measures of variation are reported, even though the paper states that all experiments are conducted three times and average results are reported. Key comparisons in Tables 5 and 6 involve differences of 0.8-1.7 percentage points, which could plausibly be within run-to-run noise for stochastic LLM inference with self-consistency and debugging. Please report standard deviations or confidence intervals, or run paired significance tests, for at least the headline comparisons and the Multi-LLM claims.
minor comments (4)
- [X-Admin (Schema Understanding)] There is a typo in the section heading: 'Schmea Understanding' should be 'Schema Understanding'.
- [Tables 1, 3, 6, and Appendix C] The backbone model name is inconsistent: Tables 1 and 3 refer to CodeQwen1.5-7B-Instruct for X-Linking, while the text in 'X-Linking Ablation Studies' and Table 6 refer to CodeQwen1.5-7B-Chat; Appendix C defines CQ as CodeQwen1.5-7B-Instruct. Please standardize the model naming throughout.
- [End-to-end Text-to-SQL Results] The sentence 'We use the its best result for analysis' contains a grammatical error; it should read 'We use its best result for analysis.'
- [Reproducibility Checklist] The checklist states that source code will be made publicly available, but no repository link or release plan is given in the paper; please add a concrete availability statement, along with the seed handling and software versions noted as partial in the checklist.
Circularity Check
Headline 84.9/82.5 is assembled from per-split best LLM rows (CQ+DC on Dev, QW+LS on Test); no single X-SQL configuration achieves both, making the Multi-LLM SOTA claim partly a selection artifact.
-
fitted input called prediction
[Abstract; Section 'Multi-LLMs Ablation Studies' (Table 6); Appendix C (Table 7).]
"we present only the top-performing results for the Multi-LLM system in Table 6, with the complete experimental results detailed in Appendix C. ... We present the best results, specifically the CQ + DC and QW + LS results, in the “Multi-LLMs Ablation Studies” section of the main paper. [Table 6 rows: CodeQwen1.5-7B-Chat + deepseek-coder-7b-instruct-v1.5 84.9 81.7; Qwen2-7B-Instruct + llama-3-sqlcoder-8B 83.4 82.5]"
The claimed system-level result is not an evaluation of one fixed LLM assignment. Appendix C enumerates all 16 X-Admin+Debugging combinations; the abstract's 84.9 on Spider-Dev is the Dev maximum (CQ+DC, which scores 81.7 on Test), while 82.5 on Spider-Test is the Test maximum (QW+LS, which scores 83.4 on Dev). No single configuration attains both numbers, so the reported X-SQL result is defined, per split, as the argmax of the evaluated grid rather than as the accuracy of a fixed framework. The claimed Multi-LLM gain of 1.3%/2.2% is also computed by comparing this selected maximum against a single fixed baseline (CQ+CQ), so the improvement is partly guaranteed by the max operation and is not independent evidence of a generalizable Multi-LLM design property.
full rationale
The paper's core schema-linking and schema-understanding derivations are not circular: X-Linking is an SFT model trained on Spider's training split and evaluated with held-out Re/Rs metrics against DIN-SQL, MAC-SQL, and PET-SQL, and X-Admin is ablated on the same fixed pipeline; these results rest on external benchmark evidence rather than on the paper's own definitions. I found no self-citation chains, no imported uniqueness theorems, and no ansatz smuggled in by citation. The one load-bearing circular element is the Multi-LLM/SOTA result: the reported 84.9 (Dev) and 82.5 (Test) are the per-split maxima of a 16-configuration grid (Table 7), with Dev's best row scoring only 81.7 on Test and Test's best row scoring only 83.4 on Dev. Thus the headline pair describes no single reproducible configuration, and the claimed Multi-LLM gain is computed by comparing the selected maximum against one fixed baseline, so the improvement is partly forced by the selection procedure. This makes the SOTA claim partially circular, although the X-Linking and X-Admin contributions retain independent empirical content. The undocumented X-Linking SFT training-data curation is a reproducibility and validity concern, not a circularity concern.
Assumptions & free parameters
free parameters (6)
- Number of self-consistency shuffles =
5
- X-Admin backbone LLM =
deepseek-coder-7b-instruct-v1.5 (Dev) / llama-3-sqlcoder-8B (Test)
- Debugging backbone LLM =
deepseek-coder-7b-instruct-v1.5 (Dev) / llama-3-sqlcoder-8B (Test)
- Schema Linking SFT backbone =
CodeQwen1.5-7B-Chat
- SQL Generation backbone =
CodeQwen1.5-7B-Chat
- Q-LoRA hyperparameters =
rank=128, alpha=256, lr=1e-4, batch=32, epoch=1
assumptions (5)
- domain assumption Spider is a representative benchmark for cross-domain Text-to-SQL.
- domain assumption Execution Accuracy on SQLite is a valid proxy for SQL query correctness.
- domain assumption Fine-tuning the schema linker with Q-LoRA on the schema-linking task does not degrade the base model's ability in other components.
- ad hoc to paper Taking the union of five shuffled outputs yields the true set of relevant tables.
- ad hoc to paper Three averaged runs are sufficient for the reported accuracy differences.
Cite this review
Pith. "Pith review of X-SQL: Expert Schema Linking and Understanding of Text-to-SQL with Multi-LLMs." pith.science (2026). https://pith.science/paper/24OCBZDT
@misc{pith2026250905899,
author = {Pith},
title = {Pith review of: X-SQL: Expert Schema Linking and Understanding of Text-to-SQL with Multi-LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/24OCBZDT}},
note = {Machine review of arXiv:2509.05899}
}
read the original abstract
With Large Language Models' (LLMs) emergent abilities on code generation tasks, Text-to-SQL has become one of the most popular downstream applications. Despite the strong results of multiple recent LLM-based Text-to-SQL frameworks, the research community often overlooks the importance of database schema information for generating high-quality SQL queries. We find that such schema information plays a significant or even dominant role in the Text-to-SQL task. To tackle this challenge, we propose a novel database schema expert with two components. We first introduce X-Linking, an LLM Supervised Finetuning (SFT)-based method that achieves superior Schema Linking results compared to existing open-source Text-to-SQL methods. In addition, we innovatively propose an X-Admin component that focuses on Schema Understanding by bridging the gap between abstract schema information and the user's natural language question. Aside from better learning with schema information, we experiment with Multi-LLMs for different components within the system to further boost its performance. By incorporating these techniques into our end-to-end framework, X-SQL, we have achieved Execution Accuracies of 84.9% on the Spider-Dev dataset and 82.5% on the Spider-Test dataset. This outstanding performance establishes X-SQL as the leading Text-to-SQL framework based on open-source models.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[3]
Feng, S.; Shi, W.; Wang, Y .; Ding, W.; Balachandran, V .; and Tsvetkov, Y
C3: Zero-shot text-to-sql with chatgpt.arXiv preprint arXiv:2307.07306. Feng, S.; Shi, W.; Wang, Y .; Ding, W.; Balachandran, V .; and Tsvetkov, Y
-
[4]
Gan, Y .; Chen, X.; Xie, J.; Purver, M.; Woodward, J
Don’t Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration.arXiv preprint arXiv:2402.00367. Gan, Y .; Chen, X.; Xie, J.; Purver, M.; Woodward, J. R.; Drake, J.; and Zhang, Q
-
[6]
Guo, D.; Zhu, Q.; Yang, D.; Xie, Z.; Dong, K.; Zhang, W.; Chen, G.; Bi, X.; Wu, Y .; Li, Y .; et al
Text-to-sql empowered by large lan- guage models: A benchmark evaluation.arXiv preprint arXiv:2308.15363. Guo, D.; Zhu, Q.; Yang, D.; Xie, Z.; Dong, K.; Zhang, W.; Chen, G.; Bi, X.; Wu, Y .; Li, Y .; et al
-
[7]
Lee, D.; Park, C.; Kim, J.; and Park, H
DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence.arXiv preprint arXiv:2401.14196. Lee, D.; Park, C.; Kim, J.; and Park, H
-
[8]
Li, H.; Zhang, J.; Li, C.; and Chen, H
Mcs-sql: Leveraging multiple prompts and multiple-choice selection for text-to-sql generation.arXiv preprint arXiv:2405.07467. Li, H.; Zhang, J.; Li, C.; and Chen, H
-
[11]
Pezeshkpour, P.; and Hruschka, E
Llm evaluators recognize and favor their own generations.arXiv preprint arXiv:2404.13076. Pezeshkpour, P.; and Hruschka, E
-
[12]
Large language models sensitivity to the order of options in multiple-choice questions.arXiv preprint arXiv:2308.11483. Pourreza, M.; and Rafiei, D
- [14]
Show all 23 references
-
[15]
Saxena, A
Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950. Saxena, A
-
[16]
Stratica, N.; Kosseim, L.; and Desai, B
Small llms are weak tool learners: A multi-llm agent.arXiv preprint arXiv:2401.07324. Stratica, N.; Kosseim, L.; and Desai, B. C
-
[17]
Wang, B.; Ren, C.; Yang, J.; Liang, X.; Bai, J.; Zhang, Q.- W.; Yan, Z.; and Li, Z
Chess: Contextual harnessing for efficient sql synthesis.arXiv preprint arXiv:2405.16755. Wang, B.; Ren, C.; Yang, J.; Liang, X.; Bai, J.; Zhang, Q.- W.; Yan, Z.; and Li, Z
-
[18]
Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D
Mac-sql: Multi-agent collabo- ration for text-to-sql.arXiv preprint arXiv:2312.11242. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D
-
[19]
arXiv preprint arXiv:2203.11171
Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171. Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; et al
-
[20]
Yu, T.; Zhang, R.; Yang, K.; Yasunaga, M.; Wang, D.; Li, Z.; Ma, J.; Li, I.; Yao, Q.; Roman, S.; et al
Qwen2 Technical Report.arXiv preprint arXiv:2407.10671. Yu, T.; Zhang, R.; Yang, K.; Yasunaga, M.; Wang, D.; Li, Z.; Ma, J.; Li, I.; Yao, Q.; Roman, S.; et al
-
[22]
###Needed schema names
Seq2sql: Gen- erating structured queries from natural language using rein- forcement learning.arXiv preprint arXiv:1709.00103. Reproducibility Checklist This paper • Includes a conceptual outline and/or pseudocode descrip- tion of AI methods introduced:Y es • Clearly delineate...
-
[23]
Experiments
• Epoch:1 Encoder-Decoder Hyperparameter Settings:Following the methodology described in the original Flan-T5 paper (Chung et al. 2024), we adapt our learning rate scheduler to use AdaFactor (Shazeer and Stern 2018). Furthermore, we increase the number of training epochs to mi...
2024
-
[36]
Li, Z.; Wang, X.; Zhao, J.; Yang, S.; Du, G.; Hu, X.; Zhang, B.; Ye, Y .; Li, Z.; Zhao, R.; et al. 2024b. PET-SQL: A Prompt-enhanced Two-stage Text-to-SQL Framework with Cross-consistency.arXiv preprint arXiv:2403.09732. Liu, N. F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacq...
-
[2017]
Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101. OpenAI
-
[2018]
Zhong, V .; Xiong, C.; and Socher, R
Spider: A large-scale human-labeled dataset for complex and cross- domain semantic parsing and text-to-sql task.arXiv preprint arXiv:1809.08887. Zhong, V .; Xiong, C.; and Socher, R
-
[2021]
Gao, D.; Wang, H.; Li, Y .; Sun, X.; Qian, Y .; Ding, B.; and Zhou, J
Natural SQL: Making SQL easier to infer from natural language specifications.arXiv preprint arXiv:2109.05153. Gao, D.; Wang, H.; Li, Y .; Sun, X.; Qian, Y .; Ding, B.; and Zhou, J
-
[2022]
Qu, G.; Li, J.; Li, B.; Qin, B.; Huo, N.; Ma, C.; and Cheng, R
A survey on text-to- sql parsing: Concepts, methods, and future directions.arXiv preprint arXiv:2208.13629. Qu, G.; Li, J.; Li, B.; Qin, B.; Huo, N.; Ma, C.; and Cheng, R
-
[2023]
Cen, J.; Liu, J.; Li, Z.; and Wang, J
Gpt-4 technical report.arXiv preprint arXiv:2303.08774. Cen, J.; Liu, J.; Li, Z.; and Wang, J
-
[2024]
Chung, H
SQLFixAgent: Towards Semantic-Accurate SQL Generation via Multi- Agent Collaboration.arXiv preprint arXiv:2406.13408. Chung, H. W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y .; Fe- dus, W.; Li, Y .; Wang, X.; Dehghani, M.; Brahma, S.; et al
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.