Pith. sign in

REVIEW 2 major objections 2 minor 24 references

Schema structure changes cause LLMs to generate different SQL answers even when the underlying data and questions stay the same.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 19:27 UTC pith:V2535XHT

load-bearing objection The paper shows LLMs for text-to-SQL give inconsistent answers across structurally different but supposedly equivalent schemas, using a new E/R-based generation method. the 2 major comments →

arxiv 2605.25838 v1 pith:V2535XHT submitted 2026-05-25 cs.DB

Same Data, Different Schemas: Robustness of LLM-based Text-to-SQL

Nitin Kanchinadam , Aditya Menachery , Amol Deshpande This is my paper
classification cs.DB
keywords text-to-SQLLLM robustnessschema variationsequivalent schemasE/R model shreddingdatabase benchmarks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a method to create multiple relational schemas that represent identical information by starting from one entity-relationship diagram and applying different shredding choices. It then runs the same natural-language questions and data against four leading LLMs on these variants and measures how often the produced SQL queries return different results. The experiments show that schema differences alone drive large inconsistencies in model outputs. Supplying the original entity-relationship diagram as extra context reduces but does not eliminate the inconsistencies. The framework also yields a systematic way to produce many synthetic training examples for future models.

Core claim

By generating conceptually equivalent schemas through controlled variation of shredding choices from a shared E/R model, the authors demonstrate that LLM text-to-SQL systems produce SQL queries whose results differ substantially across those schemas even though the natural language questions and stored data remain fixed.

What carries the argument

A framework that translates one E/R model into multiple relational schemas by varying shredding choices, keeping semantics identical while changing structure.

Load-bearing premise

Different shredding choices produce schemas that differ only in structure while preserving exactly the same meaning, so any change in LLM output can be blamed on the schema rather than on hidden semantic differences.

What would settle it

Run the same questions on two schema variants generated from the same E/R model and observe that every LLM returns identical result sets for every question.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Current text-to-SQL benchmarks that use only one schema per domain understate the robustness problem.
  • Providing the original E/R diagram as context improves consistency but leaves residual inconsistencies.
  • Systematic schema variation can be used to generate large numbers of synthetic training examples.
  • One practical mitigation is to produce multiple candidate SQL queries by querying the same question against several schema variants.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the effect holds across more domains, text-to-SQL systems may need explicit mechanisms to normalize or canonicalize schema presentation before inference.
  • The same framework could be applied to test whether other structured-prediction tasks, such as semantic parsing, show comparable sensitivity to input representation.
  • Training on data augmented by schema variants might reduce the observed inconsistencies without requiring changes at inference time.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper introduces a framework to evaluate LLM robustness in text-to-SQL by generating multiple structurally distinct relational schemas from a shared E/R model via different shredding choices. These variants are claimed to preserve identical semantics while the NL questions and underlying data remain fixed. The authors evaluate four LLMs on two domains, report frequent inconsistencies in generated SQL answers across schemas via pairwise heatmaps, note that E/R context helps but does not eliminate issues, and suggest the approach for synthetic data generation and robustness improvement.

Significance. If the semantic equivalence of the schema variants is established, the work would usefully expose a gap in existing text-to-SQL benchmarks and provide a controlled method for testing structural sensitivity. The heatmap-based consistency analysis and the proposal to use systematic schema variation for training data are constructive contributions.

major comments (2)
  1. [Section 3] Section 3 (Framework): The description of schema generation via shredding does not include an explicit verification procedure (e.g., canonical query generation per NL question, cross-schema result-set comparison, or data-mapping checks) to confirm that all variants preserve identical semantics for the evaluated questions. Without this, differences in LLM outputs cannot be attributed solely to structure.
  2. [Section 4] Section 4 (Experiments): The consistency claims and heatmaps rest on the unverified equivalence assumption; if any shredding choice alters effective semantics (e.g., via nullability or join-table effects), the reported answer divergences may reflect semantic mismatch rather than robustness failure.
minor comments (2)
  1. [Results section] The heatmaps in the results section would benefit from explicit axis labels, a color-scale legend, and a definition of the consistency metric used.
  2. A short table summarizing the number of schema variants, questions, and domains would improve readability of the experimental setup.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback highlighting the importance of explicit semantic equivalence verification. We address each major comment below and will revise the manuscript accordingly to strengthen the framework description and experimental claims.

read point-by-point responses

  1. Referee: [Section 3] Section 3 (Framework): The description of schema generation via shredding does not include an explicit verification procedure (e.g., canonical query generation per NL question, cross-schema result-set comparison, or data-mapping checks) to confirm that all variants preserve identical semantics for the evaluated questions. Without this, differences in LLM outputs cannot be attributed solely to structure.

    Authors: We agree that an explicit verification procedure would make the equivalence claim more rigorous. The current manuscript relies on equivalence by construction: all schemas are derived from the same E/R model via different shredding choices, which by definition preserve the underlying conceptual semantics and data. However, to directly address the concern, we will add a new subsection in Section 3 describing a verification procedure. This will include (1) generating canonical SQL queries for each NL question against a reference schema, (2) executing them on all variants to confirm identical result sets, and (3) data-mapping checks to ensure no information loss from shredding choices such as nullability or join tables. This addition will allow readers to confirm that observed divergences are due to schema structure rather than semantic mismatch. revision: yes

  2. Referee: [Section 4] Section 4 (Experiments): The consistency claims and heatmaps rest on the unverified equivalence assumption; if any shredding choice alters effective semantics (e.g., via nullability or join-table effects), the reported answer divergences may reflect semantic mismatch rather than robustness failure.

    Authors: We acknowledge that without explicit verification, the consistency analysis could be confounded. As noted in our response to the Section 3 comment, we will incorporate the verification procedure into the revised manuscript. Once added, the heatmaps and consistency metrics in Section 4 will be supported by confirmed semantic equivalence across all schema variants for the evaluated questions and domains. We will also include a brief discussion of potential edge cases (e.g., nullability effects) and how the verification mitigates them. This revision ensures the reported LLM inconsistencies can be confidently attributed to structural sensitivity. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical framework is self-contained.

full rationale

The paper introduces an evaluation framework that generates schema variants from a shared E/R model via shredding choices, then measures LLM consistency on fixed NL questions and data. No equations, fitted parameters, predictions, or derivations are present. The central claim rests on the stated construction that variants preserve semantics (an assumption, not a reduction to prior fitted results or self-citations). No load-bearing self-citation chains or ansatzes appear. The method is presented as an independent benchmarking tool rather than a result derived from its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract supplies no explicit free parameters, axioms, or invented entities; the central claim rests on the unelaborated premise that shredding variants preserve semantics exactly.

pith-pipeline@v0.9.1-grok · 5808 in / 1081 out tokens · 22249 ms · 2026-06-29T19:27:10.240777+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Same Data, Different Schemas: Robustness of LLM-based Text-to-SQL." pith.science (2026). https://pith.science/paper/V2535XHT

@misc{pith2026260525838,
  author       = {Pith},
  title        = {Pith review of: Same Data, Different Schemas: Robustness of LLM-based Text-to-SQL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V2535XHT}},
  note         = {Machine review of arXiv:2605.25838}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models (LLMs) consistently achieve strong results on text-to-SQL benchmarks, but their robustness to schema variations remains poorly understood. Recent work suggests that the schema structure matters, but does not provide a clear and systematic way to evaluate model behavior when different schemas represent the same underlying data. We address this problem by presenting a framework to evaluate and benchmark text-to-SQL techniques over equivalent relational schemas generated from a common E/R model. By varying the ``shredding'' choices used to translate the conceptual design into relations, we create multiple schema variants that differ structurally while preserving the same underlying semantics. This gives us a controlled setting in which the natural language questions and data remain fixed, and only the schema changes. We use this framework to evaluate four leading LLMs on the same questions across multiple schema variants (for two separate domains), and summarize consistency patterns using pairwise comparison heatmaps. Our results show that schema structure significantly affects LLM behavior: across conceptually equivalent schemas, models often produce SQL queries with very different answers. We also find that providing additional context (specifically, the original E/R specification) improves the performance, but does not fully ameliorate the inconsistencies. In addition to demonstrating that the current text-to-SQL evaluations miss an important notion of robustness, our framework provides a way to generate a large number of synthetic datasets that can be used to train new models, and suggests a mechanism to make text-to-SQL more robust by generating additional candidate plans for a given natural language query through systematic schema variations.

Figures

Figures reproduced from arXiv: 2605.25838 by Aditya Menachery, Amol Deshpande, Nitin Kanchinadam.

Figure 1
Figure 1. Figure 1: Overall benchmark generation and evaluation pipeline. Most of the steps, including initial E/R diagram, can themselves [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Social Media E/R diagram used as a running example. We omit relationship names for brevity. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Pairwise agreement heatmaps for all models for [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Agreement by question across schemas for each [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Pairwise agreement heatmaps for all models for [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Pairwise agreement heatmaps for all models for the [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: E/R Diagram for the Retail Dataset (11) List the 5 customers with the highest total spend. Define total spend as the sum of the prices of all products that appear in orders the customer has placed: for each order the customer has placed, add the price of every product in that order (a product appearing in two different orders of the same customer is counted twice, once per order). Return the customer’s fir… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 6 canonical work pages

  1. [1]

    Bench- marking and improving text-to-SQL generation under ambiguity

    Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. Bench- marking and improving text-to-SQL generation under ambiguity. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7053–7074, 2023

    2023

  2. [2]

    LiveSQLBench: A dynamic and contamination-free benchmark for evaluating LLMs on real-world text-to-SQL tasks

    BIRD-Team. LiveSQLBench: A dynamic and contamination-free benchmark for evaluating LLMs on real-world text-to-SQL tasks. https://github.com/bird- bench/livesqlbench, 2025. Accessed: 2026-04-05. Same Data, Different Schemas: Robustness of LLM-based Text-to-SQL aiDM ’26, May 31-June 05, 2026, Bengaluru, India

    2025

  3. [3]

    Reverse engineering of relational databases: Extraction of an EER model from a relational database

    Roger HL Chiang, Terence M Barron, and Veda C Storey. Reverse engineering of relational databases: Extraction of an EER model from a relational database. Data & knowledge engineering, 12(2):107–142, 1994

    1994

  4. [4]

    Beyond relations: A case for elevating to the entity-relationship abstraction

    Amol Deshpande. Beyond relations: A case for elevating to the entity-relationship abstraction. InProceedings of the Conference on Innovative Data Systems Research (CIDR), 2025

    2025

  5. [5]

    AmbiSQL: interactive ambiguity detection and resolution for text-to-SQL.arXiv preprint arXiv:2508.15276, 2025

    Zhongjun Ding, Yin Lin, Tianjing Zeng, Rong Zhu, Bolin Ding, and Jingren Zhou. AmbiSQL: interactive ambiguity detection and resolution for text-to-SQL.arXiv preprint arXiv:2508.15276, 2025

    work page arXiv 2025

  6. [6]

    Evaluating the data model robustness of text-to-SQL systems based on real user queries.arXiv preprint arXiv:2402.08349, 2024

    Jonathan Fürst, Catherine Kosten, Farhad Nooralahzadeh, Yi Zhang, and Kurt Stockinger. Evaluating the data model robustness of text-to-SQL systems based on real user queries.arXiv preprint arXiv:2402.08349, 2024

    work page arXiv 2024

  7. [7]

    Next-generation database interfaces: A survey of LLM-based text-to-SQL.IEEE Transactions on Knowledge and Data Engineering, 2025

    Zijin Hong, Zheng Yuan, Q Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. Next-generation database interfaces: A survey of LLM-based text-to-SQL.IEEE Transactions on Knowledge and Data Engineering, 2025

    2025

  8. [8]

    BIRD-INTERACT: Re-imagining text-to-SQL evaluation via lens of dy- namic interactions

    Nan Huo, Xiaohan Xu, Jinyang Li, Per Jacobsson, Shipei Lin, Bowen Qin, Binyuan Hui, Xiaolong Li, Ge Qu, Shuzheng Si, Linheng Han, Edward Alexander, Xin- tong Zhu, Rui Qin, Ruihan Yu, Yiyao Jin, Feige Zhou, Weihao Zhong, Yun Chen, Hongyu Liu, Chenhao Ma, Fatma Ozcan, Yannis Papakonstantinou, and Reynold Cheng. BIRD-INTERACT: Re-imagining text-to-SQL evalua...

    2026

  9. [9]

    A survey on deep learning approaches for text-to-SQL.The VLDB Journal, 32(4):905–936, 2023

    George Katsogiannis-Meimarakis and Georgia Koutrika. A survey on deep learning approaches for text-to-SQL.The VLDB Journal, 32(4):905–936, 2023

    2023

  10. [10]

    Exploring database normalization effects on SQL generation

    Ryosuke Kohita. Exploring database normalization effects on SQL generation. InProceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 5788–5796, 2025

    2025

  11. [11]

    Re-examining the role of schema linking in text-to-SQL

    Wenqiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua. Re-examining the role of schema linking in text-to-SQL. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6943–6954, 2020

    2020

  12. [12]

    Codes: Towards building open- source language models for text-to-SQL.ACM SIGMOD, 2(3):1–28, 2024

    Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. Codes: Towards building open- source language models for text-to-SQL.ACM SIGMOD, 2(3):1–28, 2024

    2024

  13. [13]

    Exploring schema generalizability of text-to-SQL

    Jieyu Li, Lu Chen, Ruisheng Cao, Su Zhu, Hongshen Xu, Zhi Chen, Hanchong Zhang, and Kai Yu. Exploring schema generalizability of text-to-SQL. InFindings of the Association for Computational Linguistics: ACL 2023, pages 1344–1360, 2023

    2023

  14. [14]

    Can LLM already serve as a database interface? a BIg bench for large-scale database grounded text-to-SQLs

    Jinyang Li, Binyuan Hui, GE QU, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can LLM already serve as a database interface? a BIg bench for large-scale database grounded text-to-SQLs. InNeurIPS Datasets and Benchmarks Track, 2023

    2023

  15. [15]

    A survey of text-to-sql in the era of llms: Where are we, and where are we going?, 2025

    Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuyu Luo, Yuxin Zhang, Ju Fan, Guoliang Li, and Nan Tang. A survey of NL2SQL with large language models: Where are we, and where are we going.arXiv preprint arXiv:2408.05109, 2024

    work page arXiv 2024

  16. [16]

    NL2SQL-bugs: A benchmark for detecting semantic errors in NL2SQL translation

    Xinyu Liu, Shuyu Shen, Boyan Li, Nan Tang, and Yuyu Luo. NL2SQL-bugs: A benchmark for detecting semantic errors in NL2SQL translation. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 5662–5673, 2025

    2025

  17. [17]

    arXiv preprint arXiv:2408.07702 , year=

    Karime Maamari, Fadhil Abubaker, Daniel Jaroslawicz, and Amine Mhedhbi. The death of schema linking? text-to-SQL in the age of well-reasoned language models.arXiv preprint arXiv:2408.07702, 2024

    work page arXiv 2024

  18. [18]

    arXiv preprint arXiv:2410.01943

    Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O Arik. Chase-SQL: Multi-path reasoning and preference optimized candidate selection in text-to-SQL.arXiv preprint arXiv:2410.01943, 2024

    work page arXiv 2024

  19. [19]

    Din-SQL: Decomposed in-context learning of text-to-SQL with self-correction.Advances in neural information processing systems, 36:36339–36348, 2023

    Mohammadreza Pourreza and Davood Rafiei. Din-SQL: Decomposed in-context learning of text-to-SQL with self-correction.Advances in neural information processing systems, 36:36339–36348, 2023

    2023

  20. [20]

    Generating test data for killing SQL mutants: A constraint- based approach

    Shetal Shah, S Sudarshan, Suhas Kajbaje, Sandeep Patidar, Bhanu Pratap Gupta, and Devang Vira. Generating test data for killing SQL mutants: A constraint- based approach. In2011 IEEE 27th International Conference on Data Engineering, pages 1175–1186. IEEE, 2011

    2011

  21. [21]

    A survey on employing large language models for text-to-SQL tasks.ACM Computing Surveys, 2024

    Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. A survey on employing large language models for text-to-SQL tasks.ACM Computing Surveys, 2024

    2024

  22. [22]

    Rat-SQL: Relation-aware schema encoding and linking for text- to-SQL parsers

    Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. Rat-SQL: Relation-aware schema encoding and linking for text- to-SQL parsers. InProceedings of the 58th annual meeting of the association for computational linguistics, pages 7567–7578, 2020

    2020

  23. [23]

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, ...

    2018

  24. [24]

    Evoschema: Towards text-to-SQL robustness against schema evolution.arXiv preprint arXiv:2603.10697, 2026

    Tianshu Zhang, Kun Qian, Siddhartha Sahai, Yuan Tian, Shaddy Garg, Huan Sun, and Yunyao Li. Evoschema: Towards text-to-SQL robustness against schema evolution.arXiv preprint arXiv:2603.10697, 2026. A Retail Dataset Figure 7 shows the E/R diagram for the retail dataset. The natural- language questions we use for this database are: (1) For each department, ...

    work page arXiv 2026