Pith. sign in

REVIEW 4 major objections 6 minor 21 references

A five-stage NL-to-SQL pipeline lifts DataBench QA accuracy from 26% to 70.5% with GPT-4o.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A five-stage NL-to-SQL pipeline with GPT-4o achieves 70.5% on DataBench QA and 71.6% on DataBench Lite QA, beating baselines of 26% and 27%.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Plausible SemEval numbers, but the evidence is a self-reported table; worth refereeing for the reproducibility ask. the 4 major comments →

arxiv 2509.09234 v1 pith:7RUC6QUK submitted 2025-09-11 cs.CL

Agentic LLMs for Question Answering over Tabular Data

Rishit Tyagi , Mohit Gupta , Rahul Bouri This is my paper
classification cs.CL
keywords table question answeringNL-to-SQLLLM agentsDataBenchchain-of-thought promptinganswer verificationSQL generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a multi-stage NL-to-SQL pipeline—embedding-based example selection, SQL generation, chain-of-thought answer extraction, format verification, and reprocessing—can answer natural-language questions over real-world tables far more accurately than the provided baseline. The headline empirical claim is 70.5% accuracy on DataBench QA and 71.6% on DataBench Lite QA with GPT-4o, compared with baseline scores of 26% and 27%. The authors attribute the gains to the example-selection mechanism, CoT reasoning during extraction, and a verification loop that catches malformed or irrelevant outputs before a second, value-focused SQL pass. They report that the same pipeline lifts GPT-4o-mini and DeepSeek v2:16b above baseline, and they acknowledge remaining weaknesses in multi-step numerical reasoning and categorical selection. If accurate, the results show that structured agentic pipelines, not just larger models, are a promising direction for table QA.

Core claim

The central discovery is that a five-stage pipeline—example selection, SQL generation, chain-of-thought answer extraction, verification, and reprocessing—turns a capable LLM into a much stronger table-answerer. With GPT-4o, the pipeline reaches 70.50% on DataBench QA and 71.65% on DataBench Lite QA, against baseline scores of 26% and 27%. The same pipeline lifts GPT-4o-mini to 60.34%/61.49% and DeepSeek v2:16b to 39.68%/45.78%, indicating the mechanism rather than model size drives the improvement.

What carries the argument

The load-bearing mechanism is the verification–reprocessing loop: a GPT-4o judge that rejects answers on format or relevance grounds, followed by a second SQL generation pass that requests exact values rather than entire rows. The loop is complemented by embedding-based example selection—two closest examples from a 25-pair pool, retrieved by cosine similarity with text-embedding-ada-002—and chain-of-thought prompting during extraction. The verification step is deliberately non-factual: it checks whether the answer looks like the expected type and is on-topic, defaulting borderline cases to acceptance, so it catches formatting failures and irrelevant retrieval, not numeric correctness.

Load-bearing premise

The central claim rests on the premise that the accuracy numbers in Table 1 reflect a single, stable evaluation on the full benchmark under the official scoring pipeline; because no seeds, temperatures, repeated runs, test-set size, or normalization details are given, the point estimate could shift under replication.

What would settle it

Look up the submitted run's score on the official DataBench leaderboard for the same model and pipeline; if the leaderboard accuracy deviates materially from 70.5% (or if the run is not listed), the central claim is not reproduced. Alternatively, run the pipeline on the same benchmark twice with different random seeds and a fixed evaluation harness; a swing larger than a few accuracy points would indicate the reported number is unstable.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the reported numbers hold, the 26%-to-70.5% jump implies that careful pipeline design can extract near-practical value from proprietary LLMs on table QA without fine-tuning.
  • Because GPT-4o-mini and DeepSeek v2:16b also beat the baseline, the pipeline should generalize to other models, and even open-weight models become useful for table QA.
  • The verification/reprocessing design suggests that spending inference budget on a second, refocused SQL pass is more effective than a single high-temperature run, though the paper does not directly test that comparison.
  • The paper's own error analysis points to multi-step numerical reasoning and categorical misclassification as the main residual weaknesses, so near-term gains should come from targeted numerical-reasoning prompts rather than further format enforcement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The absence of repeated runs, seeds, or confidence intervals makes the 70.5% figure a point estimate; a few percent of sampling variance is plausible and could change the ranking. Reproducing under the benchmark's official scoring would settle this.
  • An ablation that removes the verification step, or replaces cosine-selected examples with random ones, would directly test the paper's attribution of the gains to its multi-stage design; without it, the contribution of each stage is unknown.
  • Because verification checks format and relevance rather than factual correctness, this pipeline might excel at formatted but numerically wrong answers—consistent with the paper's admitted numerical-reasoning failures. A fact-checking or self-consistency stage could be a natural next step.
  • The 25-example pool and two-example retrieval are arbitrary hyperparameters; the approach's sensitivity to example count, embedding model, and similarity threshold is untested and could be explored on the same benchmark.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents a five-stage NL-to-SQL pipeline for SemEval 2025 Task 8 (DataBench). The stages are example selection (25 curated Q/SQL pairs, top-2 via cosine similarity), SQL generation, answer extraction, verification, and reprocessing. Using GPT-4o, GPT-4o-mini, and deepseek-v2:16b, the authors report accuracies in Table 1, with GPT-4o at 70.50% on DataBench and 71.65% on DataBench-Lite, versus baselines of 26% and 27%. They state the system ranked 10th and 9th on the proprietary-model leaderboards.

Significance. If the reported results are reproducible, this is a strong empirical result, more than doubling the baseline accuracy. The architecture sensibly integrates well-known prompt-based techniques, and comparing multiple models is useful. The paper is readable and the pipeline is described at a high level. However, the central claim is entirely empirical and currently lacks the details needed for verification: no code, predictions, test-set statistics, or leaderboard link. It also does not evaluate the contribution of verification/reprocessing. As it stands, the manuscript is an incomplete system description rather than a verifiable research claim.

major comments (4)
  1. [Section 5, Table 1] The central accuracy claim is not verifiable. The paper reports a single number per model with no confidence intervals, no repeated runs, no temperature/seed settings, and no description of the official evaluation split, preprocessing, or answer-normalization. The words 'significantly' and 'significant improvement' appear in the abstract and Section 5 without any statistical test. The leaderboard ranks (10th/9th) are mentioned but no link or submission ID is provided. Since every conclusion rests on these percentages, the authors must supply the official evaluation protocol, exact outputs or code, and at least basic uncertainty quantification.
  2. [Appendix A.1 vs Section 4.2] The SQL generation prompt says 'You are a PostgreSQL expert ... create a syntactically correct PostgreSQL query,' while Section 4.2 says queries are executed on an SQLite database. This is an internal contradiction. If the prompt was actually used literally, SQL dialect differences could affect query success and final answers; if SQLite was intended, the prompt template in the appendix misdescribes the system. The authors must state the actual dialect used in every prompt and make the prompt consistent with the execution engine.
  3. [Sections 4.4 and 4.5] The verification and reprocessing stages are claimed to be key contributors to accuracy ('minimized incorrect responses', 'significantly reduced extraneous outputs'), but they are not evaluated. No statistics are given for how often responses are flagged, how often reprocessing changes an answer, or whether the final merged output improves over the initial SQL-only result. Without an ablation or per-stage analysis, these claims are unsupported and the reader cannot determine whether the five-stage pipeline is necessary for the reported performance.
  4. [Section 4.1] The example-selection component is underspecified. The paper says 25 curated Q/SQL pairs were used and top-2 were selected by cosine similarity, but the examples themselves, the embedding details, and the similarity threshold (if any) are not provided, nor is any sensitivity analysis over the number of examples. This is part of the pipeline and is needed for reproducibility.
minor comments (6)
  1. [Conclusion] There are typos: 'answwering' and 'taular' should be 'answering' and 'tabular'.
  2. [Throughout] The benchmark name is inconsistent: 'Databench' in the text and Table 1 versus 'DataBench' in the abstract and task description.
  3. [References] The reference for Grijalba et al. (2025) lacks full proceedings details, and the Soliman and Gurevych reference appears incomplete.
  4. [Section 5] The baseline model is not described. State what model and prompt were used to produce the 26% and 27% baseline accuracies.
  5. [Table 1] GPT-4o row has no space in 'GPT-4o70.50'; also numbers could be aligned for readability.
  6. [Appendix A.1] The instruction 'always use SELECT (*)' conflicts with the stated aim of retrieving only relevant rows; clarify the intended behavior.

Circularity Check

0 steps flagged

No circularity: empirical system paper with no fitted parameters, equations, or load-bearing self-citations.

full rationale

The paper derives no formal result and fits no parametric model. Its central claim is an empirical accuracy comparison (Table 1) between a fixed five-stage NL-to-SQL pipeline and a provided baseline on SemEval 2025 Task 8. The pipeline components—example selection by cosine similarity, SQL generation, CoT answer extraction, verification, and reprocessing—are defined independently of the test-set answers: the 25 examples are fixed hand-curated question-SQL pairs, the prompt templates are static, and no equation or fitted parameter is back-solved from the reported accuracies. The only numerical quantities, 70.50% and 71.65%, are reported outputs rather than constructed consequences of inputs. The paper contains no self-citations used to license a conclusion, no imported uniqueness theorem, and no renamed known empirical pattern. Concerns about verifiability (absence of code, seeds, test-set size, leaderboard link; the PostgreSQL prompt vs SQLite execution mismatch in Sections 4.2 and A.1) are correctness/reproducibility risks, not circularity. Therefore no circular step can be exhibited, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

No numerical fitting is performed; the pipeline relies on hand-chosen hyperparameters (example pool size, top-k) and unverified assumptions about embedding-based example selection and LLM verification. No new theoretical entities are postulated.

free parameters (2)
  • example_pool_size = 25
    Number of hand-curated question-query examples used for retrieval in Section 4.1; chosen by the authors, affects SQL generation quality.
  • top_k_examples = 2
    Number of most similar examples passed to the SQL generation prompt in Section 4.1; hand-chosen trade-off between context and noise.
axioms (4)
  • domain assumption The 25 hand-written examples cover the structural diversity of questions in DataBench.
    Section 4.1 assumes these examples provide useful few-shot context for all query types; no analysis of coverage is provided.
  • domain assumption text-embedding-ada-002 cosine similarity between questions selects the most useful SQL examples.
    Section 4.1 relies on this without validation; no comparison to other selection strategies is given.
  • domain assumption The verification LLM can reliably flag format violations or irrelevant answers without rejecting correct ones.
    Section 4.4 defaults borderline cases to acceptance; no precision or recall of the verifier is reported.
  • domain assumption SQL generated under the PostgreSQL prompt can be executed on the SQLite database used in Section 4.2.
    Appendix A.1 says PostgreSQL while Section 4.2 says SQLite; if incompatible, generated queries may behave differently on the real data.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Agentic LLMs for Question Answering over Tabular Data." pith.science (2026). https://pith.science/paper/7RUC6QUK

@misc{pith2026250909234,
  author       = {Pith},
  title        = {Pith review of: Agentic LLMs for Question Answering over Tabular Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7RUC6QUK}},
  note         = {Machine review of arXiv:2509.09234}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Question Answering over Tabular Data (Table QA) presents unique challenges due to the diverse structure, size, and data types of real-world tables. The SemEval 2025 Task 8 (DataBench) introduced a benchmark composed of large-scale, domain-diverse datasets to evaluate the ability of models to accurately answer structured queries. We propose a Natural Language to SQL (NL-to-SQL) approach leveraging large language models (LLMs) such as GPT-4o, GPT-4o-mini, and DeepSeek v2:16b to generate SQL queries dynamically. Our system follows a multi-stage pipeline involving example selection, SQL query generation, answer extraction, verification, and iterative refinement. Experiments demonstrate the effectiveness of our approach, achieving 70.5\% accuracy on DataBench QA and 71.6\% on DataBench Lite QA, significantly surpassing baseline scores of 26\% and 27\% respectively. This paper details our methodology, experimental results, and alternative approaches, providing insights into the strengths and limitations of LLM-driven Table QA.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 1 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Muhammad Shahzaib Baig, Azhar Imran, Aman Ullah Yasin, Abdul Haleem Butt, and Muhammad Imran Khan. 2022. Natural language to sql queries: A review. International Journal of Innovations in Science Technology, 4:147--162

    2022

  4. [4]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, Hao Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian L...

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.04434 2024

  5. [5]

    Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. 2024. https://arxiv.org/abs/2402.17944 Large language models(llms) on tabular data: Prediction, generation, and understanding -- a survey . Preprint, arXiv:2402.17944

    Pith/arXiv arXiv 2024

  6. [6]

    Jorge Os\' e s Grijalba, Luis Alfonso Ure\ n a L\' o pez, Eugenio Mart\' i nez C\' a mara, and Jose Camacho-Collados. 2025. Semeval-2025 task 8: Question answering over tabular data. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), TVienna, Austria

    2025

  7. [7]

    Aditya Gupta, Jingfeng Yang, Luheng He, Rahul Goel, Shachi Paul, and Shyam Upadhyay. 2022. Tableformer: Robust transformer modeling for table-text encoding. In ACL

    2022

  8. [8]

    Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. https://doi.org/10.18653/v1/2020.acl-main.398 Tapas: Weakly supervised table parsing via pre-training . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.398 2020

  9. [9]

    Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2025. https://arxiv.org/abs/2406.08426 Next-generation database interfaces: A survey of llm-based text-to-sql . Preprint, arXiv:2406.08426

    arXiv 2025

  10. [10]

    Nengzheng Jin, Joanna Siebert, Dongfang Li, and Qingcai Chen. 2022. https://arxiv.org/abs/2207.05270 A survey on table question answering: Recent advances . Preprint, arXiv:2207.05270

    Pith/arXiv arXiv 2022

  11. [11]

    Mahboob Alam Khalid, Valentin Jijkoun, and Maarten de Rijke. 2007. https://doi.org/10.1109/DEXA.2007.119 Machine learning for question answering from tabular data . In 18th International Workshop on Database and Expert Systems Applications (DEXA 2007), pages 392--396

    work page doi:10.1109/dexa.2007.119 2007

  12. [12]

    Xiping Liu and Zhao Tan. 2023. https://arxiv.org/abs/2304.11556 Divide and prompt: Chain of thought prompting for text-to-sql . Preprint, arXiv:2304.11556

    Pith/arXiv arXiv 2023

  13. [13]

    Maida, and Raju Gottumukkala

    Ali Mohammadjafari, Anthony S. Maida, and Raju Gottumukkala. 2025. https://arxiv.org/abs/2410.01066 From natural language to sql: Review of llm-based text-to-sql systems . Preprint, arXiv:2410.01066

    Pith/arXiv arXiv 2025

  14. [14]

    Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. 2023. Enhancing text-to-sql capabilities of large language models: A study on prompt design strategies. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14935--14956

    2023

  15. [15]

    Feifei Pan, Mustafa Canim, Michael Glass, Alfio Gliozzo, and James Hendler. 2022. End-to-end table question answering via retrieval-augmented generation. arXiv preprint arXiv:2203.16714

    Pith/arXiv arXiv 2022

  16. [16]

    Panupong Pasupat and Percy Liang. 2015. https://arxiv.org/abs/1508.00305 Compositional semantic parsing on semi-structured tables . Preprint, arXiv:1508.00305

    Pith/arXiv arXiv 2015

  17. [17]

    Neelabh Sinha, Vinija Jain, and Aman Chadha. 2024. Evaluating open language models across task types, application domains, and reasoning types: An in-depth experimental analysis. arXiv preprint arXiv:2406.11402

    Pith/arXiv arXiv 2024

  18. [18]

    A survey on advances in retrieval-augmented generation over tabular data and table qa

    Hassan Soliman and Iryna Gurevych. A survey on advances in retrieval-augmented generation over tabular data and table qa. In ELLIS workshop on Representation Learning and Generative Models for Structured Data

  19. [19]

    Zhongyuan Wang, Richong Zhang, Zhijie Nie, and Jaein Kim. 2024. https://arxiv.org/abs/2408.16991 Tool-assisted agent on sql inspection and refinement in real-world scenarios . Preprint, arXiv:2408.16991

    Pith/arXiv arXiv 2024

  20. [20]

    Zhiguang Wu, Fengbin Zhu, Xuequn Shang, Yupei Zhang, and Pan Zhou. 2024. https://arxiv.org/abs/2412.05850 Cooperative sql generation for segmented databases by using multi-functional llm agents . Preprint, arXiv:2412.05850

    Pith/arXiv arXiv 2024

  21. [21]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2019. https://arxiv.org/abs/1809.08887 Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task . Preprint, arXiv:1809.08887

    Pith/arXiv arXiv 2019

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.