Pith. sign in

REVIEW 2 major objections 2 minor 28 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

LLMs often ignore table evidence and answer from pretraining knowledge when the two conflict.

2026-06-26 06:02 UTC pith:2CHH2LDH

load-bearing objection The paper builds a paired benchmark to measure whether LLMs read the table or fall back on pretraining, and the accuracy gap on harder queries is the main takeaway. the 2 major comments →

arxiv 2606.23667 v1 pith:2CHH2LDH submitted 2026-06-22 cs.DB

The Table Says Otherwise: Testing LLMs with Counterfactual Relational Data

Xinzhi Wang , Chunwei Liu This is my paper
classification cs.DB
keywords large language modelstable question answeringcounterfactual evaluationrelational databasesfaithfulnessbenchmark construction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds a benchmark of paired original and counterfactual relational tables that preserve structure and relationships while altering selected facts about countries, clubs, and players. It poses 214 matched questions at increasing levels of complexity, from single-table lookups to multi-table joins and temporal reasoning, and compares model accuracy on the two versions. Strong models maintain reasonable performance on direct lookups even in the counterfactual setting, but accuracy falls as questions require joins, comparisons, and time-based reasoning, with the drop larger when table facts contradict familiar real-world information. The resulting gap is presented as evidence that models can default to memorized knowledge rather than treating the provided table as the sole source of truth. The work concludes that table-question-answering evaluation must therefore track faithfulness to the input database in addition to raw accuracy.

Core claim

When an LLM is given a relational table whose facts conflict with information learned during pretraining, its answers increasingly reflect the pretraining knowledge rather than the table, as shown by the measurable accuracy difference between an original database and an otherwise identical counterfactual version that changes only selected attribute values.

What carries the argument

ContraTable, the paired original-counterfactual benchmark that keeps schemas, identifiers, and relationships fixed while selectively altering country, club, and player attributes.

Load-bearing premise

The chosen changes to country, club, and player attributes create genuine conflicts with what the models already know without introducing structural inconsistencies that would make the comparison invalid.

What would settle it

Observing no accuracy difference between the original and counterfactual versions on the same set of 214 questions would indicate that the models are consistently reading the provided table instead of falling back on prior knowledge.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Instruction-tuned models handle single-table lookups reliably in both original and counterfactual settings.
  • Performance declines sharply once questions require multi-table joins, comparisons, or temporal reasoning.
  • The size of the original-counterfactual gap widens with question complexity, indicating greater reliance on prior knowledge for harder queries.
  • Table QA evaluations should report both accuracy and a faithfulness measure to the supplied database.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The benchmark design could be reused to test whether retrieval-augmented systems reduce the observed fallback to pretraining knowledge.
  • Database applications that treat tables as the authoritative source may need additional constraints or post-processing to enforce grounding.
  • The same paired-construction approach might reveal similar behavior in other structured-data tasks such as spreadsheet formula generation or knowledge-graph query answering.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper introduces ContraTable, a paired original-counterfactual benchmark for testing whether LLMs answer natural-language questions over relational tables by reading the provided data or by recalling pretraining knowledge. It constructs aligned databases that preserve schemas, identifiers, and relationships while editing selected country/club/player attributes, then evaluates 214 matched questions spanning single-table lookup, multi-table lookup, and multi-table temporal reasoning. Experiments on commercial and open-source models report that instruction-tuned models handle direct lookups reasonably well but show reliability drops on joins, comparisons, and temporal queries, with the original-vs-counterfactual accuracy gap interpreted as evidence of fallback to prior knowledge.

Significance. If the counterfactual edits are shown to be free of schema or temporal inconsistencies, the benchmark supplies a concrete method for measuring faithfulness to input tables rather than external knowledge. This directly addresses a practical requirement in database applications where the provided data must be the source of truth, and the work supplies an empirical basis for arguing that standard accuracy metrics alone are insufficient.

major comments (2)
  1. [Abstract and §3] Abstract and §3 (benchmark construction): the central claim attributes the accuracy gap to models falling back on pretraining knowledge rather than malformed input. This requires that attribute edits preserve all relationships and temporal constraints. No verification procedure, example validation against the full schema, or check for inconsistencies (e.g., an edited club conflicting with an unedited league or transfer date) is described.
  2. [Experiments] Experiments section: the abstract states that reliability drops on complex queries and that the gap reveals knowledge conflict, yet supplies no statistical significance tests, error bars, question-construction protocol, or controls for confounding variables. These omissions make it impossible to assess whether the reported gaps are robust.
minor comments (2)
  1. [Experiments] Provide the exact list of models evaluated and the precise prompt templates used for each question level.
  2. [Benchmark design] Clarify how the 214 questions were distributed across the three complexity levels and whether any questions were discarded during matching.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which help strengthen the presentation of the benchmark and experiments. We respond to each major comment below.

read point-by-point responses

  1. Referee: [Abstract and §3] Abstract and §3 (benchmark construction): the central claim attributes the accuracy gap to models falling back on pretraining knowledge rather than malformed input. This requires that attribute edits preserve all relationships and temporal constraints. No verification procedure, example validation against the full schema, or check for inconsistencies (e.g., an edited club conflicting with an unedited league or transfer date) is described.

    Authors: We agree that the manuscript does not provide an explicit verification procedure in §3. The construction relied on manual editing that preserved schemas, identifiers, and relationships, but documenting validation steps is necessary to fully support the claim that accuracy gaps reflect knowledge conflict rather than malformed inputs. In the revised manuscript we will add to §3 a verification subsection describing schema-level checks, relationship preservation tests, temporal constraint validation, and concrete examples of edits that were confirmed to introduce no inconsistencies. revision: yes

  2. Referee: [Experiments] Experiments section: the abstract states that reliability drops on complex queries and that the gap reveals knowledge conflict, yet supplies no statistical significance tests, error bars, question-construction protocol, or controls for confounding variables. These omissions make it impossible to assess whether the reported gaps are robust.

    Authors: The referee is correct that the current experiments section omits statistical significance tests, error bars, a detailed question-construction protocol, and explicit discussion of controls for confounding variables. In the revision we will add bootstrap confidence intervals and appropriate paired statistical tests (e.g., McNemar) for the original-versus-counterfactual gaps, include error bars on all accuracy figures, expand the question-construction description, and clarify how the matched-pair design and attribute-selection criteria limit confounding. These changes will allow readers to evaluate robustness directly. revision: yes

Circularity Check

0 steps flagged

Empirical benchmark study with no derivations or self-referential reductions

full rationale

This is a pure empirical benchmark paper that constructs paired original/counterfactual tables, designs 214 questions, and reports accuracy gaps across models. No equations, fitted parameters, predictions, or uniqueness theorems are claimed. The central claim (accuracy drop indicates fallback to pretraining knowledge) rests on the experimental design and results rather than any step that reduces to its own inputs by construction. No self-citations are load-bearing for the methodology. This matches the default case of a self-contained empirical study.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities; the work is an empirical benchmark study.

pith-pipeline@v0.9.1-grok · 5744 in / 999 out tokens · 23282 ms · 2026-06-26T06:02:17.345267+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are increasingly used to answer natural-language questions over structured data. However, when a table contains familiar real-world facts, it is unclear whether the model answers by reading the provided data or by recalling knowledge learned during pretraining. This distinction is important for database applications, where the provided tables should be the source of truth. In this paper, we introduce ContraTable, a paired original-counterfactual benchmark for evaluating whether LLMs ground their answers in relational tables. We build the benchmark with two aligned versions: an original database with real-world facts and a counterfactual database that preserves the same schemas, identifiers, and relationships while changing selected country, club, and player attributes. We design 214 matched questions across three levels: single-table lookup, multi-table lookup, and multi-table temporal reasoning. Experiments on commercial closed-source and open-source models show that strong instruction-tuned models can often handle direct lookup, but their reliability drops as questions require joins, comparison, and temporal reasoning. The gap between original and counterfactual accuracy reveals that models may fall back on prior knowledge when table evidence conflicts with familiar facts. These results suggest that table-QA evaluation should measure not only accuracy, but also faithfulness to the provided database.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 3 canonical work pages · 2 internal anchors

  1. [1]

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision. 2425–2433

    2015

  2. [2]

    Wenhu Chen. 2023. Large language models are few (1)-shot table reasoners. In Findings of the association for computational linguistics: EACL 2023. 1120–1130

    2023

  3. [3]

    Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020. HybridQA: A dataset of multi-hop question answering over tabular and textual data. InFindings of the Association for Computational Linguistics: EMNLP 2020. 1026–1036

    2020

  4. [4]

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al

  5. [5]

    InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

    Finqa: A dataset of numerical reasoning over financial data. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 3697–3711

    2021

  6. [6]

    Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2024. Evalu- ating the ripple effects of knowledge editing in language models.Transactions of the Association for Computational Linguistics12 (2024), 283–298

    2024

  7. [7]

    David Cariboo. [n.d.]. Football Data from Transfermarkt. https://www.kaggle. com/datasets/davidcariboo/player-scores. Accessed: 2026-06-21

    2026

  8. [8]

    Raul Castro Fernandez, Aaron J Elmore, Michael J Franklin, Sanjay Krishnan, and Chenhao Tan. 2023. How large language models will disrupt data management. Proceedings of the VLDB Endowment16, 11 (2023), 3302–3309

    2023

  9. [9]

    Peter Hase, Thomas Hofweber, Xiang Zhou, Elias Stengel-Eskin, and Mohit Bansal. [n.d.]. Fundamental problems with model editing: How should rational belief revision work in llms?, 2024.URL https://arxiv. org/abs/2406.19354([n. d.])

    work page arXiv 2024

  10. [10]

    Benjamin Heinzerling and Kentaro Inui. 2021. Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 1772–1791

    2021

  11. [11]

    Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre- training. InProceedings of the 58th annual meeting of the association for computa- tional linguistics. 4320–4333

    2020

  12. [12]

    Jason Hoelscher-Obermaier, Julia Persson, Esben Kran, Ioannis Konstas, and Fazl Barez. 2023. Detecting edit failures in large language models: An improved specificity benchmark. InFindings of the Association for Computational Linguistics: ACL 2023. 11548–11559

    2023

  13. [13]

    Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. Search-based neural structured learning for sequential question answering. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1821–1831

    2017

  14. [14]

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics7 (2019), 453–466

    2019

  15. [15]

    Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. InProceedings of the 2021 conference on empirical methods in natural language processing. 7052–7063

    2021

  16. [16]

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. InProceedings of the IEEE/CVF winter conference on applications of computer vision. 2200–2209

    2021

  17. [17]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt.Advances in neural information processing systems35 (2022), 17359–17372

    2022

  18. [18]

    Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022. Mass-editing memory in a transformer.arXiv preprint arXiv:2210.07229 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  19. [19]

    Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2023. Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 10056–10070

    2023

  20. [20]

    Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. InProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 1470–1480

    2015

  21. [21]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. InProceedings of the 2016 conference on empirical methods in natural language processing. 2383–2392

    2016

  22. [22]

    Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model?. InProceedings of the 2020 conference on empirical methods in natural language processing (EMNLP). 5418–5426

    2020

  23. [23]

    Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. InProceedings of the 17th ACM International Conference on Web Search and Data Mining. 645–654

    2024

  24. [24]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. InProceedings of the 2018 conference on empirical methods in natural language processing. 3911–3921

    2018

  25. [25]

    Victor Zhong, Caiming Xiong, and Richard Socher. [n.d.]. Seq2SQL: Generating structured queries from natural language using reinforcement learning (2017). arXiv preprint arXiv:1709.00103([n. d.])

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  26. [26]

    Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. 2023. Mquake: Assessing knowledge editing in language models via multi-hop questions. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 15686–15702

    2023

  27. [27]

    Xuanhe Zhou, Zhaoyan Sun, and Guoliang Li. 2024. DB-GPT: Large Language Model Meets Database.Data Science & Engineering9, 1 (2024), 102

    2024

  28. [28]

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A question an- swering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th annual meeting of the Association for Computational Lin- guistics and the 11th international joint conference on natur...

    2021