RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models

Dario Satriani; Donatello Santoro; Enzo Veltri; Paolo Papotti

REVIEW 3 major objections 6 minor 61 references

RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Closed-book table recall caps every tested LLM at 25% tuple accuracy

desk verdict A genuinely useful benchmark with a real controlled-experiment core, but the headline 25% tuple-accuracy claim is weakened by an unverified knowledge-coverage assumption and metric choices that likely understate true performance. read the letter →

arxiv 2505.21409 v1 pith:BIFP75G4 submitted 2025-05-27 cs.CL cs.AIcs.DB

Dario Satriani , Enzo Veltri , Donatello Santoro , Paolo Papotti This is my paper

classification cs.CLcs.AIcs.DB

keywords RelationalFactQAclosed-bookretrievaltabulargenerationparametricmemoryLLMfactualitybenchmarktuplesimilaritychain-of-thoughtpromptinghallucinationevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces RelationalFactQA, a benchmark that asks large language models to produce complete multi-row, multi-attribute tables from their own learned knowledge, without any external retrieval. Its central finding is that this task is far harder than answering single factual questions: across nine LLMs from 7B to 235B parameters, the best tuple-level accuracy is just under 25%, and accuracy drops steadily as tables get wider or longer. The same models often give the correct value when asked for one fact in isolation, which the paper reads as evidence that the bottleneck is composing facts into a relational structure, not lacking the underlying knowledge. If correct, this points to a distinct failure mode in current LLMs that existing short-answer benchmarks do not expose.

What carries the argument

The central object is the RelationalFactQA benchmark itself: 696 triples of natural-language question, SQL query, and gold-standard tabular answer, averaging 27 rows and 5.3 attributes per answer, built from databases with Wikipedia-known entities and from YAGO-derived table templates. The load-bearing metric is tuple similarity, a holistic match requiring identical schema, cardinality, and cell values (after normalization and approximate matching), which captures the compositional alignment the paper claims is missing. Iterative prompting over three modalities—plain natural language, raw SQL, and Chain-of-Thought decomposition—lets the authors isolate where the failure occurs.

What would settle it

For a set of RelationalFactQA queries where a model answers every constituent cell correctly in isolated point-wise probes, check its tuple similarity on the full tables; if tuple accuracy stays near-perfect for known facts and only falls when facts are unknown, the reported ceiling would reflect missing knowledge rather than a compositional failure.

Watch

Extended reading notes

Core claim

Closed-book relational fact retrieval is a distinct, substantially harder capability than point-wise fact recall. On the RelationalFactQA benchmark, no tested LLM exceeds 0.247 tuple similarity, and performance degrades linearly with the number of requested attributes and with the total number of output cells. Controlled experiments show that a single attribute's accuracy drops from near-perfect to about half when 50 other attributes are requested alongside it, even though the same model answers the same values correctly when queried individually. The paper argues that LLMs retain abundant factual fragments but lack a reliable mechanism to align them into coherent tabular relations.

Load-bearing premise

The benchmark assumes the gold tables are within each model's pretraining knowledge because their entities appear on Wikipedia, without verifying per query that the model actually knows every target fact.

Editorial extensions

If this is right

If the compositional bottleneck is real, improving single-fact knowledge or scaling parameters alone will not fix tabular factuality; progress requires mechanisms for aligning and grouping facts during generation.
Chain-of-Thought decomposition raises cell-level recall but leaves tuple similarity largely unchanged, implying that reasoning steps help retrieve more fragments without solving their correct arrangement.
The linear degradation with attribute count and the sensitivity to attribute position suggest that output-size pressure—attention or working-memory limits—is a primary driver of the observed errors.
Numerical conditions in WHERE clauses cause most missing-tuple errors, while equality conditions on categorical values are comparatively handled well, giving a concrete target for hybrid retrieval designs.
Even the best model's tuple accuracy stays below 0.25, establishing a quantitative baseline for future progress on structured fact generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension would verify the compositional hypothesis directly by running point-wise probes for every cell of selected RelationalFactQA queries; if tuple accuracy stays low even when all constituent facts are retrievable individually, the composition account is strongly confirmed.
The error pattern suggests a cheap practical fix the paper only gestures at: use the LLM to recall base rows, then apply numerical predicates and joins with deterministic post-processing rather than asking the model to filter and align during generation.
The observed attribute-position sensitivity implies that table serialization order is not neutral; alternative row-major versus column-major decoding or schema reordering might shift which parts of a table are recalled correctly, which could be tested on the released benchmark.
Since pure refusal is rare (only 4% of analyzed errors are empty results), the models are not calibrating their uncertainty on this task; an uncertainty-aware extension could measure whether asking for fewer, high-confidence tuples improves tuple precision.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: this is a genuinely useful benchmark and the paper has a real empirical core, but the headline 25% tuple-accuracy claim is softer than it looks because the benchmark never verifies that individual cells are actually knowable by the models. Despite the stress-test note, I don't think that kills the paper; the controlled experiments in Section A (Euro 2016 players, US counties) do isolate composition failure.

What's new: RelationalFactQA targets multi-tuple, multi-attribute closed-book retrieval, and its output-size distribution (avg 27 rows, 5.3 attrs) is genuinely different from WikiSQL/TruthfulQA. The hybrid construction pipeline (manual from Spider/Bird/Galois plus Qatch + YAGO) is sensible, and the experiments across 9 models and 3 prompting styles are well organized. Trends replicate: TS degrades with attributes and cells; CoT helps cell F1 but not tuple alignment. The error analysis in Appendix E is honest and useful.

Soft spots, in order of severity. (1) The knowledge-coverage assumption is load-bearing for the benchmark-level claim. Wikipedia presence of an entity does not guarantee each attribute-value pair is in parametric memory. The paper reports no per-cell point-wise probe; the backlinks metadata is a popularity proxy, and the footnote saying popularity "does not impact our experiments" has no supporting analysis. So the 0.25 TS ceiling could partly reflect missing facts rather than composition failure. (2) Metric transparency: TS is strict tuple match with 10% edit distance and +/-10% numeric tolerance, and the paper's own error analysis attributes 43% of errors to canonicalization false negatives. That means the absolute numbers are likely underestimated. (3) No error bars or significance tests; some claims rest on small breakdowns. (4) No artifacts are actually released, only a promise.

The central phenomenon—compositional retrieval is harder than point-wise recall—survives because of the controlled experiments. But the quantitative headline should be framed as a lower bound pending artifact release and robustness checks.

Who this is for: anyone benchmarking LLM factuality or building parametric database interfaces. It deserves a serious referee; conditional acceptance is the right posture, with artifact release and a per-cell knowledge probe as requirements.

Referee Report

3 major / 6 minor

Summary. The paper introduces RelationalFactQA, a 696-question benchmark for closed-book relational fact retrieval, in which an LLM must generate a multi-row, multi-attribute table from parametric memory given a natural-language or SQL query. The authors evaluate nine LLMs (7B–235B parameters) under three retrieval protocols (NL, SQL, and chain-of-thought) and score outputs with cell-level F1 and tuple-level Tuple Similarity (TS). They report that no model exceeds 0.25 TS, that F1 improves with model size while TS does not, and that performance degrades as the number of requested attributes and output cells increases. The paper also includes controlled experiments in Appendix A showing that a fixed attribute's F1 degrades as more attributes are requested, which they interpret as evidence that relational composition, not single-fact knowledge, is the bottleneck.

Significance. If substantiated, the benchmark would fill a real gap in factuality evaluation: existing closed-book QA benchmarks use single-tuple outputs, while RelationalFactQA targets multi-tuple, multi-attribute tables with controlled query complexity, and the dataset release would be valuable for measuring structured factuality. The controlled incremental-attribute and attribute-position experiments (Fig. 3) are a well-designed probe of composition failure, and the evaluation covers a broad model and prompting sweep with deterministic temperature settings. However, the paper's headline interpretation—that low tuple accuracy reflects a failure of relational composition even when individual facts are known—is not yet established, because the benchmark does not verify per-cell knowledge and the tuple metric is partly a string-matching artifact. These issues are fixable, and the resource itself appears useful, but the central claim currently outruns the evidence.

major comments (3)

[Section 3 and Abstract] The central claim that relational retrieval is harder 'even when individual facts are known to the model' is not backed by any per-query knowledge check. Section 3 justifies the benchmark by selecting databases 'with schema and entities that are present on Wikipedia,' but an entity's presence on Wikipedia does not guarantee that each gold cell (e.g., the area of Maricopa County, or the birth year of a Nobel laureate) is encoded in a given model's parameters. The BACKLINKS metadata in Section B is a popularity proxy, and the footnote asserting that popularity 'does not impact our experiments' is unsupported. If many target rows are outside a model's training knowledge, low TS would be observed even if relational composition were trivial. I ask the authors to add point-wise probes (e.g., querying each gold cell, or a stratified sample, as a standalone fact) and to report TS separately for queries whose gold cells are individually retrievable. This distinguishes knowledge-coverage failure from composition failure and should be a prerequisite for the abstract's claim.
[Section 4 (Metrics) and Section E (Error Analysis)] The headline ceiling of 0.25 TS is partly an artifact of the string-matching metric. The error analysis on GPT-4.1 reports that 43% of errors are Canonicalization issues—semantically equivalent values ('USA' vs. 'United States of America', 's' vs. 's-block') that the 10% edit-distance threshold does not match. Although Section E calls these 'false negatives,' the paper never quantifies how much TS would increase if such matches were counted correctly. Because the limitations table (Table 5) acknowledges this bias, the authors should estimate a corrected TS on a sample using a semantic matcher or manual adjudication, and report both the uncorrected and corrected numbers. Without this, 'factual accuracy' is conflated with lexical similarity, and the no-model-exceeds-25% claim is not a clean factual statement.
[Section 5, Exp-3 (Figure 2)] The claimed monotonic degradation with number of attributes and output cells is presented without per-bin sample sizes, confidence intervals, or significance tests. Given the long-tailed distribution (Table 6: Q1 of output tuples is 1, max is 904), the bins are likely highly unbalanced, and the observed trend may be confounded by query source (Qatch vs. Spider/Bird/Galois), row count, or operator type. The authors should report the number of queries per bin, show error bars or bootstrapped intervals, and ideally fit a regression with source and complexity controls to verify that the dimensionality trend is not an aggregation artifact.

minor comments (6)

[Table 1 and Tables 2–8] The RFQA row in Table 1 is misformatted ('RFQA69626.942'), making the statistics unreadable, and the label 'A VG' in Tables 2, 3, and 8 should be 'AVG'.
[Section 5, Exp-1] The text contains a typo 'GTP 4.1' for 'GPT-4.1', and 'NL outperfoms slightly' should read 'NL slightly outperforms SQL'.
[Figure 9 captions] The legends in Figure 9 use 'GPT-4o mini' and 'Reasoning', which are inconsistent with the model names in Table 7 (GPT-4.1 mini and DeepSeek R1 Distill Llama 70B); these labels should be standardized.
[Section 3, footnote 2] The sentence 'we experimentally verified that this dimension does not impact our experiments' needs the supporting analysis or should be removed, as written it is an unbacked assertion.
[Section 1 and Figure 3(a)] The claim that accuracy 'degrades linearly' from 1.0 to 0.2 as the number of requested attributes increases is stronger than Figure 3(a) supports; the figure shows a nonlinear decline and does not include a fitted line. Please rephrase or add a regression fit.
[Section 4 (Output Processing) and Section 5, Exp-2 (Table 3)] The recovery heuristics (re-prompt, truncation repair) are described but their frequency is not reported; the paper should state how many responses required repair per model and strategy. Also, the conclusions about numerical vs. categorical attribute performance in Table 3 are drawn without significance testing or per-category sample sizes; add confidence intervals or treat the differences as descriptive.

Circularity Check

0 steps flagged · score 1.0 of 10

Empirical benchmark with a knowledge-coverage assumption, but no derivation that reduces to its own inputs.

full rationale

This is a benchmark-and-measurement paper rather than a derivation, so the main circularity patterns do not apply. The central result (best Tuple Similarity 0.247, with quality degrading as attributes/rows increase) is a measured outcome on a dataset and metrics the authors constructed; it could in principle have come out differently, so it is not forced by definition. The controlled Euro 2016 and US-county experiments in Section 1 and Appendix A provide independent, hand-verified evidence that requesting more attributes degrades accuracy on a fixed attribute, and those experiments do not depend on the benchmark's ground-truth tables. The one weak point is the 'even when individual facts are known to the model' clause: Section 3 supports knowledge coverage only by Wikipedia entity presence ('we identify the databases with schema and entities that are present on Wikipedia - this is important to ensure that the examples are within the knowledge scope of an LLM'), with no per-query point-wise knowledge probe reported. That is a validity or confounding concern, not circularity, because the low tuple accuracy is not produced by that assumption. The CoT method is taken from prior work co-authored by Papotti (Saeed et al., EDBT 2024) and is cited rather than asserted, and Qatch (Papicchio et al., NeurIPS 2023) is cited for the NL-SQL generator; neither citation is load-bearing for the main degradation finding. No fitted parameter is later renamed as a prediction, no uniqueness theorem is imported via self-citation, and no known result is merely renamed. Score 1 reflects only a minor, non-load-bearing reliance on the authors' own tools and prior work.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the benchmark construction and evaluation assumptions listed above. The two evaluation tolerances are free parameters that directly influence the reported accuracy. No new physical or conceptual entities are postulated; RelationalFactQA is a methodological artifact, not an invented entity in the sense of this ledger.

free parameters (2)

edit distance threshold = 10% of expected string length
Hand-set tolerance in evaluation (Section 4, Metrics). It controls how many approximate string matches count as correct and therefore affects F1 and TS.
numeric tolerance = ±10% of expected number
Hand-set tolerance for numeric cell comparison (Section 4, Metrics). Affects reported accuracy.

assumptions (4)

domain assumption Entities present on Wikipedia are within the knowledge scope of tested LLMs.
Used to select source databases and YAGO types in Section 3; not verified per query.
domain assumption Executing SQL on the original databases gives the correct gold table for the relational fact retrieval task.
Section 3: gold tables are computed by running the query on the source data, assuming the data is factually accurate and the SQL semantics are the intended target.
domain assumption An empty JSON response signals that the model has exhausted its parametric knowledge.
Iterative prompting loop in Appendix C stops when the model returns empty; early termination would understate recall.
domain assumption Closed-book setting isolates intrinsic factual competence.
Section 3 states this as the rationale for prohibiting tools; it is a hypothesis about confounding variables, not an established result.

how reviews work

0 comments

Cite this review

Pith. "Pith review of RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models." pith.science (2026). https://pith.science/paper/BIFP75G4

@misc{pith2026250521409,
  author       = {Pith},
  title        = {Pith review of: RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BIFP75G4}},
  note         = {Machine review of arXiv:2505.21409}
}

read the original abstract

Factuality in Large Language Models (LLMs) is a persistent challenge. Current benchmarks often assess short factual answers, overlooking the critical ability to generate structured, multi-record tabular outputs from parametric knowledge. We demonstrate that this relational fact retrieval is substantially more difficult than isolated point-wise queries, even when individual facts are known to the model, exposing distinct failure modes sensitive to output dimensionality (e.g., number of attributes or records). To systematically evaluate this under-explored capability, we introduce RelationalFactQA, a new benchmark featuring diverse natural language questions (paired with SQL) and gold-standard tabular answers, specifically designed to assess knowledge retrieval in a structured format. RelationalFactQA enables analysis across varying query complexities, output sizes, and data characteristics. Our experiments reveal that even state-of-the-art LLMs struggle significantly, not exceeding 25% factual accuracy in generating relational outputs, with performance notably degrading as output dimensionality increases. These findings underscore critical limitations in current LLMs' ability to synthesize structured factual knowledge and establish RelationalFactQA as a crucial resource for measuring future progress in LLM factuality.

Figures

Figures reproduced from arXiv: 2505.21409 by the authors.

**Figure 1.** RFQA dataset. Source distribution and distribution of query complexity (SQL operators). Dataset Statistics. The RFQA benchmark comprises 696 question, query, answer triples. As reported in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

**Figure 2.** TS results for LLama 3.3, GPT-4.1 and QWEN 3, with all retrieval techniques, w.r.t. the [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** F1 Quality of attribute surname in different queries A Motivation Example As discussed in the introduction, extracting structured information from an LLM’s internal knowledge in a tabular format poses unique and challenging problems, distinct from conventional, single-point natural language queries. To demonstrate and isolate these issues empirically, we designed a series of controlled experiments. Specifically, we … view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Topic distribution • QUESTION: Natural language (NL) version of the query. • TUPLES: Expected number of output tuples (rows). • ATTRS: Expected number of output attributes (columns). • ATTR NUMERICAL: Number of expected numerical attributes. • ATTR CATEGORICAL: Number …

**Figure 5.** Figure 5: NL Prompt Syntax. Text in italic is injected from the given NL query and the expected JSON schema of the response. Start Prompt: List the results of the SQL query: SQL. Respond with JSON only. Don’t add any comments. Use the following JSON schema: jsonSchema. SQL: is t…

**Figure 6.** Figure 6: SQL Prompt Syntax. Text in italic is injected from the given SQL query and the expected JSON schema of the response. First Prompt: Given the following query, populate the table with actual values. query: select attributes from table (where conditions). Respond with JSO…

**Figure 7.** Figure 7: CoT Prompt Syntax. Text in italic is injected from the given SQL query. Values between parenthesis are populated only if the condition(s) is given. Handling Output JSON Errors. All the strategies ask the LLM to return the data in a structured form respecting a JSON for…

**Figure 8.** Figure 8: Impact of the Number of attributes w.r.t. Tuple Similarity (TS) with NL, SQL, CoT [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

**Figure 9.** Figure 9: Impact of the Number of attributes w.r.t. F1 with NL, SQL, CoT strategies [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

**Figure 10.** Figure 10: Breakdown of common error types in GPT-4.1 outputs on factual table generation. Each [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 34 canonical work pages

[1]

Ai hallucination report 2025, 2025

All About AI. Ai hallucination report 2025, 2025. URL https://www.allaboutai.com/ resources/ai-statistics/ai-hallucinations/. (pp. 1 and 3)

work page 2025
[2]

R. Aly, Z. Guo, M. S. Schlichtkrull, J. Thorne, A. Vlachos, C. Christodoulopoulos, O. Cocarascu, and G. Li. FEVEROUS: Fact extraction and VERification over unstructured and structured information. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. (p. 2)

work page 2021
[3]

Balsiger, H.-R

D. Balsiger, H.-R. Dimmler, S. Egger-Horstmann, and T. Hanne. Assessing large language models used for extracting table information from annual financial reports.Computers, 13(10),

work page
[4]

Bisercic, M

A. Bisercic, M. Nikolic, M. van der Schaar, B. Delibasic, P. Lio, and A. Petrovic. Interpretable medical diagnostics with structured data extraction by large language models, 2023. URL https://arxiv.org/abs/2306.05052. (p. 2)

arXiv 2023
[5]

Borisov, K

V . Borisov, K. Sessler, T. Leemann, M. Pawelczyk, and G. Kasneci. Language models are realistic tabular data generators. InThe Eleventh International Conference on Learning Repre- sentations, 2023. URLhttps://openreview.net/forum?id=cEygmQNOeI. (p. 4)

work page 2023
[6]

Buoncristiano, G

M. Buoncristiano, G. Mecca, D. Santoro, and E. Veltri. Detective gadget: Generic iterative entity resolution over dirty data.Data, 2024. doi: 10.3390/data9120139. (p. 6)

work page doi:10.3390/data9120139 2024
[7]

Cappuzzo, G

R. Cappuzzo, G. Varoquaux, A. Coelho, and P. Papotti. Retrieve, merge, predict: Augmenting tables with data lakes.CoRR, abs/2402.06282, 2024. doi: 10.48550/ARXIV .2402.06282. URL https://doi.org/10.48550/arXiv.2402.06282. (p. 4)

arXiv doi:10.48550/arxiv.2402.06282 2024
[8]

Chang, X

Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang, W. Ye, Y . Zhang, Y . Chang, P. S. Yu, Q. Yang, and X. Xie. A survey on evaluation of large language models.ACM Trans. Intell. Syst. Technol., 15(3), mar 2024. ISSN 2157-6904. doi: 10.1145/3641289. URLhttps://doi.org/10.1145/3641289. (p. 3)

doi:10.1145/3641289 2024

Show all 61 references

[9]

W. Chen, A. Lilley, J. Gu, Z. Qian, V . Zhong, K. Gimpel, and K. Toutanova. Tabfact: A large-scale dataset for table-based fact verification. InInternational Conference on Learning Representations (ICLR), 2020. (pp. 2 and 3)

2020
[10]

Chowdhury, D

N. Chowdhury, D. Johnson, V . Huang, J. Steinhardt, and S. Schwettmann. In- vestigating truthfulness in a pre-release o3 model. https://transluce.org/ investigating-o3-truthfulness, April 2025. (pp. 1 and 3)

2025
[11]

Christophides, V

V . Christophides, V . Efthymiou, T. Palpanas, G. Papadakis, and K. Stefanidis. An overview of end-to-end entity resolution for big data.ACM Comput. Surv., 53(6):127:1–127:42, 2021. (p. 6)

2021
[12]

Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954, 2024

DeepSeek AI. Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954, 2024. doi: 10.48550/ARXIV .2401.02954. URL https://doi. org/10.48550/arXiv.2401.02954. (p. 6)

2024 doi arXiv
[13]

M. M. Dong, T. C. Stratopoulos, and V . X. Wang. A scoping review of chatgpt research in accounting and finance.International Journal of Accounting Information Systems, 55:100715,
[14]

Elnashar, J

A. Elnashar, J. White, and D. C. Schmidt. Enhancing structured data generation with gpt-4o evaluating prompt efficiency across prompt styles.Frontiers in Artificial Intelli- gence, V olume 8 - 2025, 2025. ISSN 2624-8212. doi: 10.3389/frai.2025.1558938. URL https://www.frontier...

2025
[15]

doi: https://doi.org/10.1016/j.accinf.2024.100715

ISSN 1467-0895. doi: https://doi.org/10.1016/j.accinf.2024.100715. URL https: //www.sciencedirect.com/science/article/pii/S1467089524000484. (p. 1)

2024
[16]

Gemma: Open models based on gemini research and technology

Gemma Team, Google. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. doi: 10.48550/ARXIV .2403.08295. URL https: //doi.org/10.48550/arXiv.2403.08295. (p. 6) 10

2024 doi arXiv
[17]

M. Gao, X. Hu, X. Yin, J. Ruan, X. Pu, and X. Wan. Llm-based nlg evaluation: Current status and challenges.Computational Linguistics, pages 1–28, 04 2025. ISSN 0891-2017. doi: 10.1162/coli a 00561. URLhttps://doi.org/10.1162/coli_a_00561. (pp. 2 and 9)

2025 doi
[18]

Holtzman, J

A. Holtzman, J. Buys, L. Du, M. Forbes, and Y . Choi. The curious case of neural text degeneration. InInternational Conference on Learning Representations, 2020. URL https: //openreview.net/forum?id=rygGQyrFvH. (pp. 3 and 9)

2020
[19]

Glavic, G

B. Glavic, G. Mecca, R. J. Miller, P. Papotti, D. Santoro, and E. Veltri. Similarity measures for incomplete database instances. In L. Tanca, Q. Luo, G. Polese, L. Caruccio, X. Oriol, and D. Fir- mani, editors,Proceedings 27th International Conference on Extending Database Tec...

2024 doi
[20]

Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu, E. Ishii, Y . J. Bang, A. Madotto, and P. Fung. Survey of hallucination in natural language generation.ACM Comput. Surv., 55(12), Mar. 2023. ISSN 0360-0300. doi: 10.1145/3571730. URL https://doi.org/10.1145/3571730. (pp. 1, 3, and 9)

2023 doi
[21]

Z. Hong, Z. Yuan, Q. Zhang, H. Chen, J. Dong, F. Huang, and X. Huang. Next-generation database interfaces: A survey of llm-based text-to-sql.arXiv preprint arXiv:2406.08426, 2024. (p. 3)

2024
[22]

Joshi, E

M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URL https://arxiv.org/abs/1705. 03551. (p. 3)

2017
[23]

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b.arXiv preprint arXiv:2310.06825,

arXiv
[24]

Kweon, Y

S. Kweon, Y . Kwon, S. Cho, Y . Jo, and E. Choi. Open-wikitable: Dataset for open domain question answering with complex reasoning over table, 2023. URL https://arxiv.org/ abs/2305.07288. (p. 3)

2023 arXiv
[25]

Kwiatkowski, J

T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering resea...

2019
[26]

Kasneci, K

E. Kasneci, K. Seßler, S. Kuchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, G. Hahnel, M. C. Hett, N.-E. Hett, N. K ¨arger, J. Liu, X. Liu, M. Nerdel, J. Nistor, C. Scheid, R. Stallasch, S. Stober, and G. Kasneci. ChatGPT for good? On opportunities and chal...

2023
[27]

J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024. (pp. 1, 3, and 4)

2024
[28]

S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods,
[29]

Lee, M.-W

K. Lee, M.-W. Chang, and K. Toutanova. Latent retrieval for weakly supervised open domain question answering. In A. Korhonen, D. Traum, and L. M`arquez, editors,Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, ...

2019 doi
[30]

N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024. doi: 10.1162/tacl a 00638. URL https: //aclanthology....

2024 doi
[31]

X. Liu, S. Shen, B. Li, P. Ma, R. Jiang, Y . Luo, Y . Zhang, J. Fan, G. Li, and N. Tang. A survey of NL2SQL with large language models: Where are we, and where are we going? CoRR, abs/2408.05109, 2024. doi: 10.48550/ARXIV .2408.05109. URL https://doi.org/ 10.48550/arXiv.2408.0...

2024 doi
[32]

The llama 3 herd of models, 2024

Meta AI. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. (p. 6)

2024 arXiv
[33]

C. Liu, M. Russo, M. Cafarella, L. Cao, P. B. Chen, Z. Chen, M. Franklin, T. Kraska, S. Madden, R. Shahout, and G. Vitagliano. Palimpzest: Optimizing ai-powered analytics with declarative query processing. InProceedings of the Conference on Innovative Database Research (CIDR),
[34]

Papicchio, P

S. Papicchio, P. Papotti, and L. Cagliero. Qatch: Benchmarking sql-centric tasks with table representation learning models on your data.Advances in Neural Information Processing Systems, 36:30898–30917, 2023. (pp. 4 and 6)

2023
[35]

Pasupat and P

P. Pasupat and P. Liang. Compositional semantic parsing on semi-structured tables. InPro- ceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pag...

2015
[36]

Patel, S

L. Patel, S. Jha, M. Pan, H. Gupta, P. Asawa, C. Guestrin, and M. Zaharia. Semantic operators: A declarative model for rich, ai-based data processing, 2025. URL https://arxiv.org/abs/ 2407.11418. (p. 2)

2025 arXiv
[37]

Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

OpenAI. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. doi: 10.48550/ARXIV . 2303.08774. URLhttps://doi.org/10.48550/arXiv.2303.08774. (pp. 1 and 6)

2023 doi arXiv
[38]

Petroni, P

F. Petroni, P. Lewis, A. Piktus, T. Rockt¨aschel, Y . Wu, A. H. Miller, and S. Riedel. How context affects language models’ factual predictions. InAutomated Knowledge Base Construction,
[39]

Qwen2.5 technical report, 2025

Qwen Team. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115. (p. 6)

2025 arXiv
[40]

E. S. Ristad and P. N. Yianilos. Learning string-edit distance.IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(5):522–532, 1998. (p. 6)

1998
[41]

Petroni, T

F. Petroni, T. Rockt¨aschel, A. H. Miller, P. Lewis, A. Bakhtin, Y . Wu, and S. Riedel. Language models as knowledge bases? InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019. (p. 3)

2019
[42]

Saparina and M

I. Saparina and M. Lapata. Ambrosia: A benchmark for parsing ambiguous questions into database queries.Advances in Neural Information Processing Systems, 37:90600–90628, 2024. (p. 3)

2024
[43]

Shankar, T

S. Shankar, T. Chambers, T. Shah, A. G. Parameswaran, and E. Wu. Docetl: Agentic query rewriting and evaluation for complex document processing, 2025. URL https://arxiv.org/ abs/2410.12189. (p. 2)

2025 arXiv
[44]

Singhal, S

K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole- Lewis, S. Pfohl, P. Payne, M. Seneviratne, P. Gamble, C. Kelly, N. Sch ¨arli, A. Chowdhery, P. Mansfield, B. A. y Arcas, D. Webster, G. S. Corrado, Y . Matias, K. Chou, J. Gottweis...

2023
[45]

Stuhler, C

O. Stuhler, C. D. Ton, and E. Ollion. From codebooks to promptbooks: Extracting in- formation from text with generative large language models.Sociological Methods & Re- search, 0(0):00491241251336794, 0. doi: 10.1177/00491241251336794. URL https: //journals.sagepub.com/doi/abs...

doi
[46]

Saeed, N

M. Saeed, N. D. Cao, and P. Papotti. Querying large language models with SQL. InProceedings 27th International Conference on Extending Database Technology, EDBT 2024, Paestum, Italy, March 25 - March 28, pages 365–372. OpenProceedings.org, 2024. doi: 10.48786/EDBT.2024

2024 doi
[47]

URLhttps://doi.org/10.48786/edbt.2024.32. (pp. 2, 4, and 6)

2024 doi
[48]

Truhn, J

D. Truhn, J. S. Reis-Filho, and J. N. Kather. Large language models should be used as scientific reasoning engines, not knowledge databases.Nature medicine, 29(12):2983–2984, 2023. (p. 1)

2023
[49]

J. Wei, N. Karina, H. W. Chung, Y . J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. Measuring short-form factuality in large language models.CoRR, abs/2411.04368, 2024. doi: 10. 48550/ARXIV .2411.04368. URLhttps://doi.org/10.48550/arXiv.2411.04368. (pp. 1 and 3)

2024 doi arXiv
[50]

J. Wu, L. Yang, D. Li, Y . Ji, M. Okumura, and Y . Zhang. MMQA: Evaluating LLMs with multi- table multi-hop complex questions. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=GGlpykXDCa. (p. 1)

2025
[51]

X. Wu, J. Yang, L. Chai, G. Zhang, J. Liu, X. Du, D. Liang, D. Shu, X. Cheng, T. Sun, T. Li, Z. Li, and G. Niu. Tablebench: A comprehensive and complex benchmark for table question answering. In T. Walsh, J. Shah, and Z. Kolter, editors,AAAI-25, Sponsored by the Association fo...

2025 doi
[52]

Suchanek, M

F. Suchanek, M. Alam, T. Bonald, L. Chen, P.-H. Paris, and J. Soria. Yago 4.5: A large and clean knowledge base with a rich taxonomy, 2024. URL https://arxiv.org/abs/2308.11884. (p. 4) 12

2024 arXiv
[53]

Y . Tang, Z. Wang, A. Qu, Y . Yan, Z. Wu, D. Zhuang, J. Kai, K. Hou, X. Guo, J. Zhao, Z. Zhao, and W. Ma. ItiNera: Integrating spatial optimization with large language models for open- domain urban itinerary planning. In F. Dernoncourt, D. Preo t ¸iuc-Pietro, and A. Shimorina,...

2024 doi
[54]

Zhong, C

V . Zhong, C. Xiong, and R. Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning.arXiv preprint arXiv:1709.00103, 2017. (pp. 2 and 3)

2017 arXiv
[55]

bird”, “galois

F. Zhu, W. Lei, Y . Huang, C. Wang, S. Zhang, J. Lv, F. Feng, and T.-S. Chua. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th I...

2021 doi
[58]

T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In2018 Conference on Empirical Methods in Natural Language Processing, ...

2018
[59]

Zhang, S

X. Zhang, S. Luo, B. Zhang, Z. Ma, J. Zhang, Y . Li, G. Li, Z. Yao, K. Xu, J. Zhou, D. Zhang-Li, J. Yu, S. Zhao, J. Li, and J. Tang. Tablellm: Enabling tabular data manipulation by llms in real office usage scenarios, 2025. URLhttps://arxiv.org/abs/2403.19318. (p. 2)

2025 arXiv
[2020]

URLhttps://openreview.net/forum?id=025X0zPfn. (p. 4)
[2022]

URLhttps://arxiv.org/abs/2109.07958. (p. 3)

arXiv
[2023]

URL https://doi.org/10.48550/arXiv.2310

doi: 10.48550/ARXIV .2310.06825. URL https://doi.org/10.48550/arXiv.2310. 06825. (p. 6)

doi arXiv
[2024]

doi: 10.3390/computers13100257

ISSN 2073-431X. doi: 10.3390/computers13100257. URL https://www.mdpi.com/ 2073-431X/13/10/257. (p. 2)

doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Ai hallucination report 2025, 2025

All About AI. Ai hallucination report 2025, 2025. URL https://www.allaboutai.com/ resources/ai-statistics/ai-hallucinations/. (pp. 1 and 3)

work page 2025

[2] [2]

R. Aly, Z. Guo, M. S. Schlichtkrull, J. Thorne, A. Vlachos, C. Christodoulopoulos, O. Cocarascu, and G. Li. FEVEROUS: Fact extraction and VERification over unstructured and structured information. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. (p. 2)

work page 2021

[3] [3]

Balsiger, H.-R

D. Balsiger, H.-R. Dimmler, S. Egger-Horstmann, and T. Hanne. Assessing large language models used for extracting table information from annual financial reports.Computers, 13(10),

work page

[4] [4]

Bisercic, M

A. Bisercic, M. Nikolic, M. van der Schaar, B. Delibasic, P. Lio, and A. Petrovic. Interpretable medical diagnostics with structured data extraction by large language models, 2023. URL https://arxiv.org/abs/2306.05052. (p. 2)

arXiv 2023

[5] [5]

Borisov, K

V . Borisov, K. Sessler, T. Leemann, M. Pawelczyk, and G. Kasneci. Language models are realistic tabular data generators. InThe Eleventh International Conference on Learning Repre- sentations, 2023. URLhttps://openreview.net/forum?id=cEygmQNOeI. (p. 4)

work page 2023

[6] [6]

Buoncristiano, G

M. Buoncristiano, G. Mecca, D. Santoro, and E. Veltri. Detective gadget: Generic iterative entity resolution over dirty data.Data, 2024. doi: 10.3390/data9120139. (p. 6)

work page doi:10.3390/data9120139 2024

[7] [7]

Cappuzzo, G

R. Cappuzzo, G. Varoquaux, A. Coelho, and P. Papotti. Retrieve, merge, predict: Augmenting tables with data lakes.CoRR, abs/2402.06282, 2024. doi: 10.48550/ARXIV .2402.06282. URL https://doi.org/10.48550/arXiv.2402.06282. (p. 4)

arXiv doi:10.48550/arxiv.2402.06282 2024

[8] [8]

Chang, X

Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang, W. Ye, Y . Zhang, Y . Chang, P. S. Yu, Q. Yang, and X. Xie. A survey on evaluation of large language models.ACM Trans. Intell. Syst. Technol., 15(3), mar 2024. ISSN 2157-6904. doi: 10.1145/3641289. URLhttps://doi.org/10.1145/3641289. (p. 3)

doi:10.1145/3641289 2024