LLM Inference Enhanced by External Knowledge: A Survey

Jia-Ren Zhang; Liang-Yu Hsia; Qian-Hui Chen; Yi-Hung Liu; Yi-Jie Cheng; Yu-Hsuan Lin; Yun-Nung Chen

REVIEW 3 major objections 5 minor 1 cited by

LLM Inference Enhanced by External Knowledge: A Survey

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This survey argues that hybrid symbolic-plus-neural reasoning currently delivers the most robust LLM performance on table question answering, and that tightly coupled LLM–knowledge-graph loops yield more accurate and robust reasoning than…

desk verdict A serviceable survey of inference-time knowledge integration for LLMs whose practical 'hybrid is best' conclusion is not supported by its own non-comparable benchmark table. read the letter →

arxiv 2505.24377 v1 pith:QDAN7AH3 submitted 2025-05-30 cs.CL

Yu-Hsuan Lin , Qian-Hui Chen , Yi-Jie Cheng , Jia-Ren Zhang , Yi-Hung Liu , Liang-Yu Hsia , Yun-Nung Chen This is my paper

classification cs.CL

keywords largelanguagemodelsexternalknowledgetablereasoninggraphshybridlooseandtightcouplingretrieval-augmentedgenerationtaxonomy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper offers a systematic map of how external knowledge can be plugged into large language models at inference time, focusing on two structured sources: tables and knowledge graphs. It sorts table methods into symbolic (SQL generation), neural (direct LLM reasoning), and hybrid pipelines, and sorts KG methods by how tightly the LLM and the graph interact. On the basis of benchmark numbers compiled from prior work, it argues that hybrid table methods such as TabSQLify and H-STAR are currently the most robust across question types and datasets, and that tight coupling such as Think-on-Graph tends to beat loose coupling. The survey's practical payoff is a rough design rule: prefer hybrid or tightly coupled integration when accuracy matters, at the cost of higher complexity and latency.

What carries the argument

The load-bearing machinery is the taxonomy itself plus a comparative benchmark table. The survey partitions external knowledge into unstructured and structured data, then further partitions structured integration: for tables, symbolic vs neural vs hybrid reasoning; for knowledge graphs, loose vs tight coupling. These two axes organize the entire method review, and Table 1 (GPT-3.5-Turbo results on WikiTQ and TabFact) and Table 2 (qualitative comparison of KG methods) supply the evidence for the performance ordering. The taxonomy does the argumentative work by turning a scattered literature into a set of design choices with measurable trade-offs.

What would settle it

Re-run the GPT-3.5-Turbo comparisons on WikiTQ and TabFact with every method using the same number of inference samples (e.g., greedy decoding for all, or self-consistency for all), and check whether hybrid methods still occupy the top of both leaderboards; if neural or symbolic methods overtake them under matched settings, the paper's central ordering is an artefact of decoding budgets.

Watch

Extended reading notes

Core claim

The central claim is that the design space of knowledge-enhanced LLM inference can be organized by two axes—structuredness of the knowledge source and coupling/reasoning style—and that along these axes a clear performance ordering emerges. For tables, methods that combine symbolic execution (e.g., SQL) with neural reasoning outperform purely symbolic and purely neural approaches: the best scores in the compiled GPT-3.5-Turbo comparison (H-STAR at 69.6 on WikiTQ and 85.0 on TabFact) come from hybrid designs, and hybrid methods occupy the top of the table. For knowledge graphs, methods that let the LLM iteratively explore and reason over the graph (tight coupling, exemplified by Think-on-Graph and its extensions) yield more accurate and robust reasoning than one-shot retrieval-and-prompting (loose coupling). The paper does not claim hybrid and tight integration dominate unconditionally: it notes the advantage depends on coordination between components and warns that symbolic errors can propagate into the neural stage.

Load-bearing premise

The benchmark comparison assumes that the accuracy numbers taken from different papers were produced under the same inference conditions, even though some methods used a self-consistency decoding that reran the model many times and others did not.

Editorial extensions

If this is right

If the paper's comparative conclusion holds, practitioners building table QA systems should start from a hybrid design (SQL-based subtable extraction plus LLM reasoning) rather than pure text-to-SQL or pure end-to-end prompting.
For knowledge-graph QA, investing in an iterative LLM-as-agent loop over the graph should beat a single retrieval-and-prompt pass, and extensions with self-correction (Plan-on-Graph) or internally generated missing facts (Generate-on-Graph) push accuracy further.
The error-propagation finding implies future hybrid systems need fault-tolerant coordination, e.g., letting the neural stage detect and repair bad symbolic intermediate results.
Tightly coupled KG methods can lift smaller models to near the level of larger ones (ToG-2.0 claim), so coupling strategy is a cost lever independent of model size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The compiled benchmark in Table 1 mixes experiments with and without self-consistency (DATER and BINDER used 20 or 50 inference iterations), so the hybrid-vs-neural ordering may in part reflect decoding budget rather than architecture; rerunning all methods with matched inference settings is a testable check.
The taxonomy suggests a natural extension to multimodal structured data: the same loose/tight coupling distinction could categorize methods that align images or audio with graph entities, a direction the paper lists as open.
The paper's table-centred finding that hybrid methods are most robust on both fact verification (TabFact) and QA (WikiTQ) hints that hybrid coordination may generalize to other symbolic executors, such as Python or SPARQL, beyond SQL.
A direct head-to-head comparison of table vs KG enhancements on the same task suite would clarify when each structured source earns its integration cost.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a serviceable survey of inference-time external-knowledge integration for LLMs, with a clear taxonomy and a wide literature net. But the one conclusion that would make it actionable—hybrid table reasoning is the most robust—doesn't survive its own Table 1.

What it does well: the structured/unstructured split, the symbolic/neural/hybrid classification for tables, loose/tight coupling for KGs, and the method-by-method summaries in Section 3 are all readable and mostly accurate. The limitations section is genuinely honest about what the survey doesn't compare (table size, multi-hop complexity, table-vs-KG). That level of candor is rare and worth respecting. There's also a GitHub repo, which helps.

The soft spot is the comparative analysis in Section 4.1. The text promises a 'unified experimental setting' for Table 1, but the table's own footnote states that BINDER and DATER were reproduced with self-consistency—50 and 20 inference iterations, respectively—while the other entries were taken from their original papers, which did not use that decoding scheme. Self-consistency gives a known boost to GPT-3.5-class models, so the hybrid-vs-neural gap is confounded with inference budget. The claim that hybrid reasoning 'currently delivers the most robust performance' is therefore not supported by the presented evidence. It may be true, but this table doesn't prove it. This is fixable: relabel the table as a compilation of published results, delete 'unified experimental setting,' and soften the conclusion. Better yet, rerun methods under one decoding protocol. The selection of methods also lacks stated inclusion criteria; calling it 'representative' would be more honest than implying completeness.

On novelty: the taxonomy overlaps substantially with the table-reasoning survey and the KG-LLM roadmap that the paper itself cites. Surveys don't need to invent new categories, and the consolidation is still useful, but the contribution is organizational, not conceptual.

Who this is for: newcomers who want a map of the field and a starting list of methods. It's a solid introduction, not a definitive reference. If this came to me for peer review, I'd send it out—the authors did the cataloguing work, and a referee can insist on fixing the comparative claims. I wouldn't cite its benchmark numbers in my own work until they're regenerated under controlled conditions.

Referee Report

3 major / 5 minor

Summary. This survey examines methods that integrate external knowledge into LLM inference, with a focus on structured sources: tables and knowledge graphs. It introduces taxonomies (for tables: symbolic, neural, and hybrid reasoning; for KGs: loose and tight coupling), reviews representative methods, and presents comparative tables: Table 1 for table reasoning on WikiTQ and TabFact under GPT-3.5-Turbo, and Table 2 for KG-based methods. The paper concludes that hybrid table reasoning is currently the most robust and that tight coupling in KG reasoning yields more accurate and robust behavior. It also discusses limitations and future directions, including error propagation, input size constraints, efficiency, multimodal knowledge, and real-time reasoning.

Significance. The paper's taxonomy is internally consistent and provides a useful organizational frame for a rapidly growing literature on inference-time knowledge integration. Its coverage of recent table- and KG-based methods, together with the benchmark summary in Appendix A, is a helpful starting point for practitioners. The authors are transparent about several limitations in the final section. However, the paper's comparative claims rest on evidence that is not fully controlled: Table 1 is not the 'unified experimental setting' it claims to be, and Table 2 relies on qualitative, self-reported characterizations. The survey's value is primarily descriptive; its strongest conclusions currently outrun its evidence.

major comments (3)

[Section 4, Table 1] The claim in Section 4 (first paragraph) that methods are benchmarked 'under a unified experimental setting' is contradicted by the table's own footnotes. BINDER and DATER are reported with self-consistency (50 and 20 inference iterations, respectively), while other methods in the same table, including Chain-of-Table, TabSQLify, and H-STAR, are reported from their original papers or from Abhyankar et al. (2025) without that procedure. Since self-consistency is known to improve accuracy for GPT-3.5-class models, the performance gaps underlying the conclusion that 'hybrid reasoning currently delivers the most robust performance' (Section 4.1) are confounded with decoding procedure. The comparison also mixes different prompt formats, table serializations, and sampling budgets. Please either provide a matched-setting comparison, or explicitly restrict the conclusion to the reported settings and discuss the direction and possible size of the confound.
[Section 3.2.2, Table 2] The statement that 'tight coupling typically yields more accurate and robust reasoning' is not supported by the evidence in Table 2. The 'Performance' column contains qualitative, largely self-reported characterizations such as 'Better than ToG' and 'Achieves SOTA in multiple datasets,' with several entries marked 'Not mentioned.' No unified benchmark or quantitative comparison is provided for KG-based methods, and the paper's own Limitations section acknowledges that no direct comparative evaluation was performed between table-based and KG-based methods. This comparative claim should be rephrased as a qualitative trend inferred from the cited literature, or supported by a controlled comparison.
[Section 4.1, Figure 4] The narrative illustration of TabSQLify versus H-STAR in Figure 4 is a useful qualitative example, but it is not evidence that hybrid reasoning 'currently delivers the most robust performance across diverse question types and datasets.' The phrase 'across diverse question types and datasets' is too strong given that Table 1 covers only two datasets and the comparative setting is not controlled. Please temper the conclusion to reflect the scope of the presented evidence.

minor comments (5)

[Section 3.2] The headings 'KG L LLM' and 'KG N LLM' use nonstandard symbols that are not defined in the text; please replace them with clearer notation such as 'loose coupling (KG + LLM)' and 'tight coupling (KG ⊗ LLM)' or define the symbols at first use.
[Table 1 caption] The caption says 'GPT-3.5-Turbo results' but the footnote shows that some numbers are reproduced from other papers that may use different model versions or decoding settings; please state the original source for each row and clarify which numbers are directly reproduced versus newly computed.
[Section 5, first paragraph] The phrase 'our analysis reveals a critical issue about error propagation' is supported by a single qualitative example; please soften it to 'a potential issue' or support it with additional references or a broader analysis.
[Figure 2] There is a typographical error in the figure: 'Hybird' should be 'Hybrid'.
[Section 2.2.2] The sentence 'converting their structured relational data into neural representations suitable for LLM' uses a singular 'LLM'; please use 'LLMs' for consistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey's taxonomy and comparative claims are descriptive compilations of external results, not derivations from their own inputs.

full rationale

This is a survey paper, not a derivation. Its taxonomy categorizes external knowledge into unstructured/structured sources, tables/KGs, and integration paradigms (symbolic/neural/hybrid for tables; loose/tight coupling for KGs). These categories are grounded in the cited literature and are not defined in terms of the paper's conclusions. The central comparative claim that 'hybrid reasoning currently delivers the most robust performance across diverse question types and datasets' (Section 4.1) rests on Table 1, which compiles published GPT-3.5-Turbo results. The table's own footnote discloses that BINDER and DATER numbers were reproduced with self-consistency (50 and 20 inference iterations, respectively), and the Limitations section explicitly concedes that the paper did not account for input table size, multi-hop complexity, or a direct table-versus-KG comparison. Those are threats to the comparability and validity of the empirical comparison, but they are not circularity: no parameter is fitted and then renamed a prediction, no uniqueness theorem is imported from the authors' prior work, no self-citation is load-bearing, and no result is equivalent to its input by construction. None of the seven enumerated circularity patterns is present, so the honest finding is no significant circularity with score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The survey introduces no free parameters, invented entities, or new formal axioms. The assumptions listed are the domain-level premises needed for the comparative analysis to be valid: comparability of external numbers, exhaustiveness of the taxonomy, and representativeness of the selected methods.

assumptions (3)

domain assumption The reported performance numbers from external papers are accurate and directly comparable.
The survey's comparative conclusions in Section 4.1 rely on Table 1, which reproduces numbers from multiple sources with differing inference settings, without independent verification or adjustment.
domain assumption The symbolic/neural/hybrid and loose/tight coupling categories are exhaustive and mutually exclusive for the reviewed methods.
The taxonomy in Section 3 underpins the structure of the survey, but the paper does not prove that all methods fit cleanly into exactly one category.
domain assumption The selected methods are representative of the broader literature.
The survey reviews a hand-picked set of methods without specifying inclusion criteria, so it assumes these are sufficient to support the overall comparative claims.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLM Inference Enhanced by External Knowledge: A Survey." pith.science (2026). https://pith.science/paper/QDAN7AH3

@misc{pith2026250524377,
  author       = {Pith},
  title        = {Pith review of: LLM Inference Enhanced by External Knowledge: A Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QDAN7AH3}},
  note         = {Machine review of arXiv:2505.24377}
}

read the original abstract

Recent advancements in large language models (LLMs) have enhanced natural-language reasoning. However, their limited parametric memory and susceptibility to hallucination present persistent challenges for tasks requiring accurate, context-based inference. To overcome these limitations, an increasing number of studies have proposed leveraging external knowledge to enhance LLMs. This study offers a systematic exploration of strategies for using external knowledge to enhance LLMs, beginning with a taxonomy that categorizes external knowledge into unstructured and structured data. We then focus on structured knowledge, presenting distinct taxonomies for tables and knowledge graphs (KGs), detailing their integration paradigms with LLMs, and reviewing representative methods. Our comparative analysis further highlights the trade-offs among interpretability, scalability, and performance, providing insights for developing trustworthy and generalizable knowledge-enhanced LLMs.

Figures

Figures reproduced from arXiv: 2505.24377 by the authors.

**Figure 1.** Taxonomy of external knowledge source. cific integration strategies and their effectiveness will be discussed in detail in section 3. 2.2.2 Knowledge Graphs (KGs) KGs (Pan et al., 2024) represent another significant and widely adopted form of structured data utilized to enhance LLM inference. A KG encodes information as a structured network of interconnected entities. Relationships between these entities are expl… view at source ↗

**Figure 2.** Overview of table reasoning paradigms: (a) Symbolic reasoning, where the LLM generates and executes SQL queries over the table. (b) Neural reasoning, where the LLM directly reasons over the table and question in natural language. (c) Hybrid reasoning, where the LLM combines SQL-based subtable extraction with further neural reasoning. classification reflects a key design trade-off in table question answering: Whethe… view at source ↗

**Figure 3.** Overview of KG-LLM integration strategies: (a) Loose coupling: the retrieved KG is directly used by the LLM without refinement. (b) Tight coupling: the LLM actively refines or integrates retrieved KG information. In both strategies, the retriever is often implemented using an LLM-based model. We classify these methods into loose coupling and tight coupling strategies, reflecting the degree of interaction between L… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Illustration of two example questions from WikiTQ. The left example can be directly answered via SQL, [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Matching Game Preferences Through Dialogical Large Language Models: A Perspective
cs.AI 2025-07 conditional novelty 4.0 of 10

This perspective paper proposes the D-LLM framework, which couples the authors' GRAPHYP knowledge graphs with LLMs to personalize AI responses and make reasoning traceable, but no empirical validation is presented.

Reference graph

Works this paper leans on

53 extracted references · 11 canonical work pages · cited by 1 Pith paper

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Nikhil Abhyankar, Vivek Gupta, Dan Roth, and Chandan K. Reddy. 2025. https://aclanthology.org/2025.naacl-long.445/ H - STAR : LLM -driven hybrid SQL -text adaptive reasoning on tables . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long ...

work page 2025
[4]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[5]

Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023. https://doi.org/10.18653/v1/2023.nlrse-1.7 Knowledge-augmented language model prompting for zero-shot knowledge graph question answering . In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pages 78--106, Toronto, Canada. Association for Computational...

doi:10.18653/v1/2023.nlrse-1.7 2023
[6]

Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533--1544

2013
[7]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 a . Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

work page 2020
[8]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

arXiv 2020

Show all 53 references

[9]

Liyi Chen, Panrong Tong, Zhongming Jin, Ying Sun, Jieping Ye, and Hui Xiong. 2024. https://openreview.net/forum?id=CwCUEr6wO5 Plan-on-graph: Self-correcting adaptive planning of large language model on knowledge graphs . In The Thirty-eighth Annual Conference on Neural Informa...

2024
[10]

Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.02164

2019 arXiv
[11]

Smith, and Tao Yu

Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. https://arxiv.org/abs/2210.02875 Binding language models in symbolic languages . Preprint, arXiv:2210.02875

2023 arXiv
[12]

Rosen, Gerbrand Ceder, Kristin A

John Dagdelen, Alexander Dunn, Sanghoon Lee, Nicholas Walker, Andrew S. Rosen, Gerbrand Ceder, Kristin A. Persson, and Anubhav Jain. 2024. https://doi.org/10.1038/s41467-024-45563-x Structured information extraction from scientific text with large language models . Nature Comm...

2024 doi
[13]

Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,...

2024
[14]

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. https://arxiv.org/abs/2312.10997 Retrieval-augmented generation for large language models: A survey . Preprint, arXiv:2312.10997

2024 arXiv
[15]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

2024 arXiv
[16]

Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d’Amato, Gerard De Melo, Claudio Gutierrez, Sabrina Kirrane, Jos \'e Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, et al. 2021. Knowledge graphs. ACM Computing Surveys (Csur), 54(4):1--37

2021
[17]

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Info...

2025
[18]

Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Xin Zhao, and Ji-Rong Wen. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.574 S truct GPT : A general framework for large language model to reason over structured data . In Proceedings of the 2023 Conference on Empirical Method...

2023 doi
[19]

Kelvin Jiang, Dekun Wu, and Hui Jiang. 2019. https://doi.org/10.18653/v1/N19-1028 F reebase QA : A new factoid QA data set matching trivia-style question-answer pairs with F reebase . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for C...

2019 doi
[20]

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551

2017 arXiv
[21]

Kun Li, Tianhua Zhang, Xixin Wu, Hongyin Luo, James Glass, and Helen Meng. 2024 a . Decoding on graphs: Faithful and sound reasoning on knowledge graphs through generation of well-formed chains. arXiv preprint arXiv:2410.18415

2024 arXiv
[22]

Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing. 2024 b . https://openreview.net/forum?id=cPgh4gWZlz Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources . In The Twelf...

2024
[23]

Weizheng Lu, Jing Zhang, Ju Fan, Zihao Fu, Yueguo Chen, and Xiaoyong Du. 2025. https://doi.org/10.1007/s11704-024-40763-6 Large language model for table processing: a survey . Frontiers of Computer Science, 19(2)

2025 doi
[24]

Shengjie Ma, Chengjin Xu, Xuhui Jiang, Muzhi Li, Huaren Qu, Cehao Yang, Jiaxin Mao, and Jian Guo. 2025. https://openreview.net/forum?id=oFBu7qaZpS Think-on-graph 2.0: Deep and faithful large language model reasoning with knowledge-guided retrieval augmented generation . In The...

2025
[25]

Md Mahadi Hasan Nahid and Davood Rafiei. 2024 a . https://doi.org/10.18653/v1/2024.findings-emnlp.203 N orm T ab: Improving symbolic reasoning in LLM s through tabular data normalization . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3569--35...

2024 doi
[26]

Md Mahadi Hasan Nahid and Davood Rafiei. 2024 b . https://doi.org/10.18653/v1/2024.naacl-long.320 T ab SQL ify: Enhancing reasoning capabilities of LLM s through table decomposition . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Co...

2024 doi
[27]

Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kry \'s ci \'n ski, Hailey Schoelkopf, Riley Kong, Xiangru Tang, et al. 2022. Fetaqa: Free-form table question answering. Transactions of the Association for Computational Linguistics, 10:35--49

2022
[28]

Giang Nguyen, Ivan Brugere, Shubham Sharma, Sanjay Kariyappa, Anh Totti Nguyen, and Freddy Lecue. 2025. https://arxiv.org/abs/2412.12386 Interpretable llm-based table question answering . Preprint, arXiv:2412.12386

2025 arXiv
[29]

Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2024. https://doi.org/10.1109/tkde.2024.3352100 Unifying large language models and knowledge graphs: A roadmap . IEEE Transactions on Knowledge and Data Engineering, 36(7):3580–3599

2024
[30]

Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508.00305

2015 arXiv
[31]

Haritz Puerto, G \"o zde S ahin, and Iryna Gurevych. 2023. https://doi.org/10.18653/v1/2023.eacl-main.259 M eta QA : Combining expert agents for multi-skill question answering . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational ...

2023 doi
[32]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

2019
[33]

Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022. https://arxiv.org/abs/2204.00498 Evaluating the text-to-sql capabilities of large language models . Preprint, arXiv:2204.00498

2022 arXiv
[34]

Hend Selmy, Hoda Mohamed, and Walaa Medhat. 2024. https://doi.org/10.1016/j.is.2023.102318 Big data analytics deep learning techniques and applications: A survey . Information Systems, 120:102318

2024
[35]

Priyanka Sen, Sandeep Mavadia, and Amir Saffari. 2023. https://doi.org/10.18653/v1/2023.nlrse-1.1 Knowledge graph-augmented language models for complex question answering . In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pa...

2023 doi
[36]

Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel Ni, Heung-Yeung Shum, and Jian Guo. 2024. https://openreview.net/forum?id=nnVO1PvbTv Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph . In The Twelfth...

2024
[37]

Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. arXiv preprint arXiv:1803.06643

2018 arXiv
[38]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

2023 arXiv
[39]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[40]

Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2024. https://openreview.net/forum?id=4L0xnS4GQM Chain-of-table: Evolving tables in the reasoning chain...

2024
[41]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. https://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models . Preprint, arXiv:2201.11903

2023 arXiv
[42]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

2022
[43]

Yike Wu, Nan Hu, Sheng Bi, Guilin Qi, Jie Ren, Anhuan Xie, and Wei Song. 2023. https://doi.org/10.48550/arXiv.2309.11206 Retrieve-rewrite-answer: A kg-to-text enhanced llms framework for knowledge graph question answering . CoRR, abs/2309.11206

2023 doi arXiv
[44]

Zirui Wu and Yansong Feng. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.253 P ro T rix: Building models for planning and reasoning over tables with sentence context . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4378--4406, Miami, Fl...

2024 doi
[45]

Yao Xu, Shizhu He, Jiabei Chen, Zihao Wang, Yangqiu Song, Hanghang Tong, Kang Liu, and Jun Zhao. 2024. https://doi.org/10.48550/arXiv.2404.14741 Generate-on-graph: Treat llm as both agent and kg in incomplete knowledge graph question answering . CoRR, abs/2404.14741

2024 doi arXiv
[46]

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600

2018 arXiv
[47]

Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023. https://arxiv.org/abs/2301.13808 Large language models are versatile decomposers: Decompose evidence and questions for table-based reasoning . Preprint, arXiv:2301.13808

2023 arXiv
[48]

Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers...

2016
[49]

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. https://doi.org/10.1093/nsr/nwae403 A survey on multimodal large language models . National Science Review, 11(12)

2024 doi
[50]

Han Zhang, Yuheng Ma, and Hanfang Yang. 2025. https://aclanthology.org/2025.naacl-long.9/ ALTER : Augmentation for large-table-based reasoning . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human...

2025
[51]

Xuanliang Zhang, Dingzirui Wang, Longxu Dou, Qingfu Zhu, and Wanxiang Che. 2024. https://arxiv.org/abs/2402.08259 A survey of table reasoning with large language models . Preprint, arXiv:2402.08259

2024 arXiv
[52]

Yunjia Zhang, Jordan Henkel, Avrilia Floratou, Joyce Cahoon, Shaleen Deep, and Jignesh M. Patel. 2023. https://arxiv.org/abs/2310.00815 Reactable: Enhancing react for table question answering . Preprint, arXiv:2310.00815

2023 arXiv
[53]

Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103

2017 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Nikhil Abhyankar, Vivek Gupta, Dan Roth, and Chandan K. Reddy. 2025. https://aclanthology.org/2025.naacl-long.445/ H - STAR : LLM -driven hybrid SQL -text adaptive reasoning on tables . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long ...

work page 2025

[4] [4]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[5] [5]

Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023. https://doi.org/10.18653/v1/2023.nlrse-1.7 Knowledge-augmented language model prompting for zero-shot knowledge graph question answering . In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pages 78--106, Toronto, Canada. Association for Computational...

doi:10.18653/v1/2023.nlrse-1.7 2023

[6] [6]

Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533--1544

2013

[7] [7]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 a . Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

work page 2020

[8] [8]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

arXiv 2020