Pith. sign in

REVIEW 4 major objections 3 minor 37 references

SKA-Bench: A Fine-Grained Benchmark for Evaluating Structured Knowledge Understanding of LLMs

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Even top LLMs fail on long, noisy, shuffled knowledge

desk verdict SKA-Bench is a genuinely useful evaluation resource, but its Negative Rejection testbed has a validation gap in the KG and KG+Text subsets that weakens the paper's strongest hallucination claim. read the letter →

arxiv 2507.17178 v3 pith:25LT2T2K submitted 2025-07-23 cs.CL cs.AI

Zhiqiang Liu , Enpei Niu , Yin Hua , Mengshu Sun , Lei Liang , Huajun Chen , Wen Zhang This is my paper
classification cs.CLcs.AI
keywords structuredknowledgeunderstandingLLMbenchmarkgraphQAtablenoiserobustnessorderinsensitivitynegativerejectionhallucination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SKA-Bench is a benchmark for diagnosing how well large language models understand structured knowledge such as knowledge graphs and tables, a capability that existing single-format and short-context benchmarks cannot isolate. Each of its 921 instances pairs a question with an answer, a set of human-annotated positive knowledge units that entail the answer, and a set of noisy units that should not. From these instances the paper builds four testbeds—noise robustness, order insensitivity, information integration, and negative rejection—so that a single number is replaced by fine-grained capability profiles. Evaluating eight LLMs, the paper finds that even the strongest model, DeepSeek-R1, degrades sharply as noise grows, is vulnerable to reordering of knowledge units, and fails to reject roughly a fifth of questions whose knowledge base contains only noise. The central claim is that current LLMs do not yet reliably understand structured knowledge when it is long, noisy, or reordered, and that this benchmark can pinpoint which of these factors is the bottleneck.

What carries the argument

The load-bearing object is the SKA-Bench instance: a question, a gold answer, a set of human-annotated positive knowledge units (KG triples, table rows, and/or text spans) that together entail the answer, and a set of noisy units that are meant to be irrelevant. A three-stage pipeline produces these instances: (1) questions and answers are drawn from six existing QA datasets covering KG, table, KG+text, and table+text; (2) human experts iteratively label the minimal positive units, with an LLM verifying that the units entail the answer; (3) LLM-synthesized noisy units are generated under explicit 'conditions' that rule out accidental answerability, followed by manual review that removed 7.5% of unsafe synthesized table noise. The four testbeds then manipulate these instances in controlled ways—scaling noise to {1k, 4k, 12k, 24k} token budgets, permuting unit order into {prefix, random, suffix}, grouping by the number of required positive units, and removing all positive units—so that each diagnostic isolates a specific understanding failure.

What would settle it

Run the negative rejection testbed as a sanity check: if a strong LLM given only the noisy units can correctly answer any of the questions, that instance's noise is unsafe, and the rejection-rate and noise-robustness scores for that instance are invalid.

Watch

Extended reading notes

Core claim

The paper's central discovery is that existing LLMs, including DeepSeek-R1 and GPT-4o, remain fragile when structured knowledge is presented at scale and in realistic disorder. On the noise robustness testbed, DeepSeek-R1's F1 score on the KG subset falls from 89.95 at 1k tokens to 64.67 at 24k tokens, and smaller models drop by as much as 58.77 percentage points on the Table+Text subset between 4k and 24k tokens. On the order insensitivity testbed, models perform noticeably better when the positive knowledge units sit at the beginning or end of the context than when they are scattered through the middle, replicating the 'lost in the middle' effect for structured data. On the negative rejection testbed, where only noisy units are provided, DeepSeek-R1 refuses to answer only 78.71% of the time, lower than TableGPT-2's 83.40%, indicating that even advanced models hallucinate answers rather than abstain. The paper interprets these results as evidence that structured knowledge understanding is not a single skill but a set of distinct abilities—noise tolerance, order invariance, multi-unit integration, and refusal—that current LLMs master unevenly.

Load-bearing premise

The benchmark's validity rests on the assumption that every question is answerable from its annotated positive units and that no synthesized noisy unit can accidentally supply the answer; the paper's own manual review found 7.5% of table noise unsafe, so this assumption is imperfectly guaranteed.

Editorial extensions

If this is right

  • On the noise robustness testbed, every evaluated model's accuracy falls as the structured knowledge base grows from 1k to 24k tokens, with the largest drop exceeding 58 percentage points.
  • On the order insensitivity testbed, models answer more accurately when positive units appear at the start or end of the context than when they are scattered, a 'lost in the middle' effect for structured knowledge.
  • On the information integration testbed, performance declines as the number of required positive units grows, and heterogeneous (KG+text or table+text) integration is especially hard for models under 10 billion parameters.
  • On the negative rejection testbed, the best model refuses only 78.71% of pure-noise inputs, meaning more than one in five unanswerable structured questions receives a fabricated answer.
  • Because each instance stores positive and noisy units separately, the same data can be rescaled to longer contexts or reformatted into tasks such as text-to-query and structured-knowledge retrieval, as the paper's limitations section notes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension not tested in the paper would be to measure whether the 'lost in the middle' effect is caused by token position or by semantic interference from noisy triples, since the benchmark's shuffle conditions conflate the two.
  • The 78.71% rejection rate for DeepSeek-R1 suggests that abstention is a learnable behaviour; a plausible follow-up is to fine-tune a model on SKA-Bench's negative examples and check whether rejection improves without hurting accuracy on positive-only inputs.
  • Because the benchmark is English-only and built from six source datasets, the observed ability profile may not transfer to non-English structured knowledge or to domain-specific graphs such as biomedical or financial ones; a cross-lingual, cross-domain variant would test the generality of these findings.
  • The instance format separates retrieval from reasoning: feeding only positive units versus full noisy sets could be used to evaluate a retriever's contribution separately from an LLM's reasoning, though the paper does not perform this decomposition.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper introduces SKA-Bench, a structured-knowledge-augmented QA benchmark covering four data formats (KG, Table, KG+Text, Table+Text). Instances are built in three stages: collection of QA pairs from existing datasets, iterative human-plus-LLM annotation of positive knowledge units, and construction of noisy units (LLM-synthesized and condition-checked for tables; all non-positive triples for KG). From these instances the authors construct four testbeds: Noise Robustness, Order Insensitivity, Information Integration, and Negative Rejection. Eight LLMs are evaluated with macro-F1 and rejection-rate metrics. The reported findings are that model performance degrades with increasing noise/context length, is sensitive to the placement of positive units, and is often poor at rejecting unanswerable questions, with DeepSeek-R1 scoring 78.71% average rejection versus TableGPT-2's 83.40%.

Significance. The benchmark is potentially valuable: it provides fine-grained, multi-format diagnostics for structured knowledge understanding, with explicit positive-unit annotation, four ability-oriented testbeds, and public code/data. The noise-robustness and order-sensitivity findings, if reliable, have direct implications for retrieval-augmented systems that feed long, reordered, or noisy knowledge contexts to LLMs. The paper also ships machine-checkable evaluation code and detailed prompts, which is a strength. However, the negative-rejection testbed, which supports the hallucination claim, is not valid for the KG and KG+Text subsets as constructed; the lack of any answerability check on non-positive KG units means that a model may correctly answer from a 'noisy' unit and be penalized as a rejection failure. Single-seed shuffling and the absence of inter-annotator agreement further weaken the quantitative claims. The core contribution is defensible, but the validity of the negative-rejection results and the statistical grounding of the order-sensitivity results require substantive revision.

major comments (4)
  1. [§3.2.3 and §3.4, Table 5] For the KG and KG+Text subsets, noisy units are defined as every non-positive triple or textual unit in the extracted subgraph, with no check that these units cannot answer the question. Section 3.4 then asserts that providing only such units 'ensures broken reasoning paths.' This does not follow: a subgraph extracted around topic or answer entities can contain alternative reasoning paths, including inverse triples or textual descriptions that mention the gold answer. If such a unit is present, a model that outputs the answer or says 'yes' is correct, yet it is scored as a rejection failure. This directly threatens Table 5, where DeepSeek-R1's 78.71% rejection rate is compared with TableGPT-2's 83.40% and used to support the claim that advanced LLMs are vulnerable to hallucination in structured settings. The manual review with 92.5% accuracy reported in §3.2.3 applies only to synthesized table noise; no analogous answerability validation is reported for KG/KG+Text noise. Please add a systematic answerability check (LLM-based and/or human-reviewed) for all non-positive KG and KG+Text units, report the leakage rate, and re-compute the negative-rejection results after removing or re-labeling leaked instances.
  2. [§3.4 and §4.1] All shuffles in the Order Insensitivity and Noise Robustness testbeds use a single random seed (42). The 'random' condition is therefore one specific permutation, and no variance across shuffles is reported. Although the prefix/suffix versus random gap is consistent across models and subsets, the central claim of order sensitivity rests on a single draw. Please run at least three to five seeds and report mean and standard deviation (or otherwise justify that the reported single-shuffle results are representative and stable). This is a load-bearing point for the order-insensitivity analysis.
  3. [§3.2.2] Three human experts perform positive-unit annotation, but no inter-annotator agreement metric (e.g., Cohen's kappa or overlap) is reported. Positive-unit annotations serve as ground truth for every testbed: the noise/positive distinction, the information-integration grouping by number of required units, and the premise that positive units alone are sufficient to derive the answer. Without an agreement measure or a description of how disagreements were adjudicated, the reliability of this ground truth is unestablished. Please report IAA and the adjudication procedure.
  4. [§3.2.1 and Appendix A] The paper states that KG subset samples are randomly selected from the test sets of WebQSP and CWQ, but Appendix A notes that the original CWQ test set does not contain answers. The manuscript does not explain how gold answers were obtained for the CWQ-derived samples. If answers were produced by executing SPARQL queries on Freebase, that procedure should be stated explicitly; if answers came from an LLM, the gold labels are unverified and the KG results could be affected. Please specify the answer-extraction procedure for CWQ.
minor comments (3)
  1. [Appendix D, Figure 22] The caption of Figure 22 says 'KG+Text subset' but the prompt shown is the Table+Text negative-rejection prompt (a table with footballers and a Bryan Robson text). Please correct the caption.
  2. [§4.1] The sentence 'we utilize the macro-F1 score as our metrics' should read 'as our metric'.
  3. [Appendix B, Table 7] The column labeled 'Std' is not defined in the table or the surrounding text; please define it as the standard deviation across the three noise-intensity conditions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SKA-Bench is externally grounded, and no reported prediction reduces to fitted inputs or to a self-citation chain.

full rationale

SKA-Bench is an empirical benchmark assembled from external source datasets (WebQSP, CWQ, WTQ, TableBench, STaRK, HybridQA), with positive units annotated by human experts and noisy units either drawn from real Freebase subgraphs or synthesized and then manually reviewed (92.5% accuracy for the table subsets). No parameter is fitted to the evaluated models, and no reported result is defined in terms of the models' outputs; the four testbeds are constructed by recombining annotated units into controlled conditions. The use of DeepSeek-v3 and GPT-4o during construction is a quality-check and synthesis step, not a load-bearing derivation of the evaluation outcomes, and the human review plus external ground truth keep the central measurements independent. The paper also acknowledges residual selection bias in its Limitations section. One validity limitation, not a circularity, is that for the KG and KG+Text subsets noise is defined as all non-positive triples without a documented check that no noisy triple can answer the question; the Section 3.4 assertion that excluding positive units 'ensur[es] broken reasoning paths' is therefore an unverified testbed assumption rather than a self-referential reduction. Because no central claim is equivalent by construction to its own inputs, the circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to data. The manual design choices (context scale thresholds) and structural assumptions (answer correctness proxies understanding; annotation is ground truth; noise is safe) are listed. No new physical or formal entities are introduced.

free parameters (1)
  • Context length-scale test sets
    Manually selected thresholds (1k, 4k, 12k, 24k tokens) define the noise robustness testbeds and are not derived from data or fitted to model behavior.
assumptions (4)
  • domain assumption Answer correctness is a valid proxy for structured knowledge understanding.
    The task assumes A = f(Q, SK) and that understanding SK is a prerequisite for correct answers (Section 3.1); this is a reasonable but unproven modeling assumption.
  • domain assumption Human expert annotation of positive units is ground truth.
    The pipeline relies on three experts and LLM verification (Section 3.2.2); no inter-annotator agreement is reported.
  • ad hoc to paper Synthesized noisy units do not alter answer correctness.
    The noise synthesis uses LLM-generated conditions and human review (Section 3.2.3), and the paper reports 92.5% review accuracy, so a residual risk remains.
  • domain assumption Macro-F1 and rejection rate adequately capture the four abilities.
    The metrics are standard but not validated against human judgment or calibrated for answer-list partial credit in this benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SKA-Bench: A Fine-Grained Benchmark for Evaluating Structured Knowledge Understanding of LLMs." pith.science (2026). https://pith.science/paper/25LT2T2K

@misc{pith2026250717178,
  author       = {Pith},
  title        = {Pith review of: SKA-Bench: A Fine-Grained Benchmark for Evaluating Structured Knowledge Understanding of LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/25LT2T2K}},
  note         = {Machine review of arXiv:2507.17178}
}
read the original abstract

Although large language models (LLMs) have made significant progress in understanding Structured Knowledge (SK) like KG and Table, existing evaluations for SK understanding are non-rigorous (i.e., lacking evaluations of specific capabilities) and focus on a single type of SK. Therefore, we aim to propose a more comprehensive and rigorous structured knowledge understanding benchmark to diagnose the shortcomings of LLMs. In this paper, we introduce SKA-Bench, a Structured Knowledge Augmented QA Benchmark that encompasses four widely used structured knowledge forms: KG, Table, KG+Text, and Table+Text. We utilize a three-stage pipeline to construct SKA-Bench instances, which includes a question, an answer, positive knowledge units, and noisy knowledge units. To evaluate the SK understanding capabilities of LLMs in a fine-grained manner, we expand the instances into four fundamental ability testbeds: Noise Robustness, Order Insensitivity, Information Integration, and Negative Rejection. Empirical evaluations on 8 representative LLMs, including the advanced DeepSeek-R1, indicate that existing LLMs still face significant challenges in understanding structured knowledge, and their performance is influenced by factors such as the amount of noise, the order of knowledge units, and hallucination phenomenon. Our dataset and code are available at https://github.com/zjukg/SKA-Bench.

Figures

Figures reproduced from arXiv: 2507.17178 by the authors.

Figure 1
Figure 1. The components of a SKA-Bench instance and how to further construct the four ability testbeds for evaluating structured knowledge understanding. real-world scenarios (e.g., financial reports with numerous tables (Chen et al., 2021) and product knowledge graphs (Wu et al., 2024)), thus serving as a significant knowledge base for existing LLM systems (Liang et al., 2024; Wang et al., 2025). On the other hand, due to t… view at source ↗
Figure 2
Figure 2. The construct pipeline to generate SKA-Bench instance, which consists of structured knowledge augmented question & answer (SKA-QA), positive knowledge units and noisy structured units. benchmark is valuable for thoroughly evaluating LLMs’ structured knowledge understanding ability. 3 SKA-Bench 3.1 Problem Definition To comprehensively evaluate the ability of LLMs in structured knowledge understanding, SKA￾Bench inco… view at source ↗
Figure 3
Figure 3. Four Instances from different subsets of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: The distribution of the number of positive [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: Correlation coefficients of overall F1 and stan [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 8
Figure 8. Figure 8: Correlation coefficients of overall F1 and stan [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Information Integration results on four subsets demonstrates the variation in F1 score as the number of [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: The annotation guidelines for annotators. [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: The prompt for checking Positive Units. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 15
Figure 15. Figure 15: The prompt for KG subset in QA task. ### Table: |Iteration|Year|Dates|Location|Theme| |1st|1972|6 May-20 May|Suva, Fiji|"Preserving culture"| |2nd|1976|6 March-13 March|Rotorua, New Zealand|"Sharing culture"| |3rd|1980|30 June-12 July|Port Moresby, Papua New Guinea|"P…
Figure 12
Figure 12. Figure 12: The prompt for Noisy Units synthesis. ### Question: which date the colorado state team scored no points? ### Answer: September 20, 1997 ### Positive Units: |Date|Site|Winning team|Winning team score|Losing team|Losing team score|Series| |September 20, 1997|Fort Collin…
Figure 13
Figure 13. Figure 13: The prompt for “contidition” summarizing. [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 18
Figure 18. Figure 18: The prompt for Table+Text subset in QA task. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_18.png]
Figure 19
Figure 19. Figure 19: The prompt for KG subset in “negative rejec￾tion” testbed. ### Table: |Iteration|Year|Dates|Location|Theme| |1st|1972|6 May-20 May|Suva, Fiji|"Preserving culture"| |2nd|1976|6 March-13 March|Rotorua, New Zealand|"Sharing culture"| |3rd|1980|30 June-12 July|Port Moresb…
Figure 20
Figure 20. Figure 20: The prompt for Table subset in “negative rejection” testbed. ### Triples: (PARP1, expression_present, cerebellar cortex) (VCP, ppi, HSPA5) (Elevated hepatic transaminase, associated_with, SOCS1) (PSMC5, expression_present, nasal cavity mucosa) … … ### Texts: - name: t…
Figure 21
Figure 21. Figure 21: The prompt for KG+Text subset in “negative rejection” testbed. ### Table: |Name|Years|Apps|Goals|Position| |Billy Bassett|1886-99|311|77|Outside right| |Jesse Pennington|1903-22|496|0|Left back| |W. G. Richardson|1929-45|354|228|Centre forward| |Ray Barlow|1944-60|482…
Figure 22
Figure 22. Figure 22: The prompt for KG+Text subset in “negative rejection” testbed. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_22.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 7 canonical work pages

  1. [1]

    Bollacker, Colin Evans, Praveen K

    Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. 2008. https://doi.org/10.1145/1376616.1376746 Freebase: a collaboratively created graph database for structuring human knowledge . In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10-12, 2008 , pages 1...

    arXiv 2008

  2. [2]

    Yu, Qiang Yang, and Xing Xie

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. https://doi.org/10.1145/3641289 A survey on evaluation of large language models . ACM Trans. Intell. Syst. Technol. , 15(3):39:1--39:45

    doi:10.1145/3641289 2024

  3. [3]

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024. https://doi.org/10.1609/AAAI.V38I16.29728 Benchmarking large language models in retrieval-augmented generation . In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Ed...

    doi:10.1609/aaai.v38i16.29728 2024

  4. [4]

    Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020 a . https://openreview.net/forum?id=rkeJRhNYDH Tabfact: A large-scale dataset for table-based fact verification . In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net

    work page 2020

  5. [5]

    Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020 b . https://doi.org/10.18653/V1/2020.FINDINGS-EMNLP.91 Hybridqa: A dataset of multi-hop question answering over tabular and textual data . In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020 , volume EMNLP 2020 o...

    doi:10.18653/v1/2020.findings-emnlp.91 2020

  6. [6]

    Routledge, and William Yang Wang

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting - Hao Kenneth Huang, Bryan R. Routledge, and William Yang Wang. 2021. https://doi.org/10.18653/V1/2021.EMNLP-MAIN.300 Finqa: A dataset of numerical reasoning over financial data . In Proceedings of the 2021 Conference on Empirical Methods in Na...

    doi:10.18653/v1/2021.emnlp-main.300 2021

  7. [7]

    DeepSeek - AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai...

    arXiv doi:10.48550/arxiv.2501.12948 2025

  8. [8]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek - AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Hu...

    arXiv doi:10.48550/arxiv.2412.19437 2024

Show all 37 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 doi arXiv

  2. [10]

    Ritam Dutt, Sopan Khosla, Vinayshekhar Bannihatti Kumar, and Rashmi Gangadharaiah. 2023. https://doi.org/10.18653/V1/2023.IJCNLP-MAIN.58 Grailqa++: A challenging zero-shot benchmark for knowledge base question answering . In Proceedings of the 13th International Joint Conferen...

    2023 doi

  3. [11]

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat - Seng Chua, and Qing Li. 2024. https://doi.org/10.1145/3637528.3671470 A survey on RAG meeting llms: Towards retrieval-augmented large language models . In Proceedings of the 30th ACM SIGKDD Confere...

    2024

  4. [12]

    Sengamedu, and Christos Faloutsos

    Xi Fang, Weijie Xu, Fiona Anting Tan, Ziqing Hu, Jiani Zhang, Yanjun Qi, Srinivasan H. Sengamedu, and Christos Faloutsos. 2024. https://openreview.net/forum?id=IZnrCGF9WI Large language models (llms) on tabular data: Prediction, generation, and understanding - A survey . Trans...

    2024

  5. [13]

    Xinyan Guan, Yanjiang Liu, Hongyu Lin, Yaojie Lu, Ben He, Xianpei Han, and Le Sun. 2024. https://doi.org/10.1609/AAAI.V38I16.29770 Mitigating large language model hallucinations via autonomous knowledge graph-based retrofitting . In Thirty-Eighth AAAI Conference on Artificial ...

    2024 doi

  6. [14]

    Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Supryadi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, and Deyi Xiong. 2023. https://doi.org/10.48550/ARXIV.2310.19736 Evaluating large language models: A comprehensive survey . CoRR, abs/2310.19736

    2023 doi arXiv

  7. [15]

    Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi

    Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V. Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. http://papers.nips.cc/paper\_files/paper/2024/hash/efaf1c9726648c8ba363a5c927440529-Abstract-Conference.html G-retriever: Retrieval-augmented generation for t...

    2024

  8. [16]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. https://doi.org/10.48550/ARXIV.2311.05232 A survey on hallucination in large language models: Principles, taxonomy, challen...

    2023 doi arXiv

  9. [17]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L \' e lio Renard Lavaud, Marie - Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, ...

    2023 doi arXiv

  10. [18]

    Jinhao Jiang, Kun Zhou, Xin Zhao, and Ji - Rong Wen. 2023 b . https://openreview.net/forum?id=Z63RvyAZ2Vh Unikgqa: Unified retrieval and reasoning for solving multi-hop question answering over knowledge graph . In The Eleventh International Conference on Learning Representatio...

    2023

  11. [19]

    Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing. 2024 a . https://openreview.net/forum?id=cPgh4gWZlz Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources . In The Twelf...

    2024

  12. [20]

    Zhuoqun Li, Xuanang Chen, Haiyang Yu, Hongyu Lin, Yaojie Lu, Qiaoyu Tang, Fei Huang, Xianpei Han, Le Sun, and Yongbin Li. 2024 b . https://doi.org/10.48550/ARXIV.2410.08815 Structrag: Boosting knowledge intensive reasoning of llms via inference-time hybrid information structur...

    2024 doi arXiv

  13. [21]

    Lei Liang, Mengshu Sun, Zhengke Gui, Zhongshu Zhu, Zhouyu Jiang, Ling Zhong, Yuan Qu, Peilong Zhao, Zhongpu Bo, Jin Yang, Huaidong Xiong, Lin Yuan, Jun Xu, Zaoyang Wang, Zhiqiang Zhang, Wen Zhang, Huajun Chen, Wenguang Chen, and Jun Zhou. 2024. https://doi.org/10.48550/ARXIV.2...

    2024 doi arXiv

  14. [22]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. https://doi.org/10.1162/TACL\_A\_00638 Lost in the middle: How language models use long contexts . Trans. Assoc. Comput. Linguistics, 12:157--173

    2024 doi

  15. [23]

    OpenAI. 2023. https://doi.org/10.48550/ARXIV.2303.08774 GPT-4 technical report . CoRR, abs/2303.08774

    2023 doi arXiv

  16. [24]

    Panupong Pasupat and Percy Liang. 2015. https://doi.org/10.3115/V1/P15-1142 Compositional semantic parsing on semi-structured tables . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natur...

    2015 doi

  17. [25]

    Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2023. https://doi.org/10.1145/3560260 QA dataset explosion: A taxonomy of NLP resources for question answering and reading comprehension . ACM Comput. Surv. , 55(10):197:1--197:45

    2023 doi

  18. [26]

    Aofeng Su, Aowen Wang, Chao Ye, Chen Zhou, Ga Zhang, Guangcheng Zhu, Haobo Wang, Haokai Xu, Hao Chen, Haoze Li, Haoxuan Lan, Jiaming Tian, Jing Yuan, Junbo Zhao, Junlin Zhou, Kaizhe Shou, Liangyu Zha, Lin Long, Liyao Li, Pengzuo Wu, Qi Zhang, Qingyi Huang, Saisai Yang, Tao Zha...

    2024 arXiv

  19. [27]

    Alon Talmor and Jonathan Berant. 2018. https://doi.org/10.18653/V1/N18-1059 The web as a knowledge-base for answering complex questions . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno...

    2018 doi

  20. [28]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  21. [29]

    Jinyu Wang, Jingjing Fu, Rui Wang, Lei Song, and Jiang Bian. 2025. https://doi.org/10.48550/ARXIV.2501.11551 PIKE-RAG: specialized knowledge and rationale augmented generation . CoRR, abs/2501.11551

    2025 doi arXiv

  22. [30]

    Ioannidis, Karthik Subbian, James Y

    Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis N. Ioannidis, Karthik Subbian, James Y. Zou, and Jure Leskovec. 2024. http://papers.nips.cc/paper\_files/paper/2024/hash/e607b1419e9ae7cd5cb5b5bb60c2ad5c-Abstract-Datasets\_and\_Benchmarks...

    2024

  23. [31]

    Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xeron Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, Tongliang Li, Zhoujun Li, and Guanglin Niu. 2025. https://doi.org/10.1609/AAAI.V39I24.34739 Tablebench: A comprehensive and complex benchmark for table ques...

    2025 doi

  24. [32]

    Wen - tau Yih, Matthew Richardson, Christopher Meek, Ming - Wei Chang, and Jina Suh. 2016. https://doi.org/10.18653/V1/P16-2033 The value of semantic parse labeling for knowledge base question answering . In Proceedings of the 54th Annual Meeting of the Association for Computa...

    2016 doi

  25. [33]

    Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng...

    2024 doi arXiv

  26. [34]

    Victor Zhong, Caiming Xiong, and Richard Socher. 2017. https://arxiv.org/abs/1709.00103 Seq2sql: Generating structured queries from natural language using reinforcement learning . CoRR, abs/1709.00103

    2017 arXiv

  27. [35]

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat - Seng Chua. 2021. https://doi.org/10.18653/V1/2021.ACL-LONG.254 TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance . In Proceedings of ...

    2021 doi

  28. [36]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  29. [37]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.