Pith. sign in

REVIEW 4 major objections 6 minor 7 cited by

GraphRAG-Bench: Challenging Domain-Specific Reasoning for Evaluating Graph Retrieval-Augmented Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GraphRAG-Bench shows graph-structured retrieval measurably improves LLM reasoning.

desk verdict A genuinely useful new GraphRAG benchmark artifact whose headline empirical claim is not yet supported by an unvalidated LLM judge and single-run results. read the letter →

arxiv 2506.02404 v3 pith:TVV65YTU submitted 2025-06-03 cs.CL cs.AI

Yilin Xiao , Junnan Dong , Chuang Zhou , Su Dong , Qian-wen Zhang , Di Yin , Xing Sun , Xiao Huang This is my paper · ORCID
classification cs.CLcs.AI
keywords GraphRAGretrieval-augmentedgenerationbenchmarkmulti-hopreasoningLLMevaluationrationalescoringdomain-specificQAcomputerscienceeducation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GraphRAG-Bench is a new evaluation resource built to answer a specific question: does structuring a document corpus as a graph actually make large language models reason better, or is it just fancier retrieval? The paper assembles 1,018 college-level questions from 20 computer science textbooks over 16 topics, writes an expert gold rationale for every question, and runs nine graph-based retrieval-augmented generation methods plus baselines through a unified pipeline. On this testbed, every graph method raises the model's average rationale score, and most beat flat retrieval baselines, which the authors read as the first concrete quantification that graph structuring improves LLM reasoning. The benchmark also shows the improvement is uneven: graph retrieval helps true/false and open-ended questions, barely helps or hurts multiple-choice, fill-in-blank, and multi-select, and consistently hurts mathematics. A reader should care because the resource gives the field a harder, reasoning-focused test than existing QA datasets, and its per-type results identify where graph structure is and is not paying for itself.

What carries the argument

The central machinery is the benchmark itself: 1,018 college-level questions drawn from a 7-million-word corpus of 20 computer science textbooks, each paired with an expert-written gold rationale, together with an evaluation protocol that scores answer correctness (A), rationale quality (R), and correct-answer-with-correct-rationale (AR). The benchmark also records graph construction cost and time, retrieval indexing time, average retrieval time, and retrieval operators for each method. These metrics are what let the authors claim to quantify reasoning improvement rather than just final-answer accuracy, because simple content lookup is insufficient for the questions and the rationale scores specifically test whether the model reasons rather than guesses.

What would settle it

Have two expert annotators independently rate a random sample (say, 200 per method) of generated answers and rationales against the gold rationales, then compute item-level agreement with the GPT-4o-mini Accuracy, R, and AR scores; if human-rated reasoning gains are absent or the judge's scores do not track expert ratings, the central conclusion that GraphRAG enhances reasoning collapses to a judge artifact.

Watch

Extended reading notes

Core claim

The paper's central claim, stated in its own terms, is that GraphRAG substantially enhances the reasoning capabilities of LLMs and that GraphRAG-Bench is the first benchmark able to quantify this. It introduces a dataset of 1,018 expert-curated, domain-specific questions with five answer formats, drawn from 20 core textbooks and organized into a hierarchy, with gold rationales that spell out the multi-hop inferential path to the answer. Across nine contemporary GraphRAG methods sharing GPT-4o-mini as the generator and k=5 retrieval, the paper reports that all methods raise the average rationale score R above the no-retrieval model, with the best reasoning results from RAPTOR and HippoRAG, while generation accuracy gains are smaller and task-dependent. The paper also reports that every GraphRAG method degrades accuracy on mathematics questions and that graph retrieval can hurt multiple-choice, fill-in-blank, and multi-select performance when retrieval introduces noise. These results are the evidence for the paper's conclusion that graph-based knowledge structuring measurably improves reasoning over both no-retrieval and flat-retrieval baselines, with scope conditions.

Load-bearing premise

The load-bearing premise is that the GPT-4o-mini judge's scores for answers and rationales reflect genuine correctness and reasoning quality rather than surface similarity, since no human-agreement check is reported and the same model family generates the graded outputs.

Editorial extensions

If this is right

  • Future GraphRAG systems can be compared on GraphRAG-Bench's rationale metrics, so a method that retrieves the right evidence and reasons correctly can be distinguished from one that merely returns the right answer.
  • Answer-only evaluations understate the value of graph retrieval: on this benchmark the reasoning scores rise even in question types where final accuracy is unchanged, so benchmark designers should report rationale quality alongside accuracy.
  • Graph architecture is a real choice: tree-organized retrieval (RAPTOR) and PageRank-based retrieval (HippoRAG) led both accuracy and reasoning, while LLM-heavy traversal methods such as DALK and KGP gave smaller or negative gains.
  • Quantitative domains are a boundary condition: every graph method tested lowered accuracy on mathematics, so applying GraphRAG to math or programming problems will likely require formula-aware or symbolic retrieval rather than generic text graphs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the judge is GPT-4o-mini, the same model family whose outputs are graded, the size of the reported reasoning gains may be inflated by judge self-preference; a human-rated subset or a second-generation judge could confirm the effect.
  • Editorial inference: the textbook-hierarchy design makes the benchmark naturally suited to educational use, and a direct next test is whether GraphRAG improves student understanding or explanation quality, not just model scores.
  • Editorial inference: the dataset covers only computer science, so the claim that GraphRAG enhances reasoning in domain-specific corpora is untested in medicine, law, or other expert fields; building parallel benchmarks with the same gold-rationale protocol would show whether the effect generalizes.
  • Editorial inference: graph construction costs vary widely across methods with similar reasoning scores, so budget-conscious practitioners could adopt cheaper constructions without sacrificing all reasoning gains; the paper reports the cost data but does not turn it into a selection rule.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. GraphRAG-Bench is presented as a domain-specific benchmark for evaluating Graph Retrieval-Augmented Generation systems. The dataset comprises 1,018 college-level computer-science questions of five types, spanning 16 topics, grounded in a 7-million-word corpus extracted from 20 textbooks, with expert-crafted gold rationales. The paper evaluates nine GraphRAG methods plus TF-IDF and BM-25 baselines using metrics for graph construction, retrieval, generation accuracy, and two reasoning scores (R and AR), and concludes that graph-based structuring improves LLM reasoning. The main evidence for this conclusion rests on an LLM-as-judge scoring protocol that is not yet validated, and on single-run comparisons without statistical tests.

Significance. If the benchmark and its evaluation protocol were validated, this would be a useful community resource: it targets a gap in GraphRAG evaluation, provides challenging multi-hop questions with gold rationales, and attempts to assess the full pipeline from graph construction to reasoning. The authors should be credited for assembling the corpus from 20 textbooks, constructing 1,018 questions with rationales, and making the data publicly available. However, the paper's headline claims—that GraphRAG 'substantially enhances' reasoning and that this is the first quantification of such improvement—are not currently supported by the reported measurements, which rely on an unvalidated judge model and single-run comparisons.

major comments (4)
  1. [Section 4 (Generation and Rationale metrics)] The Accuracy score for open-ended and fill-in-blank questions, and the R and AR scores for all question types, are produced by an LLM judge (GPT-4o-mini) using a prompt that is not shown; no human-agreement study, judge-bias analysis, or calibration against an established metric is reported. Since GPT-4o-mini is also the generator used by every method, a systematic preference for fluent or longer outputs could create the measured gains in Table 5, where the average R increases from 55.45 to at most 60.81 and AR from 39.78 to 45.53. Please validate the judge on a human-annotated sample (report inter-annotator agreement), cross-check with a different judge model, and provide the exact evaluation prompt.
  2. [Section 4.4 and Table 5] Section 4.4 states that all GraphRAG methods significantly enhance the reasoning capabilities of LLMs, but Table 5 contradicts this: DALK (average R 58.89, AR 42.12) and KGP (average R 58.74, AR 42.22) are below the BM-25 baseline (R 59.18, AR 44.15). Furthermore, Tables 4 and 5 contain no variance, confidence interval, or significance test for any value, so the word 'significantly' is unsupported. The authors should restrict the claim to methods that are actually above baseline, and report either multiple runs with standard deviations or appropriate significance tests.
  3. [Section 3.3] Section 3.3 describes the expert-crafted gold rationales as the ground truth for the R and AR metrics, but no inter-annotator agreement, number of annotators, or disagreement-resolution procedure is reported. If the gold rationales are not reliable, the reasoning scores cannot be interpreted. Please report annotation statistics and describe how the rationales were validated for completeness and correctness.
  4. [Section 4 (Appendix reference)] The paper states in Section 4 that 'the full evaluation protocol and details can be found in the Appendix,' but the submitted version contains no appendix. The exact prompts used for Accuracy, R, and AR, the scoring rubric, and the measurement methodology for graph-construction and retrieval times are therefore not available, preventing reproduction. Please add the appendix with the full protocol before the paper can be considered complete.
minor comments (6)
  1. [Section 4.3] The heading contains a typo: 'accuray' should be 'accuracy'.
  2. [Figure 2] Figure 2 appears corrupted: some labels render as '$,QWURGXFWLRQ' instead of '$\$ Introduction', and similar garbled strings appear on multiple subplots; please regenerate the figure with correctly embedded fonts.
  3. [Tables 4 and 5 captions] The captions of Tables 4 and 5 do not explain the arrows, which appear to denote whether a method is above or below the GPT-4o-mini baseline; please state this explicitly.
  4. [Tables 2 and 3] Tables 2 and 3 report wall-clock times without specifying the hardware or the number of runs; a brief description of the experimental environment would help.
  5. [Abstract and Section 3.1] The terms '16 disciplines' (Abstract) and '16 topics' (Section 3.1) are used interchangeably; please choose one consistent term.
  6. [Table 2] In Table 2, the '-' entry for RAPTOR's Organization metric should be explained, since the text attributes it to the absence of isolated nodes in a tree structure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark construction and comparative evaluation are self-contained, and the LLM-judge concern is a validity risk rather than a derivation-from-input circularity.

full rationale

GraphRAG-Bench is a new benchmark artifact: the questions, gold answers, and gold rationales are newly authored from 20 textbooks, and the nine GraphRAG systems are evaluated on that fixed dataset. There is no fitted parameter that is later renamed as a prediction, no definition of a target quantity in terms of its own outcome, and no load-bearing self-citation chain that forces a conclusion. The claims that GraphRAG 'substantially enhances' reasoning and that the authors are 'the first to quantify this improvement' are empirical conclusions drawn from Tables 4 and 5, not quantities defined to equal their inputs. The notable concern is methodological: Section 4.3 says accuracy for open-ended and fill-in-the-blank questions is scored by 'feed[ing] both the generated output and groundtruth into an LLM via our designed prompt,' and the same GPT-4o-mini family is used both as the judge and as the backbone of every evaluated method, with no reported human-agreement or judge-bias check. This threatens the validity of the headline result, but it is not circularity in the sense required here: the judge does not compute a score that is definitionally equal to the benchmark's ground truth, and the comparisons among methods are not forced by construction. Likewise, the absence of inter-annotator agreement on the expert-crafted rationales is a quality-control limitation, not a circular step. Accordingly, the paper receives a circularity score of 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 3 invented entities

The benchmark's central measurements depend on two manually set hyperparameters (chunk size 1200 tokens, top-k 5) that are standardized across methods and affect every downstream number. Three unproven background assumptions carry the interpretation: the gold rationales are correct and complete, the LLM judge measures what it claims to measure, and nine selected methods represent the GraphRAG landscape. Three new metrics (Accuracy for open-ended/fill-in-blank, R, AR) are introduced as measurement instruments; none is validated against human ratings in the visible text.

free parameters (2)
  • chunk_size = 1200 tokens
    Manually set for all methods in Section 4, Experiment setups; it controls what enters graph construction and retrieval, so it shapes all accuracy and reasoning numbers.
  • top_k = 5
    Manually set for methods requiring top-k selection in Section 4; this caps how much retrieved context reaches the LLM and can change both accuracy and reasoning scores.
assumptions (3)
  • domain assumption The 1,018 expert-crafted questions and gold rationales are correct and complete.
    Section 3.3 states rationales 'articulate the complete logical progression necessary to solve each problem', but no inter-annotator agreement, screening criteria, or independent verification is reported.
  • domain assumption LLM-as-judge semantic scoring yields valid measures of answer correctness (Accuracy) and reasoning quality (R, AR).
    Section 4 defines the metrics via 'an LLM via our designed prompt'; no human-validation or judge-bias study appears in the visible text, and the judge is the same model family as the systems tested.
  • domain assumption The nine evaluated methods are representative of current GraphRAG approaches, so their aggregate behavior supports claims about GraphRAG generally.
    Section 4 selects nine methods; Sections 4.4-4.6 generalize from their results to statements about GraphRAG as a class.
invented entities (3)
  • Accuracy metric for OE and FB questions
    purpose: LLM-judged semantic alignment of the generated answer with the gold answer, replacing exact match.
    No correlation with human judgments or an established metric is reported; the scoring prompt is not shown in the visible text.
  • R (rationale reasoning score)
    purpose: LLM-judged semantic correspondence and reasoning consistency between generated and gold rationales.
    No validation against human-rated reasoning quality is provided; the score is defined only through the hidden prompt.
  • AR (answer-rationale) score
    purpose: Measures whether a correct answer is accompanied by correct reasoning, to detect lucky guesses.
    Derived from the two LLM-judged scores with no independent check; its meaning depends entirely on the unvalidated judge.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GraphRAG-Bench: Challenging Domain-Specific Reasoning for Evaluating Graph Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/TVV65YTU

@misc{pith2026250602404,
  author       = {Pith},
  title        = {Pith review of: GraphRAG-Bench: Challenging Domain-Specific Reasoning for Evaluating Graph Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TVV65YTU}},
  note         = {Machine review of arXiv:2506.02404}
}
abstract

Graph Retrieval Augmented Generation (GraphRAG) has garnered increasing recognition for its potential to enhance large language models (LLMs) by structurally organizing domain-specific corpora and facilitating complex reasoning. However, current evaluations of GraphRAG models predominantly rely on traditional question-answering datasets. Their limited scope in questions and evaluation metrics fails to comprehensively assess the reasoning capacity improvements enabled by GraphRAG models. To address this gap, we introduce GraphRAG-Bench, a large-scale, domain-specific benchmark designed to rigorously evaluate GraphRAG models. Our benchmark offers three key superiorities: \((i)\) Challenging question design. Featuring college-level, domain-specific questions that demand multi-hop reasoning, the benchmark ensures that simple content retrieval is insufficient for problem-solving. For example, some questions require mathematical reasoning or programming. \((ii)\) Diverse task coverage. The dataset includes a broad spectrum of reasoning tasks, multiple-choice, true/false, multi-select, open-ended, and fill-in-the-blank. It spans 16 disciplines in twenty core textbooks. \((iii)\) Holistic evaluation framework. GraphRAG-Bench provides comprehensive assessment across the entire GraphRAG pipeline, including graph construction, knowledge retrieval, and answer generation. Beyond final-answer correctness, it evaluates the logical coherence of the reasoning process. By applying nine contemporary GraphRAG methods to GraphRAG-Bench, we demonstrate its utility in quantifying how graph-based structuring improves model reasoning capabilities. Our analysis reveals critical insights about graph architectures, retrieval efficacy, and reasoning capabilities, offering actionable guidance for the research community.

Figures

Figures reproduced from arXiv: 2506.02404 by the authors.

Figure 1
Figure 1. A sketched overview of our benchmark GraphRAG-Bench, illustrating the contributions. directly returns fragmentized chunks based on similarity matching, which limits their ability to model complex relationships between concepts to answer the questions requiring multi-hop reasoning [3, 4], i.e., ‘What was the impact of [event] the 2008 Lehman Brothers bankruptcy on [person] Elon Musk’s Tesla?’ or global comprehension,… view at source ↗
Figure 2
Figure 2. Comparison of Generation Accuracy by Topic. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. A case study in the topic of computer networks. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research

    cs.IR 2026-07 conditional novelty 6.5 of 10

    A hierarchical evidence-graph benchmark reveals that multimodal deep-research models write fluent reports while failing citation, claim, and answer grounding.

  2. Youtu-GraphRAG: Vertically Unified Agents for Graph Retrieval-Augmented Complex Reasoning

    cs.IR 2025-08 conditional novelty 6.0 of 10

    A schema-bounded agentic framework unifies graph construction and retrieval, reporting SOTA accuracy and up to 90.71% token-cost savings across six benchmarks, plus a new anonymized QA dataset.

  3. GLM-RAG: Graph Language Models for Graph-Based Retrieval-Augmented Generation

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Finetuned Graph LM retrievers transfer better than GNN retrievers on multi-hop graph RAG, while vanilla vector search wins single-hop tasks.

  4. Reason Before You Retrieve: Agentic Planning for Multi-modal RAG

    cs.AI 2026-06 reject novelty 5.0 of 10

    MM-R2 claims SOTA multimodal RAG accuracy on InfoSeek and Encyclopedic VQA via intent grounding plus a 10-topic KnowledgeMap, but its teacher trajectories leak the gold Wikipedia page and omit the image.

  5. HyCE-RAG: Hypergraph Chain-of-Evidence Retrieval-Augmented Generation for Explainable Multi-hop Question Answering

    cs.AI 2026-06 reject novelty 5.0 of 10

    Organizing retrieved evidence into confidence-scored hypergraph chains is reported to lift multi-hop QA accuracy, context relevance, and faithfulness by 11–30 points over vector and graph RAG baselines on five benchmarks.

  6. Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Graph-R1 combines hypergraph knowledge storage with multi-turn reinforcement-learned retrieval and reports higher F1 than chunk-based and one-shot graph RAG on six QA benchmarks.

  7. Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)

    cs.CL 2026-06 unverdicted novelty 4.0 of 10

    A graph-augmented RAG system with vector and graph query tools halves hallucinations and raises factual correctness scores on the MoNaCo complex QA benchmark.

Reference graph

Works this paper leans on

23 extracted references · 19 canonical work pages · cited by 7 Pith papers

  1. [1]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive nlp tasks,” inAdvances in Neural Information Processing Systems(H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, eds.), vol. 33, pp. 9459–9474, Cu...

    work page 2020

  2. [2]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, M. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” 2024

    2024

  3. [3]

    A survey of graph retrieval-augmented generation for customized large language models,

    Q. Zhang, S. Chen, Y . Bei, Z. Yuan, H. Zhou, Z. Hong, J. Dong, H. Chen, Y . Chang, and X. Huang, “A survey of graph retrieval-augmented generation for customized large language models,” 2025

    work page 2025

  4. [4]

    Hierarchy-aware multi-hop question answering over knowledge graphs,

    J. Dong, Q. Zhang, X. Huang, K. Duan, Q. Tan, and Z. Jiang, “Hierarchy-aware multi-hop question answering over knowledge graphs,” inWWW, pp. 2519–2527, 2023

    work page 2023

  5. [5]

    From local to global: A graph rag approach to query-focused summarization,

    D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Truitt, D. Metropolitansky, R. O. Ness, and J. Larson, “From local to global: A graph rag approach to query-focused summarization,” 2025

    work page 2025

  6. [6]

    Graph retrieval-augmented generation: A survey,

    B. Peng, Y . Zhu, Y . Liu, X. Bo, H. Shi, C. Hong, Y . Zhang, and S. Tang, “Graph retrieval-augmented generation: A survey,” 2024

    work page 2024

  7. [7]

    In-depth analysis of graph-based rag in a unified framework,

    Y . Zhou, Y . Su, Y . Sun, S. Wang, T. Wang, R. He, Y . Zhang, S. Liang, X. Liu, Y . Ma, and Y . Fang, “In-depth analysis of graph-based rag in a unified framework,” 2025

    work page 2025

  8. [8]

    RAPTOR: Recursive ab- stractive processing for tree-organized retrieval,

    P. Sarthi, S. Abdullah, A. Tuli, S. Khanna, A. Goldie, and C. D. Manning, “RAPTOR: Recursive ab- stractive processing for tree-organized retrieval,” inThe Twelfth International Conference on Learning Representations, 2024

    work page 2024

Show all 23 references
  1. [9]

    Gfm-rag: Graph foundation model for retrieval augmented generation,

    L. Luo, Z. Zhao, G. Haffari, D. Phung, C. Gong, and S. Pan, “Gfm-rag: Graph foundation model for retrieval augmented generation,” 2025

    2025

  2. [10]

    G-retriever: Retrieval-augmented generation for textual graph understanding and question answering,

    X. He, Y . Tian, Y . Sun, N. V . Chawla, T. Laurent, Y . LeCun, X. Bresson, and B. Hooi, “G-retriever: Retrieval-augmented generation for textual graph understanding and question answering,” inAdvances in Neural Information Processing Systems(A. Globerson, L. Mackey, D. Belgra...

    2024

  3. [11]

    DALK: Dynamic co-augmentation of LLMs and KG to answer Alzheimer‘s disease questions with scientific literature,

    D. Li, S. Yang, Z. Tan, J. Y . Baik, S. Yun, J. Lee, A. Chacko, B. Hou, D. Duong-Tran, Y . Ding, H. Liu, L. Shen, and T. Chen, “DALK: Dynamic co-augmentation of LLMs and KG to answer Alzheimer‘s disease questions with scientific literature,” inFindings of the Association for C...

    2024

  4. [12]

    Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph,

    J. Sun, C. Xu, L. Tang, S. Wang, C. Lin, Y . Gong, L. Ni, H.-Y . Shum, and J. Guo, “Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  5. [13]

    HotpotQA: A dataset for diverse, explainable multi-hop question answering,

    Z. Yang, P. Qi, S. Zhang, Y . Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning, “HotpotQA: A dataset for diverse, explainable multi-hop question answering,” inProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing(E. Riloff, D. Chiang, J. H...

    2018

  6. [14]

    Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps,

    X. Ho, A.-K. Duong Nguyen, S. Sugawara, and A. Aizawa, “Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps,” inProceedings of the 28th International Conference on Computational Linguistics, (Barcelona, Spain (Online)), pp. 6609–6625, Internatio...

    2020

  7. [15]

    Musique: Multihop questions via single-hop question composition,

    H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal, “Musique: Multihop questions via single-hop question composition,”Transactions of the Association for Computational Linguistics, vol. 10, pp. 539–554, 2022

    2022

  8. [16]

    Lightrag: Simple and fast retrieval-augmented generation,

    Z. Guo, L. Xia, Y . Yu, T. Ao, and C. Huang, “Lightrag: Simple and fast retrieval-augmented generation,” 2024

    2024

  9. [17]

    Hipporag: Neurobiologically inspired long-term memory for large language models,

    B. J. Gutiérrez, Y . Shu, Y . Gu, M. Yasunaga, and Y . Su, “Hipporag: Neurobiologically inspired long-term memory for large language models,” inAdvances in Neural Information Processing Systems(A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, ...

    2024

  10. [18]

    Knowledge graph prompting for multi- document question answering,

    Y . Wang, N. Lipka, R. A. Rossi, A. Siu, R. Zhang, and T. Derr, “Knowledge graph prompting for multi- document question answering,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 19206–19214, 2024

    2024

  11. [19]

    QuALITY: Question answering with long input texts, yes!,

    R. Y . Pang, A. Parrish, N. Joshi, N. Nangia, J. Phang, A. Chen, V . Padmakumar, J. Ma, J. Thompson, H. He, and S. Bowman, “QuALITY: Question answering with long input texts, yes!,” inProceedings of the 2022 Conference of the North American Chapter of the Association for Compu...

    2022

  12. [20]

    When not to trust language models: Investigating effectiveness of parametric and non-parametric memories,

    A. Mallen, A. Asai, V . Zhong, R. Das, D. Khashabi, and H. Hajishirzi, “When not to trust language models: Investigating effectiveness of parametric and non-parametric memories,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume ...

    2023

  13. [21]

    Layoutlmv3: Pre-training for document ai with unified text and image masking,

    Y . Huang, T. Lv, L. Cui, Y . Lu, and F. Wei, “Layoutlmv3: Pre-training for document ai with unified text and image masking,” inProceedings of the 30th ACM International Conference on Multimedia, MM ’22, (New York, NY , USA), p. 4083–4091, Association for Computing Machinery, 2022

    2022

  14. [22]

    Yolov10: Real-time end-to-end object detection,

    A. Wang, H. Chen, L. Liu, K. Chen, Z. Lin, J. Han, and G. Ding, “Yolov10: Real-time end-to-end object detection,” inAdvances in Neural Information Processing Systems(A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, eds.), vol. 37, pp. 107984–1...

    2024

  15. [23]

    Mineru: An open-source solution for precise document content extraction,

    B. Wang, C. Xu, X. Zhao, L. Ouyang, F. Wu, Z. Zhao, R. Xu, K. Liu, Y . Qu, F. Shang, B. Zhang, L. Wei, Z. Sui, W. Li, B. Shi, Y . Qiao, D. Lin, and C. He, “Mineru: An open-source solution for precise document content extraction,” 2024. 12

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.