Pith. sign in

REVIEW 7 cited by

GraphRAG-Bench: Challenging Domain-Specific Reasoning for Evaluating Graph Retrieval-Augmented Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2506.02404 v3 pith:TVV65YTU submitted 2025-06-03 cs.CL cs.AI

Yilin Xiao , Junnan Dong , Chuang Zhou , Su Dong , Qian-wen Zhang , Di Yin , Xing Sun , Xiao Huang This is my paper
classification cs.CLcs.AI
keywords reasoninggraphragdomain-specificgraphgraphrag-benchmodelsretrievalbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Graph Retrieval Augmented Generation (GraphRAG) has garnered increasing recognition for its potential to enhance large language models (LLMs) by structurally organizing domain-specific corpora and facilitating complex reasoning. However, current evaluations of GraphRAG models predominantly rely on traditional question-answering datasets. Their limited scope in questions and evaluation metrics fails to comprehensively assess the reasoning capacity improvements enabled by GraphRAG models. To address this gap, we introduce GraphRAG-Bench, a large-scale, domain-specific benchmark designed to rigorously evaluate GraphRAG models. Our benchmark offers three key superiorities: \((i)\) Challenging question design. Featuring college-level, domain-specific questions that demand multi-hop reasoning, the benchmark ensures that simple content retrieval is insufficient for problem-solving. For example, some questions require mathematical reasoning or programming. \((ii)\) Diverse task coverage. The dataset includes a broad spectrum of reasoning tasks, multiple-choice, true/false, multi-select, open-ended, and fill-in-the-blank. It spans 16 disciplines in twenty core textbooks. \((iii)\) Holistic evaluation framework. GraphRAG-Bench provides comprehensive assessment across the entire GraphRAG pipeline, including graph construction, knowledge retrieval, and answer generation. Beyond final-answer correctness, it evaluates the logical coherence of the reasoning process. By applying nine contemporary GraphRAG methods to GraphRAG-Bench, we demonstrate its utility in quantifying how graph-based structuring improves model reasoning capabilities. Our analysis reveals critical insights about graph architectures, retrieval efficacy, and reasoning capabilities, offering actionable guidance for the research community.

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research

    cs.IR 2026-07 conditional novelty 6.5 of 10

    A hierarchical evidence-graph benchmark reveals that multimodal deep-research models write fluent reports while failing citation, claim, and answer grounding.

  2. Youtu-GraphRAG: Vertically Unified Agents for Graph Retrieval-Augmented Complex Reasoning

    cs.IR 2025-08 conditional novelty 6.0 of 10

    A schema-bounded agentic framework unifies graph construction and retrieval, reporting SOTA accuracy and up to 90.71% token-cost savings across six benchmarks, plus a new anonymized QA dataset.

  3. GLM-RAG: Graph Language Models for Graph-Based Retrieval-Augmented Generation

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Finetuned Graph LM retrievers transfer better than GNN retrievers on multi-hop graph RAG, while vanilla vector search wins single-hop tasks.

  4. Reason Before You Retrieve: Agentic Planning for Multi-modal RAG

    cs.AI 2026-06 reject novelty 5.0 of 10

    MM-R2 claims SOTA multimodal RAG accuracy on InfoSeek and Encyclopedic VQA via intent grounding plus a 10-topic KnowledgeMap, but its teacher trajectories leak the gold Wikipedia page and omit the image.

  5. HyCE-RAG: Hypergraph Chain-of-Evidence Retrieval-Augmented Generation for Explainable Multi-hop Question Answering

    cs.AI 2026-06 reject novelty 5.0 of 10

    Organizing retrieved evidence into confidence-scored hypergraph chains is reported to lift multi-hop QA accuracy, context relevance, and faithfulness by 11–30 points over vector and graph RAG baselines on five benchmarks.

  6. Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Graph-R1 combines hypergraph knowledge storage with multi-turn reinforcement-learned retrieval and reports higher F1 than chunk-based and one-shot graph RAG on six QA benchmarks.

  7. Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)

    cs.CL 2026-06 unverdicted novelty 4.0 of 10

    A graph-augmented RAG system with vector and graph query tools halves hallucinations and raises factual correctness scores on the MoNaCo complex QA benchmark.

Pith tools