Pith. sign in

REVIEW 4 major objections 5 minor 42 references

How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GraphRAG's reported performance gains may be largely artifacts of how answers are evaluated.

desk verdict Solid bias demonstrations, premature 'unbiased' claim: the framework removes known LLM-judge biases but never validates that its scores track human-judged quality. read the letter →

arxiv 2506.06331 v1 pith:EX6T36IW submitted 2025-05-31 cs.CL cs.AIcs.IR

Qiming Zeng , Xiao Yan , Hao Luo , Yuhao Lin , Yuxiang Wang , Fangcheng Fu , Bo Du , Quanqing Xu
show 1 more author
Jiawei Jiang
This is my paper
classification cs.CLcs.AIcs.IR
keywords GraphRAGretrieval-augmentedgenerationLLM-as-judgeevaluationbiaspositionlengthquestionwinrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard way of benchmarking GraphRAG systems overstates their real gains. It identifies two flaws in the current two-part evaluation pipeline: questions generated from dataset summaries are often unrelated to the actual corpus, and LLM-based answer comparison is biased by the position of an answer in the prompt, by answer length, and by random variation across repeated trials. To fix this, the paper introduces a framework that generates questions from the knowledge graph itself (at node, edge, and subgraph levels) and evaluates answers with length alignment, position exchange, repeated trials, and allowed ties. Using this framework on three GraphRAG methods and a non-graph RAG baseline, the paper finds that reported advantages shrink or reverse, for example LightRAG's 66.70% win rate over NaiveRAG on Agriculture drops to 39.06%. The point matters because if the framework is right, many published GraphRAG performance claims need to be re-examined before they are trusted.

What carries the argument

The load-bearing machinery is the bias-correcting evaluation pipeline. Its question side samples a node, an edge, or a random-walk subgraph from the knowledge graph, and feeds the structure plus its source text segments to the LLM to generate questions that are tied to concrete corpus details. Its assessment side aligns the two answers to similar lengths by regenerating the shorter answer at the longer answer's target length, evaluates both answer orders and averages the scores, repeats the whole evaluation across trials, and reports win-rate distributions with ties allowed.

What would settle it

Run the same three GraphRAG methods and NaiveRAG on the Agriculture dataset's 150 questions with human expert judges, then compare the human rank order and win rates with the corrected framework's box plots; a substantial disagreement, such as humans restoring LightRAG's large margin over NaiveRAG, would falsify the claim that the corrected procedure removes the bias.

Watch

Extended reading notes

Core claim

The central claim is that the answer-quality evaluation used in current GraphRAG research is not measuring what it appears to measure. The paper demonstrates three judge biases—position bias, length bias, and trial bias—and shows that they can change a same-method comparison from a tie to a 90%-vs-10% win. Its proposed correction replaces summary-based question generation with graph-text-grounded question generation and replaces single LLM judgments with length-aligned, position-exchanged, repeated scoring that permits ties. Applied to MGRAG, LightRAG, FGRAG, and NaiveRAG on three datasets, the correction yields much more moderate win rates, with FGRAG the best but by margins mostly under 10% in relative win rate, and with NaiveRAG beating LightRAG in some comparisons.

Load-bearing premise

The claim that the framework is unbiased depends on the assumption that the LLM judge's averaged scores after length alignment, position exchange, and repeated trials reflect genuine answer quality; the paper only checks this by showing that LightRAG ties itself, not by calibrating against human judgments.

Editorial extensions

If this is right

  • Reported win rates for existing GraphRAG methods should be treated with caution; under the corrected framework many margins shrink or reverse.
  • LightRAG's claimed advantage over NaiveRAG on Agriculture is not reproduced; NaiveRAG can come out ahead once position and length biases are removed.
  • FGRAG remains the strongest of the tested methods, but its advantage is moderate, suggesting that graph-based retrieval plus re-ranking helps without dominating.
  • Allowing ties changes the measurement itself: tie rates exceed 20% in most overall comparisons and often exceed 50% on individual quality aspects.
  • The gap between GraphRAG and NaiveRAG grows with dataset size, indicating that knowledge-graph retrieval matters more for larger corpora.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely extension is that the three judge biases generalize beyond GraphRAG, so LLM-as-judge comparisons in other RAG and generative systems may report inflated winners until they control for position, length, and trial variance.
  • The 15% of answer pairs that cannot be length-aligned are dropped; an untested check is whether those excluded pairs favor one method, which would change the corrected win rates if re-included.
  • The graph-text-grounded question generator is a reusable tool: it could produce grounded queries for evaluating any retrieval system over a domain corpus, not only GraphRAG variants.
  • A natural next validation would compare the framework's scores against human expert ratings on the same question-answer pairs, since the current sanity check only shows the judge agrees with itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that current GraphRAG evaluation is flawed in two ways: (1) summary-based question generation produces questions that are not closely related to the underlying dataset, and (2) LLM-based answer assessment suffers from position, length, and trial biases. The authors propose a new framework with graph-text-grounded question generation (node, edge, and subgraph levels) and an 'unbiased evaluation procedure' that performs length alignment, position exchange, and multiple evaluation trials. They apply this framework to compare MGRAG, LightRAG, and FGRAG against NaiveRAG on three UDB sub-corpora, reporting that performance gains are much more moderate than previously claimed, with high tie rates and relative win rates usually below 8% except for FGRAG and MGRAG versus LightRAG.

Significance. If the bias demonstrations and the corrected win rates hold, this is a timely and important methodological contribution to GraphRAG research. The controlled self-comparisons (Figures 2, 6, and 9) provide strong evidence that position, length, and trial biases exist in current LLM-as-judge protocols, and the graph-grounded question generation is a clear improvement in question relevance. The paper also reports execution costs and per-aspect analyses, which are useful. However, the central quantitative claim that the newly computed win rates are the 'real' performance gains rests on an untested validity assumption about the LLM judge's scores, and the comparison with prior reported numbers changes several protocol components simultaneously, so the attribution of the reduction to bias removal is not yet established.

major comments (4)
  1. [Section 5.2, Figures 10 and 11] The headline comparison—e.g., LightRAG's 66.70% vs. NaiveRAG reported by LightRAG becoming 39.06% under the new framework—changes not only bias handling but also the question generation method, the judge model, the evaluation aspects (Diversity replaced by Relevance), the scoring mechanism (winner selection replaced by scores), and the tie policy. These confounds mean the reduction cannot be attributed to bias removal. The paper needs an ablation or decomposition that isolates the effect of each protocol change, or a re-evaluation of the same original questions under only the bias-correction steps, to support the claim that the gains shrink because of bias elimination.
  2. [Section 4.2, length alignment] The paper states that length alignment succeeds for only 85% of answer pairs and discards the remaining 15%, but it provides no analysis of whether the discarded pairs correlate with method identity, question type, or answer characteristics such as extreme verbosity or structural failure. If the unalignable pairs are systematically different from the aligned pairs, the aggregate win rates are themselves biased in a way that position exchange cannot fix. Please report per-method-pair and per-dataset discard rates and run a robustness check that, for example, treats discarded pairs as ties or as wins for the longer answer, to show the conclusions are stable.
  3. [Section 4.2 and Figure 9] The framework is called 'unbiased' but its validity is checked only by a self-comparison (LightRAG vs. LightRAG must tie). This demonstrates internal consistency, not that the scores track answer quality. A judge with a stable preference for verbose, fluent, or sycophantic text would pass this sanity check while still producing misleading win rates. The paper provides no calibration against human judgments or against a benchmark with known answer-quality differences. At minimum, a small human-annotated sample of pairwise judgments should be compared with the judge's scores to establish external validity.
  4. [Section 3 and Section 5.1] There is a partial circularity in the evaluation pipeline: GPT-4o-mini is used to build the knowledge graph, generate questions from it, generate answers (as part of the RAG systems), and judge the answers. If the judge internalizes its own graph-extraction and question-generation style, it may systematically favor answers that match that style, inflating the win rates of methods whose outputs resemble its own generation patterns. The bias-existence demonstrations are independent of this issue, but the comparative win rates are not. Please test with a different judge model or with human judgments on a subset to assess this risk.
minor comments (5)
  1. [Throughout] Several figures and text passages contain garbled placeholder sequences such as '/uni00000024/uni00000003/...' (e.g., Figures 2, 6, 7, and the surrounding text). These need to be repaired so that the figures and captions are legible.
  2. [Section 4.2] The text says 'Our length aliment procedure can succeed for 85% of the answer pairs'; 'aliment' should be 'alignment'.
  3. [Section 4.1 and Figure 9 caption] The term 'trail bias' is used; this should be 'trial bias'.
  4. [Appendix A] The phrase 'without retraininging' should be 'without retraining'.
  5. [Table 3] The case study header says 'Random Select Judge' but the procedure shown is a scoring-based judge; please clarify what 'Random Select' means or rephrase the heading.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the bias demonstrations use valid self-comparison controls and the moderate win rates are new empirical measurements.

full rationale

The paper's derivation chain is an evaluation study, not a mathematical derivation from fitted inputs. The three bias claims (position, length, trial) are demonstrated with self-comparison controls: LightRAG compared with itself should tie, but the existing protocol gives lopsided win rates (Figure 2/Figure 9), and repeated trials of the same comparison yield contradictory conclusions (Figure 7). These are legitimate control experiments, not circular reasoning. The proposed fixes—position exchange, length alignment, and trial statistics—are described as procedural interventions, and the headline result (e.g., LightRAG's win rate against NaiveRAG dropping from 66.70% to 39.06% on Agriculture) is a new measurement under that protocol, not a quantity fitted from or defined by the prior win rates. The paper contains no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in via citation; all cited method papers are external works. The self-match sanity check in Figure 9 is partly entailed by the position-averaging construction, so it provides only internal consistency rather than external validation, and the paper does not calibrate its LLM judge against human judgments, nor does it analyze the 15% of answer pairs discarded by length alignment. Those are validity and representativeness limitations, which belong under correctness risk, not circularity. No step in the paper reduces its central conclusion to its own inputs by construction.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The framework's conclusions rest on several hand-set tuning choices and assumptions. The thresholds (10 words, 50 hops, N=2, M=25) are defaults with no sensitivity analysis. The success rate of length alignment is 85%, and the discarded 15% may be non-random. The framework assumes that an LLM-built knowledge graph yields faithful questions, and that using the same LLM for generation and judging does not introduce hidden self-preference. No external human validation is provided for the 'unbiased' claim.

free parameters (6)
  • length alignment tolerance = 10 words
    Default tolerance for length difference after alignment; larger or smaller values change which answer pairs are discarded, affecting win rates.
  • minimum subgraph size = 50 non-repeating hops
    Subgraph questions are only generated for random walks with at least 50 hops; this affects the mix of question types and the difficulty distribution.
  • prompt repetitions per position = N = 2
    Each AB and BA prompt is issued N times and averaged; higher N reduces variance at higher API cost.
  • evaluation trials = M = 25
    The whole question set is evaluated M times; the reported box plots depend on this choice and the resulting variance estimates.
  • questions per level = 50 per level, 150 total
    Number of node, edge, and subgraph questions generated for each dataset; serves as the evaluation sample size.
  • length alignment success rate = 85%
    Only successfully aligned answer pairs are kept; the remaining 15% are discarded, which can bias comparisons if dropped pairs differ systematically from kept ones.
assumptions (4)
  • domain assumption LLM-as-judge scores reflect human judgments of answer quality
    The entire evaluation relies on GPT-4o-mini scoring answers and selecting winners; no human validation is provided.
  • domain assumption The LLM-extracted knowledge graph faithfully represents the source text
    Graph-text-grounded question generation uses an LLM-built KG; if the KG is faulty, questions may be unfaithful to the data.
  • domain assumption Position exchange, length alignment, and trial repetition fully remove the identified biases
    Only a self-comparison sanity check supports this; no external validation against human ratings or known benchmarks.
  • domain assumption Discarded length-unalignable answer pairs do not distort win rates
    15% of pairs are dropped after failed alignment; the paper does not analyze the characteristics of dropped pairs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG." pith.science (2026). https://pith.science/paper/EX6T36IW

@misc{pith2026250606331,
  author       = {Pith},
  title        = {Pith review of: How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EX6T36IW}},
  note         = {Machine review of arXiv:2506.06331}
}
read the original abstract

By retrieving contexts from knowledge graphs, graph-based retrieval-augmented generation (GraphRAG) enhances large language models (LLMs) to generate quality answers for user questions. Many GraphRAG methods have been proposed and reported inspiring performance in answer quality. However, we observe that the current answer evaluation framework for GraphRAG has two critical flaws, i.e., unrelated questions and evaluation biases, which may lead to biased or even wrong conclusions on performance. To tackle the two flaws, we propose an unbiased evaluation framework that uses graph-text-grounded question generation to produce questions that are more related to the underlying dataset and an unbiased evaluation procedure to eliminate the biases in LLM-based answer assessment. We apply our unbiased framework to evaluate 3 representative GraphRAG methods and find that their performance gains are much more moderate than reported previously. Although our evaluation framework may still have flaws, it calls for scientific evaluations to lay solid foundations for GraphRAG research.

Figures

Figures reproduced from arXiv: 2506.06331 by the authors.

Figure 1
Figure 1. Questions produced for Harry Potter novel by the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The position bias for LLM-based answer assessment. [PITH_FULL_IMAGE:figures/full_fig_p001_2.png] view at source ↗
Figure 3
Figure 3. The evaluation results using the existing method [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Workflow of graph-text-grounded question genera [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: Examples of the questions generated by our graph [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 7
Figure 7. Figure 7: The effect of trial bias. We compare NaiveRAG and [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: An overview of our unbiased evaluation framework. [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Sanity check. We compare the answers generated [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 10
Figure 10. Figure 10: One-to-one comparison between the 3 GraphRAG methods and NaiveRAG, each row corresponds to a dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 11
Figure 11. Figure 11: Relative win rates (defined as row minus column) [PITH_FULL_IMAGE:figures/full_fig_p007_11.png]
Figure 12
Figure 12. Figure 12: Detailed win rate comparison for the 4 aspects of answer quality, the dataset is Music. [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 13
Figure 13. Figure 13: The performance comparison heatmap shows the [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 14
Figure 14. Figure 14: The performance comparison heatmap shows [PITH_FULL_IMAGE:figures/full_fig_p011_14.png]
Figure 15
Figure 15. Figure 15: The performance comparison heatmap shows the [PITH_FULL_IMAGE:figures/full_fig_p011_15.png]
Figure 16
Figure 16. Figure 16: Detailed win rate comparison across the four introduced aspects for model pairs under Agriculture dataset. [PITH_FULL_IMAGE:figures/full_fig_p012_16.png]
Figure 17
Figure 17. Figure 17: Detailed win rate comparison across the four introduced aspects for model pairs under Mix dataset. [PITH_FULL_IMAGE:figures/full_fig_p012_17.png]
Figure 18
Figure 18. Figure 18: The average win rate comparison across the four [PITH_FULL_IMAGE:figures/full_fig_p014_18.png]
Figure 19
Figure 19. Figure 19: The average win rate comparison across the four [PITH_FULL_IMAGE:figures/full_fig_p014_19.png]
Figure 20
Figure 20. Figure 20: The average win rate comparison across the four [PITH_FULL_IMAGE:figures/full_fig_p014_20.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 12 canonical work pages

  1. [1]

    Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, et al. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:2212.08037 (2022)

    arXiv 2022

  2. [2]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al . 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15, 3 (2024), 1–45

    2024

  3. [3]

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 17754–17762

    work page 2024

  4. [4]

    Xiaojun Chen, Shengbin Jia, and Yang Xiang. 2020. A review: Knowledge reason- ing over knowledge graph. Expert systems with applications 141 (2020), 112948

    work page 2020

  5. [5]

    Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937 (2023)

    arXiv 2023

  6. [6]

    CircleMind. 2024. Streamlined and promptable fast graphrag framework designed for interpretable, high-precision, agent-driven retrieval workflows. https:// github.com/circlemind-ai/fast-graphrag Accessed: 2025-02-07

    work page 2024

  7. [7]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130 (2024)

    arXiv 2024

  8. [8]

    Dieter Fensel, Umutcan Şimşek, Kevin Angele, Elwin Huaman, Elias Kärle, Olek- sandra Panasiuk, Ioan Toma, Jürgen Umbrich, Alexander Wahler, Dieter Fensel, et al. 2020. Introduction: what is a knowledge graph? Knowledge graphs: Method- ology, tools and selected use cases (2020), 1–10

    work page 2020

Show all 42 references
  1. [9]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 (2023)

    2023 arXiv

  2. [10]

    Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. 2024. LightRAG: Simple and Fast Retrieval-Augmented Generation. (2024). arXiv:2410.05779 [cs.IR]

    2024 arXiv

  3. [12]

    Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al

  4. [13]

    Haoyu Han, Yu Wang, Harry Shomer, Kai Guo, Jiayuan Ding, Yongjia Lei, Ma- hantesh Halappanavar, Ryan A Rossi, Subhabrata Mukherjee, Xianfeng Tang, et al. 2024. Retrieval-augmented generation with graphs (graphrag). arXiv preprint arXiv:2501.00309 (2024)

    2024 arXiv

  5. [14]

    Authorea Preprints (2023)

    A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints (2023)

    2023

  6. [15]

    Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. arXiv preprint arXiv:2402.07630 (2024)

    2024 arXiv

  7. [16]

    Rujun Han, Yuhao Zhang, Peng Qi, Yumo Xu, Jenyuan Wang, Lan Liu, William Yang Wang, Bonan Min, and Vittorio Castelli. 2024. Rag-qa arena: Evaluating domain robustness for long-form retrieval augmented question an- swering. arXiv preprint arXiv:2407.13998 (2024)

    2024 arXiv

  8. [17]

    Yucheng Hu and Yuxing Lu. 2024. Rag and rau: A survey on retrieval-augmented language model in natural language processing. arXiv preprint arXiv:2404.19543 (2024)

    2024 arXiv

  9. [18]

    Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo Pan, Chen Ling, and Liang Zhao. 2024. GRAG: Graph Retrieval-Augmented Generation. arXiv preprint arXiv:2405.16506 (2024)

    2024 arXiv

  10. [19]

    Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruirui Li, Xianfeng Tang, Suhang Wang, Yu Meng, et al. 2024. Graph chain-of- thought: Augmenting large language models by reasoning on graphs. arXiv preprint arXiv:2404.07103 (2024)

    2024 arXiv

  11. [20]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Hao- tian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al

  12. [21]

    arXiv preprint arXiv:2311.05232 (2023)

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232 (2023)

    2023 arXiv

  13. [22]

    Rong-Hua Li, Jeffrey Xu Yu, Lu Qin, Rui Mao, and Tan Jin. 2015. On random walk based graph sampling. In 2015 IEEE 31st international conference on data engineering. IEEE, 927–938

    2015

  14. [23]

    Tom Kocmi and Christian Federmann. 2023. Large language models are state-of- the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520 (2023)

    2023 arXiv

  15. [24]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing...

    2020

  16. [25]

    Tyler Thomas Procko and Omar Ochoa. 2024. Graph retrieval-augmented gen- eration for large language models: A survey. In 2024 Conference on AI, Science, Engineering, and Technology (AIxSET). IEEE, 166–169

    2024

  17. [26]

    Yen-Ting Lin and Yun-Nung Chen. 2023. Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. arXiv preprint arXiv:2305.13711 (2023)

    2023 arXiv

  18. [27]

    Yuanjie Lyu, Zhiyu Li, Simin Niu, Feiyu Xiong, Bo Tang, Wenjin Wang, Hao Wu, Huanyong Liu, Tong Xu, and Enhong Chen. 2024. Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models. ACM Transactions on Information Systems (2024)

    2024

  19. [28]

    SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. 2024. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313 (2024)

    2024 arXiv

  20. [29]

    Hongjin Qian, Peitian Zhang, Zheng Liu, Kelong Mao, and Zhicheng Dou. 2024. MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery. (2024). arXiv:2409.05591 https://arxiv.org/abs/2409.05591

    2024 arXiv

  21. [30]

    Alireza Salemi and Hamed Zamani. 2024. Evaluating retrieval quality in retrieval- augmented generation. In Proceedings of the 47th International ACM SIGIR Con- ference on Research and Development in Information Retrieval . 2395–2400

    2024

  22. [31]

    Yang Wang, Alberto Garcia Hernandez, Roman Kyslyi, and Nicholas Kersting

  23. [32]

    Yu-Min Tseng, Wei-Lin Chen, Chung-Chi Chen, and Hsin-Hsi Chen. 2024. Are Expert-Level Language Models Expert-Level Annotators? arXiv preprint arXiv:2410.03254 (2024)

    2024 arXiv

  24. [33]

    Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. 2023. Survey on factuality in large language models: Knowledge, retrieval and domain- specificity. arXiv preprint arXiv:2310.07521 (2023)

    2023 arXiv

  25. [34]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis

  26. [35]

    Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, and Zhaofeng Liu. 2024. Evaluation of retrieval-augmented generation: A survey. In CCF Conference on Big Data. Springer, 102–120

    2024

  27. [36]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reason- ing in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  28. [37]

    David F Williamson, Robert A Parker, and Juliette S Kendrick. 1989. The box plot: a simple visual method to interpret data. Annals of internal medicine 110, 11 (1989), 916–921

    1989

  29. [38]

    illusion

    Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. 2023. Large lan- guage models for information retrieval: A survey.arXiv preprint arXiv:2308.07107 (2023). An Unbiased Evaluation Framework for...

    2023

  30. [39]

    arXiv preprint arXiv:2309.17453 (2023)

    Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 (2023)

    2023 arXiv

  31. [41]

    Haozhen Zhang, Tao Feng, and Jiaxuan You. 2024. Graph of records: Boosting retrieval augmented generation for long-context summarization with graphs. arXiv preprint arXiv:2410.11001 (2024)

    2024 arXiv

  32. [42]

    Yujia Zhou, Yan Liu, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Zheng Liu, Chaozhuo Li, Zhicheng Dou, Tsung-Yi Ho, and Philip S Yu. 2024. Trustworthiness in retrieval- augmented generation systems: A survey.arXiv preprint arXiv:2409.10102 (2024)

    2024 arXiv

  33. [2023]

    Authorea Preprints (2023)

    Large language models: a comprehensive survey of its applications, chal- lenges, limitations, and future prospects. Authorea Preprints (2023)

    2023

  34. [2024]

    arXiv preprint arXiv:2406.18064 (2024)

    Evaluating quality of answers for Retrieval-Augmented Generation: A strong LLM is all you need. arXiv preprint arXiv:2406.18064 (2024)

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.