Pith. sign in

REVIEW 2 major objections 5 minor 15 references

RAG metric correlations with human scores may be confounded, not meaningful

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 14:37 UTC pith:LD6MDACF

load-bearing objection Honest empirical study of RAG metric correlations, but single-system design limits all findings the 2 major comments →

arxiv 2607.07302 v1 pith:LD6MDACF submitted 2026-07-08 cs.CL

Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitations

Quentin Brabant This is my paper
classification cs.CL
keywords metricsevaluatingexperimentlimitationsanalysisannotatorsappliedavenues
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper evaluates whether popular automated metrics for Retrieval-Augmented Generation (RAG) systems actually measure what they claim to measure, by comparing their scores against human evaluations on a 96-question business-domain QA dataset in French. The author runs a single RAG system, collects its retrieved passages and generated answers, scores them with metrics from four libraries, and computes Pearson correlations against two human evaluators (who agree with each other at r=0.85). The key finding is that correlations vary widely: some metrics from RAGChecker correlate strongly with human scores (claim recall near 0.7), while generation-only metrics like faithfulness correlate weakly, and some classical metrics like METEOR correlate surprisingly well. But the paper's central contribution is not the correlation numbers themselves — it is the explicit warning that these correlations can be deeply misleading. Because the study involves only one RAG system, a metric can correlate with human scores for reasons unrelated to what it purports to measure: for instance, a metric that implicitly captures question difficulty would correlate positively with human scores (easy questions get better answers) while being completely useless for comparing different RAG systems. The paper argues that single-system correlation studies cannot distinguish a metric that genuinely measures response quality from one that merely tracks input-question properties, and recommends that future metric-evaluation studies use multiple RAG systems so that metric behavior can be observed independently of question characteristics.

Core claim

The paper discovers, through both empirical results and a thought experiment, that strong correlation between an automated RAG metric and human evaluation scores on a single system is insufficient evidence that the metric measures the intended criterion. The RAGChecker claim recall metric exemplifies this: it correlates near 0.7 with human response-quality scores, which is implausibly high for a retrieval metric and likely reflects confounding factors such as question difficulty or ease of information extraction rather than genuine retrieval-quality measurement. The methodological lesson is that evaluating metrics on outputs from a single RAG system conflates metric quality with question-in难

What carries the argument

The central mechanism is a correlation study: human evaluators score 96 RAG system outputs on a 1–5 rubric combining factuality and relevance; automated metrics from four libraries (Ragas, DeepEval, RAGChecker, Opik) score the same outputs; Pearson correlation between metric scores and average human scores is computed. A word-level recall metric (measuring what fraction of reference-span words appear in retrieved passages) serves as the reference for retrieval metrics. The thought experiment about a question-difficulty metric — which would correlate with human scores while being useless for system comparison — is the key analytical device for exposing the confound.

Load-bearing premise

The study assumes that correlation between a metric's scores and human scores on a single RAG system's outputs indicates the metric is measuring the intended evaluation criterion. The author acknowledges this is insufficient: a metric could correlate with human scores by capturing question difficulty rather than response quality, and a single-system setup cannot distinguish the two.

What would settle it

If the same metrics were evaluated across multiple RAG systems on the same questions and the per-input correlations across systems were near zero — even though single-system correlations were high — this would confirm that the single-system correlations were driven by question-level confounds rather than genuine metric quality.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Single-system correlation studies can eliminate clearly bad metrics (those with near-zero correlation) but cannot validate good ones, because confounding from question properties inflates or distorts correlations.
  • Metrics that do not use reference answers (reference-free metrics) can still correlate with human scores, but this correlation may stem from implicit question-difficulty signals rather than from measuring response quality.
  • Word-level recall outperforms document-level recall as a retrieval reference metric when reference spans are short relative to source documents, because document-level matching can count a reference as 'found' even when the retrieved passage is disjoint from it.
  • Future metric-evaluation protocols should use multiple RAG systems and compute per-input correlations across systems, which partially controls for question-level confounds and increases discriminative power.
  • The cost of reliable metric validation grows with the number of systems and questions needed, creating a practical tension between validation rigor and annotation budget.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If a metric's correlation with human scores is driven by question difficulty rather than response quality, then applying that metric to compare two RAG systems on the same question set would give both systems similar scores on easy questions regardless of actual quality differences — the metric would be systematically biased toward whichever system processes easier questions.
  • A metric that correlates more strongly with human response-quality scores than with recall (as claim recall does) is likely measuring something beyond retrieval — possibly generation quality, question characteristics, or the alignment between retrieved content and answer — which makes it a poor retrieval metric even if it appears to be a good overall metric.
  • The finding that METEOR correlates well with human scores on this dataset suggests that classical n-gram metrics may remain competitive in specific domains (e.g., business QA with factual answers), challenging the assumption that LLM-based metrics are uniformly superior.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper reports an empirical study evaluating the correlation of several RAG evaluation metrics (from Ragas, DeepEval, RAGChecker, and Opik) with human judgments and word-level recall on a 96-question business-domain QA dataset in French. Two human evaluators scored system outputs on a 1-5 rubric combining factuality and relevance (inter-annotator Pearson r=0.85). The study finds varying correlations: RAGChecker metrics show the strongest correlations with human scores (claim recall near 0.7), generation-only metrics correlate weakly, and some traditional metrics like METEOR correlate surprisingly well. The paper explicitly acknowledges a key limitation: because only one RAG system is evaluated, correlations may reflect question-level confounds rather than genuine measurement of the intended criteria. The paper compares its methodology to related work and suggests multi-system designs for future research.

Significance. The paper addresses a practically important question for applied RAG development: whether off-the-shelf LLM-as-a-judge metrics reliably approximate human evaluation criteria on a given dataset. The contribution is primarily methodological and cautionary rather than a definitive benchmark. Strengths include transparent reporting of inter-annotator agreement, open sharing of raw scores and analysis code (GitHub link in Section 1), and commendable honesty about the single-system confound (Section 5). The dataset cannot be made public due to business constraints, which limits reproducibility of the experimental setup, but the shared code and scores partially mitigate this. The paper is an English translation of a workshop paper (EvalLLM 2026), and its scope and depth are consistent with that origin.

major comments (2)
  1. Section 4 reports metric-specific findings (e.g., 'METEOR correlates surprisingly well,' 'RAGChecker metrics show very strong correlation,' 'claim recall correlation appears close to 0.7') as if they are informative observations, but Section 5 explicitly acknowledges that the single-system design cannot distinguish 'metric measures the criterion' from 'metric captures a confound.' The thought experiment about a question-difficulty metric demonstrates that high correlation is compatible with zero measurement validity. This is not a peripheral limitation but the core interpretive framework. The paper should either (a) reframe the Results section to consistently flag that all metric-specific findings are confounded and cannot support comparative claims about metric quality, or (b) explain more precisely what can and cannot be inferred from single-system correlations. As written, there is a
  2. Section 2.3, paragraph on retrieval metrics: The choice of word-level recall as the reference retrieval metric is justified partly on the grounds that it 'correlates much better with evaluators' average scores (r=0.35) than other cited metrics, and notably than document-level recall (r=0.05).' However, those same human scores serve as the reference for evaluating all other metrics in the study. This introduces a partial circularity in the baseline: the reference retrieval metric was selected because it aligns with the human scores against which all retrieval metrics are then assessed. The paper should acknowledge this circularity explicitly and, if possible, provide an independent justification for word-level recall (e.g., theoretical relevance to the downstream generation task) that does not depend on correlation with the same human scores used throughout the evaluation.
minor comments (5)
  1. Section 2, first paragraph: The sentence 'The developed system processes each given question via two key modules: first, a retriever... then a generator...' is followed by a separate sentence 'The retriever is a hybrid system combining a dense approach with BM25,' which repeats information already given in the first sentence ('which combines a dense approach with BM25'). Consider consolidating.
  2. Section 2.1, step 1: 'This size limit is chosen arbitrarily, with the aim of limiting the amount of information to process for each annotation.' The word 'arbitrarily' could be replaced with a more precise statement of the practical constraint.
  3. Figures 1-3: The confidence intervals are described as 'substantial' and overlapping, making it difficult to distinguish statistically significant differences between metrics. The figures would benefit from clearer visual indication of which pairwise differences are significant (or an explicit statement that none are).
  4. Section 2.3, Overall Metrics: The distinction between 'precision' and 'recall' as aspects of factuality is described, but it would help to clarify whether these correspond to the same concepts used in retrieval metrics or are analogous but distinct.
  5. Section 6: The related work overview is described as 'non-exhaustive.' Consider adding a brief statement of the selection criteria for cited works to help the reader assess coverage.

Circularity Check

0 steps flagged

No formal circularity; one minor self-citation for translation provenance that is not load-bearing.

full rationale

The paper is an empirical correlation study. No metric is defined in terms of the human scores it is validated against — the metrics (Ragas, DeepEval, RAGChecker, Opik) are computed independently and then Pearson-correlated with reference scores. The word-level recall baseline is chosen partly because it correlates better with human scores (r=0.35 vs r=0.05 for document-level recall), which introduces a mild selection bias, but this is not circularity in the formal sense: recall is a fixed, parameter-free metric defined independently of the human rubric, and its selection does not force the correlations of other metrics. The single self-citation (Brabant, 2026) merely attributes the original French workshop version and carries no load-bearing mathematical or empirical claim. The confounding issue the skeptic raises — that metrics may correlate with human scores by capturing question difficulty rather than answer quality — is a validity threat the paper itself acknowledges in Section 5, but it is a confound, not a circular derivation. No step in the paper's chain reduces by construction to its inputs.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

No new entities, particles, or constructs are introduced.

free parameters (3)
  • k (top-k retrieved spans) = 5
    Set to 5 to match the number of spans inserted into the generator's prompt. Stated in Section 2.3.
  • page size limit for annotation = 1000 words
    Chosen arbitrarily to limit annotation effort. Stated in Section 2.1.
  • max questions per page = 5
    Arbitrary cap on annotation density. Stated in Section 2.1.
axioms (3)
  • domain assumption Pearson correlation between metric scores and human scores is a valid measure of metric relevance.
    This is the foundational assumption of the entire methodology. Invoked implicitly throughout Section 3 and Section 4. The paper acknowledges in Section 5 that this assumption is problematic when only one system is evaluated.
  • domain assumption The 1-5 rubric in Table 2 adequately captures the combined factuality-and-relevance criterion.
    The rubric is the ground truth against which all metrics are evaluated. Inter-annotator correlation of 0.85 provides partial support, but the rubric's adequacy is assumed, not independently validated.
  • domain assumption Word-level recall is an appropriate reference metric for retrieval quality.
    Section 2.3 justifies this choice over document-level recall by noting better correlation with human scores (r=0.35 vs r=0.05), but this justification is itself circular — using correlation with the same human scores that are the evaluation target.

pith-pipeline@v1.1.0-glm · 11211 in / 3154 out tokens · 132374 ms · 2026-07-09T14:37:34.837851+00:00 · methodology

0 comments
read the original abstract

This paper reports an empirical study evaluating the relevance of several RAG metrics. The experiment is based on a question-answering dataset created by human annotators from business data. The generated responses and retrieved spans of a RAG system are scored using evaluation metrics from four libraries (Ragas, DeepEval, RAGChecker, Opik). These metrics are compared to scores given by two evaluators, as well as to standard metrics such as recall. An analysis of correlations is conducted. Finally, we highlight certain limitations of our methodology, compare it to those used in the literature, and suggest some avenues for future research. This paper is an English translation of a paper originally published in the French-speaking workshop EvalLLM (Brabant, 2026).

Figures

Figures reproduced from arXiv: 2607.07302 by Quentin Brabant.

Figure 1
Figure 1. Figure 1: Pearson correlation of overall metrics with average human rates. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Pearson correlation of overall metrics with recall. Metrics that do not rely [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Pearson correlations of retrieval metrics with average human rates. Metrics [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · 4 internal anchors

  1. [1]

    OPT­Tree: Speculative Decoding with Adaptive Draft Tree Structure,

    ISSN 2307-387X. doi: 10.1162/tacl a 00417. URLhttps://doi.org/10.1162/tacl_a_00417. Tu Anh Dinh, Carlos Mullov, Leonard B¨ armann, Zhaolin Li, Danni Liu, Simon Reiß, Jueun Lee, Nathan Lerzer, Jianfeng Gao, Fabian Peller-Konrad, Tobias R¨ oddiger, Alexander Waibel, Tamim Asfour, Michael Beigl, Rainer Stiefelhagen, Carsten Dachsbacher, Klemens B¨ ohm, and J...

    work page internal anchor Pith review doi:10.1162/tacl 2024

  2. [2]

    doi: 10.18653/v1/2024.emnlp-main

    Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main

    work page doi:10.18653/v1/2024.emnlp-main 2024

  3. [3]

    Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert

    URLhttps://aclanthology.org/2024.emnlp-main.647/. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. RAGAs: Automated Evaluation of Retrieval Augmented Generation. In Nikolaos Ale- tras and Orphee De Clercq, editors,Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Sys- tem Demons...

    work page 2024

  4. [4]

    RAGAs: Automated evaluation of retrieval augmented generation

    Associa- 11 tion for Computational Linguistics. doi: 10.18653/v1/2024.eacl-demo.16. URL https://aclanthology.org/2024.eacl-demo.16/. Mingqi Gao, Xinyu Hu, Li Lin, and Xiaojun Wan. Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluation. In Luis Chiruzzo, Alan Rit- ter, and Lu Wang, editors,Proceedings of the 2025 Conference of the Nations of ...

    work page doi:10.18653/v1/2024.eacl-demo.16 2024

  5. [5]

    ISBN 979-8-89176-189-6

    Association for Computational Linguis- tics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long.111. URL https://aclanthology.org/2025.naacl-long.111/. Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. CritiqueLLM: Towards an Informative Critique Gene...

    work page doi:10.18653/v1/2025.naacl-long.111 2025

  6. [6]

    doi: 10.18653/v1/2024.acl-long.704

    Association for Computational Linguis- tics. doi: 10.18653/v1/2024.acl-long.704. URLhttps://aclanthology.org/ 2024.acl-long.704/. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Langua...

    work page doi:10.18653/v1/2024.acl-long.704 2024

  7. [7]

    doi: 10.18653/v1/2024.emnlp-main.248

    As- sociation for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.248. URLhttps://aclanthology.org/2024.emnlp-main.248/. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench: Evaluating Reward Models f...

    work page doi:10.18653/v1/2024.emnlp-main.248 2024

  8. [8]

    ISBN 979-8-89176-195-7

    Association for Computational Lin- guistics. ISBN 979-8-89176-195-7. doi: 10.18653/v1/2025.findings-naacl.96. URL https://aclanthology.org/2025.findings-naacl.96/. Minqian Liu, Ying Shen, Zhiyang Xu, Yixin Cao, Eunah Cho, Vaibhav Kumar, Reza Ghanadan, and Lifu Huang. X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with ...

    work page doi:10.18653/v1/2025.findings-naacl.96 2025

  9. [9]

    doi: 10.18653/v1/2024.naacl-long.473

    Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.473. URL https://aclanthology.org/2024.naacl-long.473/. Stefano Perrella, Lorenzo Proietti, Alessandro Scir` e, Edoardo Barba, and Roberto Navigli. Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In! In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, edi...

    work page doi:10.18653/v1/2024.naacl-long.473 2024

  10. [10]

    doi: 10.18653/v1/2024.acl-long.856

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.856. URLhttps://aclanthology.org/2024.acl-long.856/. Dongyu Ru, Lin Qiu, Xiangkun Hu, Tianhang Zhang, Peng Shi, Shuaichen Chang, Cheng Jiayang, Cunxiang Wang, Shichao Sun, Huanyu Li, Zizhao Zhang, Bin- jie Wang, Jiarong Jiang, Tong He, Zhiguo Wang, Pengfei Liu, Yue Zhang, and Zheng ...

    work page doi:10.18653/v1/2024.acl-long.856 2024

  11. [11]

    RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation

    doi: 10.48550/arXiv.2408.08067. URL http://arxiv.org/abs/2408.08067. Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. LLLaVA-Critic: Learning to Evaluate Multi- modal Models.2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13618–13628, June

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2408.08067 2025

  12. [12]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference (2025)

    doi: 10.1109/CVPR52734. 2025.01271. URLhttps://ieeexplore.ieee.org/document/11093772/. Con- ference Name: 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) ISBN: 9798331543648 Place: Nashville, TN, USA. Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Wang, and Lei Li. INSTRUCTSCORE: Towards Explainabl...

    work page doi:10.1109/cvpr52734 2025

  13. [13]

    doi: 10.18653/v1/2023.emnlp-main.365

    As- sociation for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.365. URLhttps://aclanthology.org/2023.emnlp-main.365/. Anar Yeginbergen, Maite Oronoz, and Rodrigo Agerri. Dynamic Knowledge Inte- gration for Evidence-Driven Counter-Argument Generation with Large Language Models

    work page doi:10.18653/v1/2023.emnlp-main.365 2023

  14. [14]

    Dynamic Knowledge Integration for Evidence-Driven Counter-Argument Generation with Large Language Models

    doi: 10.48550/ARXIV.2503.05328. URLhttps://arxiv.org/ abs/2503.05328. Version Number:

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2503.05328

  15. [15]

    JudgeLM: Fine-tuned Large Language Models are Scalable Judges

    URLhttp://arxiv.org/ abs/2310.17631. arXiv:2310.17631 [cs]. 13

    work page internal anchor Pith review Pith/arXiv arXiv