CCRS: A Zero-Shot LLM-as-a-Judge Framework for Comprehensive RAG Evaluation

Aashiq Muhamed

REVIEW 3 major objections 4 minor 25 references

CCRS: A Zero-Shot LLM-as-a-Judge Framework for Comprehensive RAG Evaluation

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single zero-shot LLM judge, prompted for five quality dimensions, can evaluate RAG outputs as well as a multi-stage pipeline while running roughly five times faster.

desk verdict Solid, honest zero-shot RAG evaluation suite with careful statistics; the RAGChecker comparison uses a substituted extractor, so treat the DP comparison as suggestive, not decisive. read the letter →

arxiv 2506.20128 v1 pith:UE3QTNKN submitted 2025-06-25 cs.CL cs.AIcs.LG

Aashiq Muhamed This is my paper

classification cs.CLcs.AIcs.LG

keywords Retrieval-AugmentedGenerationRAGevaluationLLM-as-a-judgezero-shotContextualCoherenceAnswerCorrectnessInformationRecallBioASQ

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a single, off-the-shelf large language model, used with nothing but five short prompts, can be a complete judge of retrieval-augmented generation (RAG) quality. The five scores — branded CCRS — cover contextual coherence, question relevance, information density, answer correctness, and information recall, all read directly from the judge with no fine-tuning. On the BioASQ biomedical question-answering benchmark, applied to six RAG configurations, the scores cleanly separate good from bad systems (the Mistral-7B reader beats the Llama readers on every dimension) and match or exceed the much heavier RAGChecker pipeline's power to tell systems apart, at about one-fifth of the compute. If the claim holds, comprehensive RAG evaluation no longer needs claim-extraction pipelines, fine-tuned judge models, or question-generation steps.

What carries the argument

The machinery is the CCRS metric suite: five constructs — Contextual Coherence (CC), Question Relevance (QR), Information Density (ID), Answer Correctness (AC), and Information Recall (IR) — each scored by the same Llama-70B-Instruct judge from a single prompt asking for an integer 0–100, normalized to $[0,1]$. The efficiency claim rests on this end-to-end design: unlike RAGChecker, no claims are extracted and no pairwise entailment checks run. The only composite is Answer Correctness, $AC(r,g,C) = \lambda \cdot \mathrm{EM}(r,g) + (1-\lambda) \cdot \mathrm{LLMJudge}(r,g,C,\mathrm{prompt}_{AC})/100$ with $\lambda = 0.7$, blending a strict exact-match test with the judge's semantic reading so that the benchmark's precision expectations are respected. Discriminative power is quantified by Tukey's HSD tests with 10,000 permutations across all 15 system pairs, giving each metric the same yardstick for comparing CCRS against RAGChecker.

What would settle it

Take a sample of the same 4,719 BioASQ responses, have human annotators rate each on the five quality dimensions, and compute the rank correlation between human scores and the corresponding CCRS metrics; if the Spearman correlation for Answer Correctness or Information Recall falls below roughly 0.5 on held-out data, the zero-shot judge premise fails. A second decisive check is to rerun the six-system evaluation with a different judge model (a frontier API model, or a much smaller local one) and see whether the system ranking and the 15-pair significance pattern survive; if the order flips, the result is a property of one specific judge rather than of zero-shot judgment in general.

Watch

Extended reading notes

Core claim

The paper's central claim is that five metrics, each obtained by prompting Llama-70B-Instruct zero-shot to return a 0–100 score, are enough to evaluate and rank RAG systems comprehensively on biomedical question answering. The evidence offered is discriminative power — the fraction of the 15 pairwise system comparisons that a metric separates statistically: Question Relevance separates 14, Information Recall 13, and Answer Correctness 12, whereas RAGChecker's Faithfulness and Recall separate 12 each and its Precision only 3. The metrics also behave as a credible judge should: Answer Correctness and Information Recall correlate strongly ($r = 0.756$), Contextual Coherence stands apart from the other four (weak to moderate correlations), and the system-level findings the paper expects — Mistral-7B beating both Llama readers on all five dimensions, and the E5 retriever lifting relevance and recall for Llama readers — come out significant at adjusted $p < 0.0001$. The paper therefore concludes that zero-shot, end-to-end LLM judgment delivers discrimination for relevance, completeness, and correctness comparable to or better than the claim-extraction pipeline's, at roughly one-fifth of the compute and with far less implementation complexity.

Load-bearing premise

Everything rests on the assumption that Llama-70B-Instruct's zero-shot scores mean what a human evaluator would mean by good RAG output — an alignment the paper itself states has not been formally established, so a biased or noisy judge would invalidate the system rankings and the framework's conclusions.

Editorial extensions

If this is right

RAG evaluation can be run with five LLM calls per response instead of a multi-stage pipeline, which the paper measures as roughly five times faster than RAGChecker's claim-extraction route.
The reader model is the dominant lever: Mistral-7B beat both Llama readers on every CCRS dimension by large margins (up to about 20 points in contextual coherence), so component-level comparisons that ignore the generator will misattribute failures.
Switching from BM25 to the E5 dense retriever raised question relevance and information recall for Llama-based systems without yielding corresponding gains in answer correctness, isolating a generation-stage bottleneck that better retrieval alone does not fix.
For fine-grained per-response ranking, Answer Correctness and Information Recall are the workhorses (tie rates of 15–18%, good discrimination), whereas Question Relevance, though the best system-level separator, saturates with 65–85% of responses receiving a perfect score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the study uses one judge model on one biomedical benchmark, its central claim would be most convincing if the fivefold efficiency gap and the discrimination pattern survive a judge swap; that stability is untested, and a cheaper or stronger judge could plausibly change the picture.
The strong Answer Correctness–Information Recall correlation ($r = 0.756$) suggests the two prompts track overlapping signal, so a leaner variant of CCRS might drop one of them and keep most of its discrimination at four calls per response.
The Question Relevance ceiling effect (65–85% perfect scores) looks like an artifact of the 0–100 integer scale with a single perfect-score anchor; rubric-anchored prompting is an obvious testable fix for recovering granularity among strong answers.
A human-correlation study on the same outputs would do double duty: it would validate or invalidate the zero-shot judge premise, and it would reveal whether RAGChecker's extra pipeline complexity buys anything that ranking-based tests cannot already see.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a well-executed workshop paper that packages five zero-shot LLM-as-a-judge metrics for RAG evaluation and tests them on BioASQ. It is not a conceptual breakthrough—RAGAS, TruLens, and others already use zero-shot LLM judging for RAG—but the specific metric suite and the detailed empirical analysis give practitioners a cheap, multi-dimensional diagnostic before they invest in heavier pipelines.

What it does well: the statistics are careful. The authors use permutation-based Tukey HSD with B=10,000, report distributions, tie rates, bounds, and convergent/discriminant validity, and they correct for multiple testing across hypotheses. They include the testing code in the appendix. They are also transparent about limitations: no human validation, ceiling effects on QR, and the fact that AC is capped because exact match almost never fires. That honesty is the paper's best trait.

Shortcomings, in order of importance:

1. The AC formula is degenerate in practice. With lambda=0.7 and EM=0 for every observed response, AC = 0.3 times the LLM score, never exceeding about 0.307. The paper acknowledges this but still reports AC as a headline metric. It should be fixed, for example by replacing exact match with a fuzzy semantic matching term.

2. The RAGChecker comparison is not apples-to-apples. Claim extraction for RAGChecker is done with Llama-70B instead of the original GPT-4o. The paper discloses this, but the claim of "comparable or superior discriminative power" rests on a baseline that may be degraded. RAGChecker Precision having DP=0.200 (only 3/15 pairs significant) looks suspicious and is likely an artifact of the extractor substitution. The conclusion should be softened, or the original RAGChecker re-run with GPT-4o.

3. No human alignment data. The paper names this in Section 5; it is the main open question for any LLM-as-a-judge. The discriminative power argument is internal; without external anchoring we do not know whether the scores track what a human would call good RAG output.

The correlation between AC and IR is high (r=0.756), which weakens the claim of five distinct constructs, but that is a minor point.

Who is this for: practitioners building RAG systems who want a quick, open-source evaluation tool, and researchers interested in comparative metric studies. The paper deserves a serious referee—the empirical work is solid and reproducible in principle, and the limitations are clearly flagged. With the RAGChecker caveat and a fix to AC, it would be a solid workshop paper. Send it to review, but ask for the RAGChecker baseline to be re-run or appropriately caveated, and for a discussion of the AC degeneracy.

Referee Report

3 major / 4 minor

Summary. The paper proposes CCRS, a suite of five metrics (CC, QR, ID, AC, IR) that use a single Llama-70B-Instruct model as a zero-shot judge to evaluate RAG outputs. The framework is applied to six RAG configurations on the BioASQ dataset, and the paper analyzes metric distributions, validity, tie rates, and discriminative power (via permutation-based Tukey HSD tests), concluding that CCRS provides comparable or superior discriminative power to RAGChecker while being about 5x faster. The paper also tests three hypotheses about retriever and reader effects, all of which are supported.

Significance. The paper's strength is its careful empirical methodology: permutation-based Tukey HSD tests with B=10,000, Holm-Bonferroni correction, distribution diagnostics, and accompanying code in Appendix F. If the zero-shot judge's scores were valid, CCRS would be a practical, efficient alternative to multi-stage RAG evaluation frameworks. The contribution is potentially valuable for rapid system comparison, and the finding that QR and IR are highly discriminative while AC and IR strongly correlate is an interesting empirical observation. However, the paper's central claims are currently limited by two issues: the AC metric reduces to a rescaled LLM score because exact match is never achieved, and the RAGChecker comparison uses a substituted claim extractor rather than the original pipeline. Additionally, no human correlation study is provided, so the validity of the metrics as proxies for user satisfaction remains unestablished. With these gaps addressed, the framework could be a strong practical contribution.

major comments (3)

[Section 2.3.2, Table 6, Table 15] The Answer Correctness (AC) metric is defined as AC = lambda * EM(r,g) + (1-lambda) * LLMJudge(r,g,C)/100 with lambda=0.7. In the reported results, EM is never equal to 1 (Table 6 shows zero responses achieve AC=1.0, and the maximum AC across all systems is 30.7 in Table 15), which is consistent with EM=0 for every response and AC reducing to 0.3 times the LLM judgment. Thus the exact-match component is effectively inert, and the choice of lambda=0.7 is a free parameter that merely rescales the LLM score, capping AC at about 30. This means the AC metric does not actually combine two sources of evidence as claimed, and its reported discriminative power is simply the discriminative power of the scaled LLM score. The authors should either modify the AC formulation so that the EM component can be nonzero (e.g., using a semantic-equivalence threshold), or explicitly present AC as a rescaled LLM score and discuss the implications for score interpretability and the choice of lambda.
[Section 3.3 and Section 4.1.4] The comparison with RAGChecker is not carried out against the RAGChecker framework as originally specified. Section 3.3 states that claim extraction for RAGChecker was performed with Llama-70B-Instruct instead of the GPT-4o extractor used in the original RAGChecker paper. Since RAGChecker's Precision, Recall, and Faithfulness are all computed from the extracted claim sets, substituting the extractor changes the measurements. The reported DP values, especially RAGChecker Precision's DP=0.200 in Table 3 (despite small mean differences across systems in Table 19), may reflect properties of the substituted extractor rather than RAGChecker itself. To support the conclusion that CCRS offers 'comparable or superior' discriminative power, the authors should either run the original RAGChecker pipeline with its specified GPT-4o extractor, or clearly reframe the comparison as being against a modified RAGChecker variant and temper the claim accordingly.
[Section 5 and Section 2.3.1] The paper defines the CCRS constructs as proxies for 'user satisfaction' (Section 2.3.1) and states in Section 5 that CCRS is a 'viable, efficient, and effective evaluation framework,' but the alignment between CCRS scores and human judgments is not validated. The paper explicitly acknowledges in Section 5 that 'the alignment between CCRS scores and human perception of quality remains to be formally established.' Because all reported analyses (distributions, correlations, DP) are internal to the judge model, and the only external comparison (RAGChecker) shares the same underlying Llama-70B model for claim extraction, the current evidence establishes only that CCRS produces stable system rankings, not that those rankings correspond to human-perceived RAG quality. The authors should add at least a small human annotation study or use a reference-based validation set to demonstrate the metrics' validity, or substantially weaken the claim that CCRS measures RAG quality rather than merely the judge's preferences.

minor comments (4)

[Table 15] The maximum AC value reported is 30.7, but if EM=0 and the LLM judge score is capped at 100 (normalized to 1.0), the formula gives a maximum of 30.0. Please clarify the computation or the exact-match implementation; this inconsistency is confusing and should be corrected.
[Section 4.1.4] The claim that CCRS is 'approximately 5 times faster' than RAGChecker lacks measurement details. Please specify the hardware, the number of LLM calls made by each framework, and how the wall-clock time was measured, since this is a central practical advantage.
[Section 3.3] The use of a substituted claim extractor for RAGChecker is stated only in the implementation details, yet it directly affects the validity of the main comparative conclusion. This caveat should be moved to the beginning of Section 4.1.4 and given prominent emphasis.
[References] Reference [19] is the original Llama paper, but the judge model is Meta-Llama-3-70B-Instruct; please cite the appropriate Llama 3 model release (or the specific model card) instead.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; the paper's claims are empirical and self-contained, with the main caveat being a disclosed baseline modification rather than a circular reduction.

full rationale

The CCRS metrics are defined directly as zero-shot judgments of Llama-70B-Instruct, normalized to [0,1], with Answer Correctness combining a fixed exact-match term (lambda=0.7) and an LLM semantic score. No parameter is fitted to an outcome that is later reported as a prediction, and no metric is defined in terms of another CCRS metric or in terms of the RAGChecker scores it is compared against. The discriminative-power analysis is an internal descriptive property of the scores, and the paper explicitly acknowledges that human alignment 'remains to be formally established,' so the absence of external validation is a stated limitation rather than a hidden circular reliance. The comparison with RAGChecker does alter the baseline by using Llama-70B for claim extraction instead of the original GPT-4o; however, the paper discloses this substitution and it weakens or confounds the comparison without making the CCRS derivation reduce to its own inputs. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The strongest caveat is a correctness/validity threat about the modified RAGChecker baseline, not circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The framework introduces five evaluation constructs that are not externally anchored: their validity is assumed. The AC formula has a hand-chosen lambda that is effectively a scaling factor, and several experimental choices (k, chunking, judge model) are fixed without sensitivity analysis. No human correlation data are provided.

free parameters (5)

lambda in AC formula = 0.7
Chosen by hand to emphasize exact matches (Section 2.3.2). Since exact match is effectively never achieved (Table 6 shows no AC score reaches 1.0), AC becomes 0.3 times the LLM score for essentially all responses, capping scores at about 30 and distorting the metric.
top-k retrieval count = 20
Section 3.3 sets k=20 chunks for all systems without sensitivity analysis. This choice affects the retrieved context and thus all metric scores.
chunking parameters = 300 tokens, 20% overlap
Section 3.3 chooses these values for document chunking; they are not varied and could affect retrieval quality and downstream scores.
judge model = Llama-70B-Instruct
The paper uses a single judge LLM (Section 3.3). Results may not generalize to other LLMs, and the prompts were not validated with different models.
prompt wording = Hand-designed prompts in Appendix B
The evaluation prompts are constructed by the author and not calibrated or tested for sensitivity to wording changes, a known issue with LLM-as-a-judge.

assumptions (4)

domain assumption Llama-70B-Instruct provides valid zero-shot judgments for contextual coherence, question relevance, information density, answer correctness, and information recall.
This is the core assumption of the CCRS framework, introduced in Section 2.3.2 and explicitly left unvalidated against human judgments in Section 5.
domain assumption BioASQ expert-curated ground truth answers are a reliable reference for correctness and recall.
The dataset (Section 3.1) is used as the gold standard without discussion of annotation noise, answer style variability, or coverage.
standard math Permutation-based Tukey HSD controls the family-wise error rate for the 15 pairwise comparisons.
Used for discriminative power analysis (Section 4.1.3) and hypothesis tests (Section 4.2). The procedure is generally accepted, though implementation details are in Appendix F.
domain assumption The retrieved context C is always available and meaningful, and the judge uses it consistently across all metrics.
The CC and ID metrics depend on C, but the paper does not analyze cases where retrieval returns empty or malformed contexts, which could affect judge behavior.

invented entities (1)

CCRS metric suite (CC, QR, ID, AC, IR)
purpose: To provide a zero-shot, end-to-end evaluation of RAG responses across five quality dimensions without intermediate pipelines or fine-tuning.
The metrics are defined by this paper and have not been validated against human judgments or independent benchmarks. The only evidence is internal discriminative power and correlation with RAGChecker, which shares the same judge LLM.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CCRS: A Zero-Shot LLM-as-a-Judge Framework for Comprehensive RAG Evaluation." pith.science (2026). https://pith.science/paper/UE3QTNKN

@misc{pith2026250620128,
  author       = {Pith},
  title        = {Pith review of: CCRS: A Zero-Shot LLM-as-a-Judge Framework for Comprehensive RAG Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UE3QTNKN}},
  note         = {Machine review of arXiv:2506.20128}
}

read the original abstract

RAG systems enhance LLMs by incorporating external knowledge, which is crucial for domains that demand factual accuracy and up-to-date information. However, evaluating the multifaceted quality of RAG outputs, spanning aspects such as contextual coherence, query relevance, factual correctness, and informational completeness, poses significant challenges. Existing evaluation methods often rely on simple lexical overlap metrics, which are inadequate for capturing these nuances, or involve complex multi-stage pipelines with intermediate steps like claim extraction or require finetuning specialized judge models, hindering practical efficiency. To address these limitations, we propose CCRS (Contextual Coherence and Relevance Score), a novel suite of five metrics that utilizes a single, powerful, pretrained LLM as a zero-shot, end-to-end judge. CCRS evaluates: Contextual Coherence (CC), Question Relevance (QR), Information Density (ID), Answer Correctness (AC), and Information Recall (IR). We apply CCRS to evaluate six diverse RAG system configurations on the challenging BioASQ dataset. Our analysis demonstrates that CCRS effectively discriminates between system performances, confirming, for instance, that the Mistral-7B reader outperforms Llama variants. We provide a detailed analysis of CCRS metric properties, including score distributions, convergent/discriminant validity, tie rates, population statistics, and discriminative power. Compared to the complex RAGChecker framework, CCRS offers comparable or superior discriminative power for key aspects like recall and faithfulness, while being significantly more computationally efficient. CCRS thus provides a practical, comprehensive, and efficient framework for evaluating and iteratively improving RAG systems.

Figures

Figures reproduced from arXiv: 2506.20128 by the authors.

**Figure 2.** Distribution of Mean CCRS Metric Values Across All Systems (N=4719 queries). These histograms show the frequency [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** CCRS Metric Validity (left) assessed via averaged [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Averaged Pearson Correlation Matrix comparing [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Box plots illustrating performance distributions for all CCRS metrics across the six RAG systems (A-F), plus ASL curves [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Appendix: Distribution of Contextual Coherence (CC) scores for each RAG system. [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]

**Figure 7.** Figure 7: Appendix: Distribution of Question Relevance (QR) scores for each RAG system. [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]

**Figure 8.** Figure 8: Appendix: Distribution of Information Density (ID) scores for each RAG system. [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]

**Figure 9.** Figure 9: Appendix: Distribution of Answer Correctness (AC) scores for each RAG system. [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]

**Figure 10.** Figure 10: Appendix: Distribution of Information Recall (IR) scores for each RAG system. [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 5 canonical work pages

[1]

Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Han- naneh Hajishirzi, and Wen-tau Yih. 2024. Reliable, adaptable, and attributable language models with retrieval. arXiv preprint arXiv:2403.03187 (2024)

arXiv 2024
[2]

Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937 (2023)

arXiv 2023
[3]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

arXiv 2024
[4]

Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023. Ra- gas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217 (2023)

arXiv 2023
[5]

Joe Ferrara, Ethan-Tonic, and Oguzhan Mete Ozturk. 2024. The RAG Triad . https://www.trulens.org/trulens_eval/core_concepts_rag_triad/

work page 2024
[6]

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 (2023)

arXiv 2023
[7]

Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large Language Models Cannot LLM4Eval@SIGIR ’25, July 13-17, 2025, Padua, Italy Aashiq Muhamed Self-Correct Reasoning Yet. arXiv:2310.01798 [cs.CL]

arXiv 2023
[8]

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al . 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023)

arXiv 2023

Show all 25 references

[9]

Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin Xu, Qiuxia Li, and Jun Zhao. 2024. Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models. arXiv preprint arXiv:2402.14409 (2024)

2024 arXiv
[10]

Evgenii Kortukov, Alexander Rubinstein, Elisa Nguyen, and Seong Joon Oh. 2024. Studying Large Language Model Behaviors Under Realistic Knowledge Conflicts. arXiv preprint arXiv:2404.16032 (2024)

2024 arXiv
[11]

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81

2004
[12]

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. arXiv preprint arXiv:2303.16634 (2023)

2023 arXiv
[13]

Yuanjie Lyu, Zhiyu Li, Simin Niu, Feiyu Xiong, Bo Tang, Wenjin Wang, Hao Wu, Huanyong Liu, Tong Xu, and Enhong Chen. 2024. CRUD-RAG: A comprehensive chinese benchmark for retrieval-augmented generation of large language models. arXiv preprint arXiv:2401.17043 (2024)

2024 arXiv
[14]

Meta. 2024. Introducing Meta Llama 3: The most capable openly available LLM to date. (2024). https://ai.meta.com/blog/meta-llama-3/

2024
[15]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

2002
[16]

Stephen Robertson, Hugo Zaragoza, et al . 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval 3, 4 (2009), 333–389

2009
[17]

Dongyu Ru, Lin Qiu, Xiangkun Hu, Tianhang Zhang, Peng Shi, Shuaichen Chang, Jiayang Cheng, Cunxiang Wang, Shichao Sun, Huanyu Li, Zizhao Zhang, Bin- jie Wang, Jiarong Jiang, Tong He, Zhiguo Wang, Pengfei Liu, Yue Zhang, and Zheng Zhang. 2024. RAGCHECKER: A Fine-grained Framewo...

2024 arXiv
[18]

Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2023. ARES: An Automated Evaluation Framework for Retrieval-Augmented Genera- tion Systems. arXiv:2311.09476 [cs.CL]

2023 arXiv
[19]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

2023 arXiv
[20]

George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Ser- gios Petridis, Dimitris Polychronopoulos, et al. 2015. An overview of the BIOASQ large-scale biomedical semantic inde...

2015
[21]

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368 (2023)

2023 arXiv
[22]

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi
[23]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623

2023
[24]

query_id

Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can Large Language Models Transform Computational Social Sci- ence? arXiv:2305.03514 [cs.CL] A BioASQ Dataset Examples Examples illustrating the structure of the BioASQ dataset used in our e...

2024 arXiv
[2019]

In International Confer- ence on Learning Representations

BERTScore: Evaluating text generation with BERT. In International Confer- ence on Learning Representations

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Han- naneh Hajishirzi, and Wen-tau Yih. 2024. Reliable, adaptable, and attributable language models with retrieval. arXiv preprint arXiv:2403.03187 (2024)

arXiv 2024

[2] [2]

Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937 (2023)

arXiv 2023

[3] [3]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

arXiv 2024

[4] [4]

Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023. Ra- gas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217 (2023)

arXiv 2023

[5] [5]

Joe Ferrara, Ethan-Tonic, and Oguzhan Mete Ozturk. 2024. The RAG Triad . https://www.trulens.org/trulens_eval/core_concepts_rag_triad/

work page 2024

[6] [6]

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 (2023)

arXiv 2023

[7] [7]

Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large Language Models Cannot LLM4Eval@SIGIR ’25, July 13-17, 2025, Padua, Italy Aashiq Muhamed Self-Correct Reasoning Yet. arXiv:2310.01798 [cs.CL]

arXiv 2023

[8] [8]

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al . 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023)

arXiv 2023