THELMA: Task Based Holistic Evaluation of Large Language Model Applications-RAG Question Answering

Cibi Chakravarthy Senthilkumar; Jay Roberts; Naumaan Nayyar; Parul Kalra; Rafael Castrillo; Rutu Mulkar; Sujay Gandhi; Udita Patel; Xiaofei Zheng

REVIEW 3 major objections 5 minor 21 references

THELMA: Task Based Holistic Evaluation of Large Language Model Applications-RAG Question Answering

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read THELMA introduces six reference-free metrics that pinpoint which component of a RAG question-answering system needs improvement.

desk verdict The metric decomposition is genuinely useful for RAG diagnostics, but the reported validation is inconsistent (20 vs 240 points) and the self-distinctness formula has a normalization bug, so the reliability claims need more work. read the letter →

arxiv 2505.11626 v2 pith:Y5ZYSTXP submitted 2025-05-16 cs.CL cs.AI

Udita Patel , Rutu Mulkar , Jay Roberts , Cibi Chakravarthy Senthilkumar , Sujay Gandhi , Xiaofei Zheng , Naumaan Nayyar , Parul Kalra

show 1 more author

Rafael Castrillo

This is my paper · ORCID

classification cs.CLcs.AI

keywords RAGevaluationreference-freemetricsLLM-as-a-judgequestionansweringgroundednessretrieverdiagnosisresponseprecisionsourcequerycoverage

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

THELMA is a suite of six reference-free evaluation metrics for retrieval-augmented generation (RAG) question answering, built for systems where no labeled sources or reference answers exist. The paper's central claim is that by decomposing the query, the retrieved sources, and the generated response into atomic pieces and matching those pieces against each other, the framework can score retrieval relevance, answer coverage, verbosity, repetition, and factual grounding separately. Because the six scores are designed to move together in recognizable patterns, a developer can read a low score on one metric and a high score on another to decide whether to fix the retriever, the generator prompt, or the source chunking. The authors validate this by showing that their metrics agree with human pairwise preferences more consistently than coarse-grained relevance and faithfulness scores do, on a set of 240 hand-modified question-answer instances. The framework matters because production RAG monitoring currently lacks actionable, fine-grained, label-free diagnostics.

What carries the argument

The load-bearing mechanism is the decompose-match-aggregate pipeline. A decompose module turns each input into atomic units — queries into standalone questions with pronouns resolved, responses into individually verifiable claims, sources into facts (or, in one variant, keeps whole retrieved chunks). A match module then applies a metric-specific criterion to each unit against another member of the triad, returning a binary verdict; an aggregate module averages the verdicts into a score between 0 and 1. This shared machinery is what lets all six metrics be reference-free, because every comparison is internal to the (query, source, response) triple, and it is what gives the scores a common scale so their relative sizes carry diagnostic meaning.

What would settle it

Re-run the paper's pairwise human-preference study on an independent 240-point sample drawn from a different domain, using the same judge LLM. If Source Precision and Source Query Coverage fail to rank the human-preferred source set above the negative set at the same rate as Table 1, the central claim that the metrics consistently score the better input higher is falsified; the same test applies to each of the six metrics.

Watch

Extended reading notes

Core claim

The central discovery is that a RAG QA pipeline can be evaluated holistically without any reference response or relevance labels by scoring all three members of the RAG triad — query, retrieved sources, response — through six interdependent metrics. Each metric isolates one failure mode: Source Precision asks whether each retrieved chunk is essential; Source Query Coverage asks whether the chunks together answer every sub-question; Response Precision asks whether every sentence in the answer is necessary; Response Query Coverage asks whether the answer addresses all sub-questions; Response Self-Distinctness measures paraphrase-level repetition; and Groundedness checks each claim against the sources. The paper reports that on a 240-point human-annotated subset of WikiEval, these metrics score the human-preferred source set and response higher than the worse one, and that the interplay of the six scores points to a specific component to improve — retriever, prompt, generator, or source chunking.

Load-bearing premise

The whole framework inherits the reliability of the LLM used as judge: if the judge's decomposition and match verdicts do not track what a human would call essential, repetitive, or grounded, all six scores can be wrong even with perfectly defined formulas.

Editorial extensions

If this is right

If the six scores behave as reported, a RAG application can be monitored in production without maintaining reference answers, so evaluation keeps working when the source corpus is continually updated.
The metric-interplay table gives developers a concrete diagnostic rule: for instance, low response-query-coverage with high source-query-coverage points to the generator prompt, while low source-precision with high source-query-coverage points to the retriever.
Because THELMA uses the same underlying LLM judge for all six scores, swapping the judge from a proprietary model to an open-source one preserves the ranking of better versus worse inputs in the reported experiments, which matters for cost-sensitive deployments.
The granularity separates failure modes that coarse relevance scores conflate: a response can be verbosely repetitive (low self-distinctness) while still being fully grounded and complete, and the three response metrics score those dimensions independently.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension the paper does not run: the same decompose-match-aggregate machinery could be applied to semi-structured sources such as tables or JSON, since the matching criterion only needs a claim-level comparison; whether the judge LLM can reliably ground claims in tabular data is an open empirical question.
If the judge-LLM dependency is the weakest link, then a practical product consequence follows that the authors state as a limitation rather than a design goal: teams adopting THELMA should periodically audit judge agreement with human preference, because score drift could reflect judge drift rather than system drift.
The metric-interplay rules in Table 2 read like a decision tree with only two scores per rule; a natural extension would be to learn a stronger diagnostic mapping from the six-vector to component-level defect labels on a larger annotated dataset, which the current 240-point study is too small to validate.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short take: this is a practical reference-free RAG evaluation framework with a useful fine-grained split of the RAG triad into six metrics, but the evidence that the metrics reliably match human preferences is thinner than the paper claims.

What's genuinely new: the separation of source precision, source query coverage, response precision, response query coverage, response self-distinctness, and groundedness, plus the interplay table that maps metric patterns to the RAG component that likely needs fixing. The formal definitions with decompose/match/aggregate modules are clear, and the appendix prompts are reproducible enough to reimplement. This gives enterprise developers a way to diagnose retrievers, generators, prompts, and chunking without reference answers.

Soft spots, in order: 1. Dataset inconsistency. Section 5 says 240 data points; Appendix C says 20 base datapoints manually augmented. That's a big gap. If it's actually 20, then every agreement rate in Table 1 is a binomial proportion with standard error roughly 0.11, and values like SP2=0.35 are not clearly above chance. 2. No error bars or significance tests. Table 1 is the core reliability evidence and it's just point estimates. 3. Equation 7 normalizes the pairwise sum by |Dsen(r)| instead of the number of pairs. For a two-sentence response the maximum would be 0.5, and for four sentences it would exceed 1, contradicting the stated 0–1 range. The worked example in A.5 has three sentences so the arithmetic accidentally works for that case, but the formula is wrong. 4. No code or data released. The decomposition and match steps are LLM calls, so reproducibility depends on those artifacts. 5. LLM dependency is acknowledged in Section 7 but not reconciled with the strong claim in Section 5 that the metrics 'consistently' score better inputs higher. That's a leap.

The framework itself is coherent, and the writing is honest about limitations. The core idea is worth taking seriously.

Who this is for: people building or monitoring RAG QA systems who need diagnostic signals beyond a single relevance number. Practitioners will find the metric split and interplay table immediately useful.

Recommendation: send to peer review, but with heavy revision. Fix the data description, add confidence intervals and significance tests, fix Equation 7, release code and data, and temper the reliability claim. As it stands, the paper is a solid proposal with weak validation.

Referee Report

3 major / 5 minor

Summary. The paper proposes THELMA, a reference-free LLM-as-judge evaluation framework for RAG-based question answering. It defines six metrics: source precision in two variants, source query coverage, response precision, response query coverage, response self-distinctness, and groundedness. Each metric decomposes one input (query, source, or response) into atomic components, matches those components against another input using LLM-based prompts, and aggregates the match scores to a 0--1 value. The authors validate the metrics by comparing their agreement with human pairwise preferences on a modified WikiEval dataset and against RAGAs metrics, and they present a table of metric interplays intended to diagnose which RAG component (retriever, prompt, generator, or source chunking) needs improvement.

Significance. If the claims were fully supported, THELMA would be a practically valuable contribution: reference-free, component-specific evaluation with interpretable precision--coverage tradeoffs is clearly useful for monitoring enterprise RAG QA systems. The paper has genuine strengths: formal metric definitions, detailed prompts in Appendix B, worked examples in Appendix A, and a direct comparison with RAGAs. The observation that an open-source Llama-based judge gives comparable agreement to proprietary models is also useful, assuming the validation holds. However, the validation as reported is too thin to establish the central reliability claim, and one of the formal definitions contains a normalization error. The potential is real, but the current manuscript does not yet substantiate the headline claim that the metrics consistently score the better source set and response higher.

major comments (3)

[Section 5 and Appendix C] The central reliability claim is not supported by the validation as reported. Section 5 says results are reported on 'a randomly sampled subset (240 data points) of WikiEval', while Appendix C says 'we took a subset of 20 datapoints' and describes manually augmenting those 20 datapoints. These numbers are never reconciled, and no per-metric counts, confidence intervals, or significance tests are given for Table 1. With only 20 independent binary comparisons per metric, the standard error of an agreement proportion is about 0.11, so values such as THELMA Sonnet SP2=0.35 (below the 0.5 chance level for a pairwise preference task) and RQC=0.75 are not distinguishable from chance. This undercuts the statement that 'Our proposed metrics consistently score the better source set and response with higher value'.
[Eq. (7) and Appendix A.5] The response self-distinctness metric is not correctly normalized. Equation (7) divides the sum over pairs (r_j, r_u) by |Dsen(r)|, the number of sentences, rather than by the number of sentence pairs. For a response with n>3 mutually distinct sentences, the resulting score exceeds 1 (for n=4 it is 6/4=1.5), contradicting the stated 0--1 range in Section 4. The worked example in A.5 computes 1−0.66=0.33, which is 1 minus the average similarity, not the average of pairwise (1−msd) terms required by Eq. (7). The definition and the example must be reconciled, and all SD values in Table 1 should be re-derived from the corrected definition.
[Section 5 and Section 7] The validation does not establish the reliability of the LLM-as-judge modules that all six metrics depend on. Section 7 acknowledges that 'the framework heavily depends on the performance of underlying LLMs,' but the paper reports no experiment on agreement between the judge's decomposition and match decisions and human judgment on those intermediate steps, no sensitivity analysis across judge prompts, and no code, data, or annotation protocol is released. Consequently, the empirical claim in Table 1 cannot be independently checked or reproduced from the manuscript as it stands.

minor comments (5)

[Abstract] Grammar and typography need cleanup: 'THELMA consist' should be 'THELMA consists', and 'response.We' is missing a space after the period.
[Section 4.2] The sentence 'Ideally, generator should reject answering such queries (or part thereof) but only do so with 45% success rate' is incomplete and unsupported; either cite the relevant source or rephrase to state the intended point clearly.
[Appendix C] The sentence 'RAGAs does not contain coarse-grained metrics' should presumably read 'fine-grained metrics' given the preceding discussion; please correct.
[Appendix A.5] The list items in the self-distinctness example are labeled 'Decomposes response and match scores', but Dsen returns sentences, not atomic claims; clarify that the listed scores are pairwise similarity judgments between sentences.
[References] Reference formatting should be cleaned: the TruLens citation is inconsistent with the journal style, and the RAGChecker preprint appears twice as 2024a and 2024b with the same title and venue.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: THELMA's metrics are defined directly from the RAG triad, and the validation against human preference is an external benchmark rather than a fitted input.

full rationale

The paper's derivation chain is self-contained with respect to the circularity patterns this pass targets. Each THELMA metric is formally defined in Section 4 as a decompose-match-aggregate procedure over the query, source, and response inputs, with no fitted parameter that is later renamed as a prediction. For example, Source Precision (Eq. 2-3), Source Query Coverage (Eq. 4), Response Precision (Eq. 5), Response Query Coverage (Eq. 6), Response Self-Distinctness (Eq. 7), and Groundedness (Eq. 8) are all direct LLM-as-judge computations, and the paper explicitly credits external prior work for components such as FactScore's claim-extraction methodology. The central empirical claim, that THELMA 'consistently score[s] the better source set and response with higher value', is tested by agreement with human pairwise annotations, which is an external benchmark rather than an input used to construct the metrics. No load-bearing self-citation appears: the cited prior work (RAGAs, FactScore, TruLens, RAGChecker) is external, and no uniqueness theorem or ansatz is imported from the authors' own prior publications. The paper's own Limitations section acknowledges that 'the framework heavily depends on the performance of underlying LLMs'; that is an important reliability concern but not a circularity, since it does not mean the metrics reduce to their inputs by construction. Likewise, the inconsistencies between Section 5's '240 data points' and Appendix C's 'subset of 20 datapoints', the lack of confidence intervals, and the normalization issue in Eq. 7 are correctness and reproducibility risks, not circular derivation. Under the hard rule that circularity must be exhibited by quoting a specific reduction or fit-renamed-as-prediction, no such step is present, so the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The framework introduces no new physical or conceptual entities; it builds on existing LLM-based decomposition and matching techniques. The main external dependencies are the LLM judge and the ground-truth source assumption, which are explicit in the paper.

assumptions (3)

domain assumption The source corpora contain only accurate information and can be treated as ground truth.
Stated in Section 3. If sources contain errors, groundedness and query coverage scores become invalid because the framework treats sources as authoritative.
domain assumption The LLM used as judge reliably decomposes text and makes alignment judgments that match human preferences.
Every metric relies on LLM-based decompose and match modules. The authors admit in the Limitations that performance depends on the underlying LLM.
domain assumption The generator should not use internal knowledge and should only answer from retrieved sources.
Assumption 2 in Section 3. This is necessary for interpreting groundedness and for the framework's premise that sources are the only ground truth.

how reviews work

0 comments

Cite this review

Pith. "Pith review of THELMA: Task Based Holistic Evaluation of Large Language Model Applications-RAG Question Answering." pith.science (2026). https://pith.science/paper/Y5ZYSTXP

@misc{pith2026250511626,
  author       = {Pith},
  title        = {Pith review of: THELMA: Task Based Holistic Evaluation of Large Language Model Applications-RAG Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y5ZYSTXP}},
  note         = {Machine review of arXiv:2505.11626}
}

read the original abstract

We propose THELMA (Task Based Holistic Evaluation of Large Language Model Applications), a reference free framework for RAG (Retrieval Augmented generation) based question answering (QA) applications. THELMA consist of six interdependent metrics specifically designed for holistic, fine grained evaluation of RAG QA applications. THELMA framework helps developers and application owners evaluate, monitor and improve end to end RAG QA pipelines without requiring labelled sources or reference responses.We also present our findings on the interplay of the proposed THELMA metrics, which can be interpreted to identify the specific RAG component needing improvement in QA applications.

Figures

Figures reproduced from arXiv: 2505.11626 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 3 canonical work pages

[1]

Eleftheria Briakou, Zhongtao Liu, Colin Cherry, and Markus Freitag. 2024. https://arxiv.org/abs/2410.00863 On the implications of verbose llm outputs: A case study in translation evaluation . Preprint, arXiv:2410.00863

arXiv 2024
[2]

Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2023. https://arxiv.org/abs/2309.01431 Benchmarking large language models in retrieval-augmented generation . Preprint, arXiv:2309.01431

arXiv 2023
[3]

Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. https://doi.org/10.1145/3626772.3657834 The power of noise: Redefining retrieval for rag systems . In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Informati...

arXiv 2024
[4]

Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. https://aclanthology.org/2024.eacl-demo.16 RAGA s: Automated evaluation of retrieval augmented generation . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150--158, St. Julians, Malta. Asso...

2024
[5]

Xiangkun Hu, Dongyu Ru, Lin Qiu, Qipeng Guo, Tianhang Zhang, Yang Xu, Yun Luo, Pengfei Liu, Yue Zhang, and Zheng Zhang. 2024. http://arxiv.org/abs/2405.14486 RefChecker : Reference -based Fine -grained Hallucination Checker and Benchmark for Large Language Models . arXiv preprint. ArXiv:2405.14486 [cs]

arXiv 2024
[6]

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. https://doi.org/10.1145/3703155 A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions . ACM Transactions on Information Systems, 43(2):1–55

doi:10.1145/3703155 2025
[7]

O. M. Ozturk J. Ferrara, Ethan-Tonic. 2024. Rag triad; T ru L ens trulens.org. https://www.trulens.org/getting_started/core_concepts/rag_triad/. [Accessed 11-03-2025]

work page 2024
[8]

u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf Retrieval-augmented generation for knowledge-intens...

2020

Show all 21 references

[9]

Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. https://doi.org/10.1162/tacl_a_00638 Lost in the middle: How language models use long contexts . Transactions of the Association for Computational Linguistics, 12...

2024 doi
[10]

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.741 FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generatio...

2023 doi
[12]

Dongyu Ru, Lin Qiu, Xiangkun Hu, Tianhang Zhang, Peng Shi, Shuaichen Chang, Cheng Jiayang, Cunxiang Wang, Shichao Sun, Huanyu Li, Zizhao Zhang, Binjie Wang, Jiarong Jiang, Tong He, Zhiguo Wang, Pengfei Liu, Yue Zhang, and Zheng Zhang. 2024 b . http://arxiv.org/abs/2408.08067 R...

2024 arXiv
[13]

Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024. https://doi.org/10.18653/v1/2024.naacl-long.20 ARES : An automated evaluation framework for retrieval-augmented generation systems . In Proceedings of the 2024 Conference of the North American Chapter o...

2024 doi
[14]

Alireza Salemi and Hamed Zamani. 2024. Evaluating retrieval quality in retrieval-augmented generation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2395--2400

2024
[15]

Voorhees and Dawn M

Ellen M. Voorhees and Dawn M. Tice. 2000. https://aclanthology.org/L00-1018/ The TREC -8 question answering track . In Proceedings of the Second International Conference on Language Resources and Evaluation ( LREC `00) , Athens, Greece. European Language Resources Association (ELRA)

2000
[16]

Yang Wang, Alberto Garcia Hernandez, Roman Kyslyi, and Nicholas Kersting. 2024. https://arxiv.org/abs/2406.18064 Evaluating quality of answers for retrieval-augmented generation: A strong llm is all you need . Preprint, arXiv:2406.18064

2024 arXiv
[17]

Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. 2024. https://openreview.net/forum?id=S7NVVfuRv8 How easily do irrelevant inputs skew the responses of large language models? In First Conference on Language Modeling

2024
[18]

Zhentao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, and Zheng Li. 2024. https://doi.org/10.1145/3626772.3661370 Retrieval-augmented generation with knowledge graphs for customer service question answering . In Proceedings of the 47th Inter...

2024
[19]

Tianyang Zhang, Zhuoxuan Jiang, Shengguang Bai, Tianrui Zhang, Lin Lin, Yang Liu, and Jiawei Ren. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-industry.56 RAG 4 ITO ps: A supervised fine-tunable and comprehensive RAG framework for IT operations and maintenance . In Proceedi...

2024 doi
[20]

Weinberger, and Yoav Artzi

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

2020 arXiv
[21]

Yusen Zhang, Sarkar Snigdha Sarathi Das, and Rui Zhang. 2024 b . https://arxiv.org/abs/2411.07858 Verbosity veracity: Demystify verbosity compensation behavior of large language models . Preprint, arXiv:2411.07858

2024 arXiv
[22]

Xing, Hao Zhang, Joseph E

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th Internat...

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Eleftheria Briakou, Zhongtao Liu, Colin Cherry, and Markus Freitag. 2024. https://arxiv.org/abs/2410.00863 On the implications of verbose llm outputs: A case study in translation evaluation . Preprint, arXiv:2410.00863

arXiv 2024

[2] [2]

Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2023. https://arxiv.org/abs/2309.01431 Benchmarking large language models in retrieval-augmented generation . Preprint, arXiv:2309.01431

arXiv 2023

[3] [3]

Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. https://doi.org/10.1145/3626772.3657834 The power of noise: Redefining retrieval for rag systems . In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Informati...

arXiv 2024

[4] [4]

Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. https://aclanthology.org/2024.eacl-demo.16 RAGA s: Automated evaluation of retrieval augmented generation . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150--158, St. Julians, Malta. Asso...

2024

[5] [5]

Xiangkun Hu, Dongyu Ru, Lin Qiu, Qipeng Guo, Tianhang Zhang, Yang Xu, Yun Luo, Pengfei Liu, Yue Zhang, and Zheng Zhang. 2024. http://arxiv.org/abs/2405.14486 RefChecker : Reference -based Fine -grained Hallucination Checker and Benchmark for Large Language Models . arXiv preprint. ArXiv:2405.14486 [cs]

arXiv 2024

[6] [6]

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. https://doi.org/10.1145/3703155 A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions . ACM Transactions on Information Systems, 43(2):1–55

doi:10.1145/3703155 2025

[7] [7]

O. M. Ozturk J. Ferrara, Ethan-Tonic. 2024. Rag triad; T ru L ens trulens.org. https://www.trulens.org/getting_started/core_concepts/rag_triad/. [Accessed 11-03-2025]

work page 2024

[8] [8]

u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf Retrieval-augmented generation for knowledge-intens...

2020