Pith. sign in

REVIEW 20 cited by

RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.00396 v2 pith:FX7ZWR4Q submitted 2023-12-31 cs.CL

Cheng Niu , Yuanhao Wu , Juno Zhu , Siliang Xu , Kashun Shum , Randy Zhong , Juntong Song , Tong Zhang This is my paper · ORCID
classification cs.CL
keywords hallucinationllmsragtruthlanguagemodelsbenchmarkcorpusdetection
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Retrieval-augmented generation (RAG) has become a main technique for alleviating hallucinations in large language models (LLMs). Despite the integration of RAG, LLMs may still present unsupported or contradictory claims to the retrieved contents. In order to develop effective hallucination prevention strategies under RAG, it is important to create benchmark datasets that can measure the extent of hallucination. This paper presents RAGTruth, a corpus tailored for analyzing word-level hallucinations in various domains and tasks within the standard RAG frameworks for LLM applications. RAGTruth comprises nearly 18,000 naturally generated responses from diverse LLMs using RAG. These responses have undergone meticulous manual annotations at both the individual cases and word levels, incorporating evaluations of hallucination intensity. We not only benchmark hallucination frequencies across different LLMs, but also critically assess the effectiveness of several existing hallucination detection methodologies. Furthermore, we show that using a high-quality dataset such as RAGTruth, it is possible to finetune a relatively small LLM and achieve a competitive level of performance in hallucination detection when compared to the existing prompt-based approaches using state-of-the-art large language models such as GPT-4.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG

    cs.CL 2026-07 conditional novelty 6.5 of 10

    Answer-paired analysis of a 3×3 GPT/Grok/Gemini judge matrix finds near-zero same-model recall bias for induced RAG grounding errors; remaining flag gaps reflect label-task mismatch.

  2. Knowledge Base Poisoning Attacks and Defense for Policy-Aware LLM-RAG Framework

    cs.CR 2026-07 conditional novelty 6.0 of 10

    Query-agnostic KB poisoning corrupts 85% of IoBT LLM contexts from one rule; taxonomy-aware dual detection restores 100% integrity with 7 ms overhead.

  3. MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Them

    cs.AI 2025-07 conditional novelty 6.0 of 10

    A new benchmark with a three-way hallucination taxonomy, snapshot-based test cases, and an LLM judge shows LLM agents hallucinate at over 30% of risky decision points, with open and closed models closer than expected.

  4. Beyond Facts: Evaluating Intent Hallucination in Large Language Models

    cs.CL 2025-06 reject novelty 6.0 of 10

    The paper proposes a query-centric evaluation of LLM "intent hallucination" via constraint decomposition, but the headline metric comparison is undermined by a self-referential human evaluation design.

  5. When to Trust Context: Self-Reflective Debates for Context Reliability

    cs.CL 2025-06 conditional novelty 6.0 of 10

    SR-DCR uses an asymmetric debate plus self-confidence to gate whether a model follows context or its prior, improving ClashEval accuracy on several models.

  6. Ask a Local: Detecting Hallucinations With Specialized Model Divergence

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A multilingual hallucination detector that flags words where a language-specialized model's perplexity diverges from the rest, achieving IoU around 0.3.

  7. Mixed response geometry and critical crossover in the Ising model

    cond-mat.stat-mech 2026-04 unverdicted novelty 5.0 of 10

    Thermodynamic curvature on the (β, h) manifold in the Ising model produces a ridge that geometrically identifies the Widom line as the locus of maximal response extending from the critical point.

  8. GOSU: Retrieval-Augmented Generation with Global-Level Optimized Semantic Unit-Centric Framework

    cs.CL 2025-08 reject novelty 5.0 of 10

    GOSU globally merges semantic units from text chunks into a unit-centric knowledge graph and uses three-tier keyword retrieval to improve RAG generation quality, according to LLM-judge win rates.

  9. CCL-XCoT: An Efficient Cross-Lingual Knowledge Transfer Method for Mitigating Hallucination Generation

    cs.CL 2025-07 conditional novelty 5.0 of 10

    CCL-XCoT combines curriculum-based contrastive pretraining with cross-lingual chain-of-thought fine-tuning, lifting hallucination-free rates in low-resource QA from 1-18% to 55-74%.

  10. Data-efficient Meta-models for Evaluation of Context-based Questions and Answers in LLMs

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Using TabPFNv2 on compressed LLM hidden states and attention lookback features detects RAG hallucinations with 250 training samples at levels near GPT-4o-based judges, though clearly below them on EManual.

  11. CogniBench: A Legal-inspired Framework and Dataset for Assessing Cognitive Faithfulness of Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    CogniBench is a sentence-level benchmark that labels LLM inferences, explanations, and opinions as faithful or hallucinated, expanding hallucination evaluation beyond verbatim factual claims.

  12. HalluMix: A Task-Agnostic, Multi-Domain Benchmark for Real-World Hallucination Detection

    cs.CL 2025-05 conditional novelty 5.0 of 10

    HalluMix is a 6.5k-example, multi-domain benchmark for hallucination detection built from NLI, QA, and summarization data; the authors report Quotient Detections as best with 0.82 accuracy and 0.84 F1.

  13. Beyond ROUGE: N-Gram Subspace Features for LLM Hallucination Detection

    cs.CL 2025-09 reject novelty 4.0 of 10

    Singular values of label-grouped n-gram frequency tensors are used as MLP features for hallucination detection, with reported gains on HaluEval that rely on label-aware grouping.

  14. Machine Assistant with Reliable Knowledge: Enhancing Student Learning via RAG-based Retrieval

    cs.IR 2025-06 reject novelty 4.0 of 10

    A RAG-based tutoring and support chatbot with hybrid search and an instructor feedback loop is described, but no quantitative evaluation of its accuracy is provided.

  15. Osiris: A Lightweight Open-Source Hallucination Detection System

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Fine-tuning Qwen2.5-7B on GPT-4o-generated perturbed multi-hop QA data yields recall 0.938 vs GPT-4o's 0.710 on RAGTruth hallucination detection, with lower precision (0.366 vs 0.446).

  16. ORION Grounded in Context: Retrieval-Based Method for Hallucination Detection

    cs.LG 2025-04 conditional novelty 4.0 of 10

    A retrieval-plus-NLI pipeline for hallucination detection reports F1 0.83 on RAGTruth, but the key components are proprietary and undocumented.

  17. Enhancing RAG with Active Learning on Conversation Records: Reject Incapables and Answer Capables

    cs.CL 2025-02 conditional novelty 4.0 of 10

    AL4RAG uses a retrieval-aware similarity metric to select annotation-worthy RAG conversation records, yielding DPO-trained models that reject hallucination-prone queries and preserve answer quality.

  18. A Survey on Responsible LLMs: Inherent Risk, Malicious Use, and Mitigation Strategy

    cs.AI 2025-01 conditional novelty 4.0 of 10

    A survey that organizes responsible-LLM research into five risk dimensions and four intervention phases, reviewing privacy, hallucination, value, toxicity, and jailbreak mitigation.

  19. Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models

    cs.CL 2025-06 reject novelty 3.0 of 10

    A survey of LLM hallucination research that formalizes hallucination types and argues, via incompleteness and undecidability arguments, that hallucinations cannot be fully eliminated.

  20. 100% Elimination of Hallucinations on RAGTruth for GPT-4 and GPT-3.5 Turbo

    cs.CL 2024-12 reject novelty 3.0 of 10

    Acurai reports 100% hallucination-free outputs on 37 RAGTruth conflict examples for GPT-4 and GPT-3.5 Turbo by rewriting queries and context into simplified 'Fully-Formatted Facts' and splitting similar terms.

Pith tools