REVIEW 16 cited by
RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models
read the original abstract
Retrieval-augmented generation (RAG) has become a main technique for alleviating hallucinations in large language models (LLMs). Despite the integration of RAG, LLMs may still present unsupported or contradictory claims to the retrieved contents. In order to develop effective hallucination prevention strategies under RAG, it is important to create benchmark datasets that can measure the extent of hallucination. This paper presents RAGTruth, a corpus tailored for analyzing word-level hallucinations in various domains and tasks within the standard RAG frameworks for LLM applications. RAGTruth comprises nearly 18,000 naturally generated responses from diverse LLMs using RAG. These responses have undergone meticulous manual annotations at both the individual cases and word levels, incorporating evaluations of hallucination intensity. We not only benchmark hallucination frequencies across different LLMs, but also critically assess the effectiveness of several existing hallucination detection methodologies. Furthermore, we show that using a high-quality dataset such as RAGTruth, it is possible to finetune a relatively small LLM and achieve a competitive level of performance in hallucination detection when compared to the existing prompt-based approaches using state-of-the-art large language models such as GPT-4.
Forward citations
Cited by 16 Pith papers
-
REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations
REALISTA optimizes continuous combinations of valid editing directions in latent space to produce realistic adversarial prompts that elicit hallucinations more effectively than prior methods, including on large reason...
-
Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench
AgentProp-Bench shows substring judging agrees with humans at kappa=0.049, LLM ensemble at 0.432, bad-parameter injection propagates with ~0.62 probability, rejection and recovery are independent, and a runtime fix cu...
-
RAGognizer: Hallucination-Aware Fine-Tuning via Detection Head Integration
RAGognizer adds a detection head to LLMs for joint training on generation and token-level hallucination detection, yielding SOTA detection and fewer hallucinations in RAG while preserving output quality.
-
Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG
Answer-paired analysis of a 3×3 GPT/Grok/Gemini judge matrix finds near-zero same-model recall bias for induced RAG grounding errors; remaining flag gaps reflect label-task mismatch.
-
Knowledge Base Poisoning Attacks and Defense for Policy-Aware LLM-RAG Framework
Query-agnostic KB poisoning corrupts 85% of IoBT LLM contexts from one rule; taxonomy-aware dual detection restores 100% integrity with 7 ms overhead.
-
ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence
Introduces ConflictScore (CS-C and CS-R) to quantify how language model responses acknowledge conflicting evidence in grounding documents, plus ConflictBench for systematic evaluation.
-
Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?
GroundedCache reduces unsafe-served rate in RAG answer caching to 0-1.5% (vs 15-51.5% naive) via four validation gates while keeping p50 latency within 1.07x of no-cache baseline.
-
The Attribution Blind Spot: Detecting When Language Models Rely on Memory Rather Than Retrieved Context
Computational Reality Monitoring detects internal representational signatures of pretraining exposure in language models, revealing when retrieved context is not actually used for generation.
-
REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations
REALISTA generates semantically coherent adversarial prompts via latent-space optimization over input-dependent editing directions, achieving stronger hallucination elicitation than prior realistic attacks on open-sou...
-
Hallucination Detection in LLMs with Topological Divergence on Attention Graphs
TOHA detects LLM hallucinations via topological divergence on attention graphs, showing consistent patterns and competitive benchmark results with minimal data.
-
Always-OnAgents:A Survey of Persistent Memory, State, and Governance in LLMAgents
Survey mapping persistent state in LLM agents along six axes and proposing the AOEP-v0 protocol to evaluate governance and recovery obligations.
-
Evaluating Retrieval-Augmented Generation for Explainable Malware Analysis
RAG frequently degrades LLM malware explanations when structured VirusTotal input is already available by introducing irrelevant context and narrative noise.
-
Mixed response geometry and critical crossover in the Ising model
In the 2D Ising model, the mixed response Ω_βh = −N cov(m,e) forms a localized ridge from criticality into finite-field crossover and collapses onto a susceptibility-constrained curve in normalized coordinates.
-
Mixed response geometry and critical crossover in the Ising model
Thermodynamic curvature on the (β, h) manifold in the Ising model produces a ridge that geometrically identifies the Widom line as the locus of maximal response extending from the critical point.
-
Policy-Aware Edge LLM-RAG Framework for Internet of Battlefield Things Mission Orchestration
PA-LLM-RAG adds policy retrieval and dual-LLM verification to enable reliable low-latency mission orchestration in simulated IoBT environments, with Gemma-2B reaching 100% policy compliance at 4.17s latency.
-
Query Expansion in the Age of Pre-trained and Large Language Models: A Comprehensive Survey
A comprehensive survey that organizes query expansion methods in the PLM/LLM era along four design dimensions, synthesizes application patterns, and outlines future directions.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.