REVIEW 4 cited by
SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Hallucination detection is a critical step toward understanding the trustworthiness of modern language models (LMs). To achieve this goal, we re-examine existing detection approaches based on the self-consistency of LMs and uncover two types of hallucinations resulting from 1) question-level and 2) model-level, which cannot be effectively identified through self-consistency check alone. Building upon this discovery, we propose a novel sampling-based method, i.e., semantic-aware cross-check consistency (SAC3) that expands on the principle of self-consistency checking. Our SAC3 approach incorporates additional mechanisms to detect both question-level and model-level hallucinations by leveraging advances including semantically equivalent question perturbation and cross-model response consistency checking. Through extensive and systematic empirical analysis, we demonstrate that SAC3 outperforms the state of the art in detecting both non-factual and factual statements across multiple question-answering and open-domain generation benchmarks.
Forward citations
Cited by 4 Pith papers
-
Hallucination Detection and Mitigation with Diffusion in Multi-Variate Time-Series Foundation Models
Pre-trained multivariate time-series imputation models frequently return values that violate known relations between variables, and a diffusion-based score can detect and filter these errors.
-
Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning
HyperClick trains GUI grounding models with GRPO to output clicks plus confidence scores, jointly rewarding correct clicks and Brier-calibrated confidence, and reports SOTA accuracy on six of seven benchmarks with bet...
-
How Knowledge Popularity Influences and Enhances LLM Knowledge Boundary Perception
Entity popularity and entity co-occurrence in Wikipedia correlate with LLM QA accuracy, confidence, and calibration, and combining them with confidence improves answer-correctness prediction by 5.24% on average.
-
Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models
A survey of LLM hallucination research that formalizes hallucination types and argues, via incompleteness and undecidability arguments, that hallucinations cannot be fully eliminated.
Discussion (0). Continue with ORCID to comment.