Cleanse detects hallucinated LLM answers by computing the share of hidden-embedding cosine similarity that falls inside semantic clusters, and it beats several baselines in AUROC across four models and two QA benchmarks.
Predicting Question-Answering Performance of Large Language Models through Semantic Consistency
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Semantic consistency of a language model is broadly defined as the model's ability to produce semantically-equivalent outputs, given semantically-equivalent inputs. We address the task of assessing question-answering (QA) semantic consistency of contemporary large language models (LLMs) by manually creating a benchmark dataset with high-quality paraphrases for factual questions, and release the dataset to the community. We further combine the semantic consistency metric with additional measurements suggested in prior work as correlating with LLM QA accuracy, for building and evaluating a framework for factual QA reference-less performance prediction -- predicting the likelihood of a language model to accurately answer a question. Evaluating the framework on five contemporary LLMs, we demonstrate encouraging, significantly outperforming baselines, results.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Cleanse: Uncertainty Estimation Approach Using Clustering-based Semantic Consistency in LLMs
Cleanse detects hallucinated LLM answers by computing the share of hidden-embedding cosine similarity that falls inside semantic clusters, and it beats several baselines in AUROC across four models and two QA benchmarks.