REVIEW 7 cited by
AlignScore: Evaluating Factual Consistency with a Unified Alignment Function
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Many text generation applications require the generated text to be factually consistent with input information. Automatic evaluation of factual consistency is challenging. Previous work has developed various metrics that often depend on specific functions, such as natural language inference (NLI) or question answering (QA), trained on limited data. Those metrics thus can hardly assess diverse factual inconsistencies (e.g., contradictions, hallucinations) that occur in varying inputs/outputs (e.g., sentences, documents) from different tasks. In this paper, we propose AlignScore, a new holistic metric that applies to a variety of factual inconsistency scenarios as above. AlignScore is based on a general function of information alignment between two arbitrary text pieces. Crucially, we develop a unified training framework of the alignment function by integrating a large diversity of data sources, resulting in 4.7M training examples from 7 well-established tasks (NLI, QA, paraphrasing, fact verification, information retrieval, semantic similarity, and summarization). We conduct extensive experiments on large-scale benchmarks including 22 evaluation datasets, where 19 of the datasets were never seen in the alignment training. AlignScore achieves substantial improvement over a wide range of previous metrics. Moreover, AlignScore (355M parameters) matches or even outperforms metrics based on ChatGPT and GPT-4 that are orders of magnitude larger.
Forward citations
Cited by 7 Pith papers
-
Hallucination Detection and Mitigation with Diffusion in Multi-Variate Time-Series Foundation Models
Pre-trained multivariate time-series imputation models frequently return values that violate known relations between variables, and a diffusion-based score can detect and filter these errors.
-
ICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMs
A residual-stream update metric, the ICR Score, and a lightweight probe built on it detect LLM hallucinations with higher AUROC than several baselines across three open models.
-
BEnchmarking LLMs for Ophthalmology (BELO) for Ophthalmological Knowledge and Reasoning
BELO is a new ophthalmology benchmark of 900 expert-checked multiple-choice questions with reasoning, used to evaluate six LLMs on accuracy and explanation quality.
-
Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinations
SSP adds a learned, sample-specific noise prompt to an LLM input and scores hallucination by the cosine shift in intermediate representations, outperforming output-confidence baselines on QA benchmarks.
-
Fact-Controlled Diagnosis of Hallucinations in Medical Text Summarization
A fact-alignment LLM method detects clinical summarization hallucinations better than existing metrics, with correlations of 0.43 on controlled data and 0.37 on natural errors.
-
CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactions
A new 109-conversation, 86-API benchmark for LLM function-calling in multi-turn dialogue shows top models at about 40% accuracy and near-zero performance on chains of 4+ calls.
-
UTSA-NLP at ArchEHR-QA 2025: Improving EHR Question Answering via Self-Consistency Prompting
Sentence selection, improved by self-consistency sampling with hand-tuned thresholds, is the main driver of answer quality in a small EHR question-answering shared task.
Discussion (0). Sign in to comment.