REVIEW 5 cited by
QuestEval: Summarization Asks for Fact-based Evaluation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Summarization evaluation remains an open research problem: current metrics such as ROUGE are known to be limited and to correlate poorly with human judgments. To alleviate this issue, recent work has proposed evaluation metrics which rely on question answering models to assess whether a summary contains all the relevant information in its source document. Though promising, the proposed approaches have so far failed to correlate better than ROUGE with human judgments. In this paper, we extend previous approaches and propose a unified framework, named QuestEval. In contrast to established metrics such as ROUGE or BERTScore, QuestEval does not require any ground-truth reference. Nonetheless, QuestEval substantially improves the correlation with human judgments over four evaluation dimensions (consistency, coherence, fluency, and relevance), as shown in the extensive experiments we report.
Forward citations
Cited by 5 Pith papers
-
Fact-Controlled Diagnosis of Hallucinations in Medical Text Summarization
A fact-alignment LLM method detects clinical summarization hallucinations better than existing metrics, with correlations of 0.43 on controlled data and 0.37 on natural errors.
-
AllSummedUp: un framework open-source pour comparer les metriques d'evaluation de resume
On SummEval, LLM-based summary evaluators are expensive and unstable, and several published correlations did not reproduce when using open-weight models.
-
CogniBench: A Legal-inspired Framework and Dataset for Assessing Cognitive Faithfulness of Large Language Models
CogniBench is a sentence-level benchmark that labels LLM inferences, explanations, and opinions as faithful or hallucinated, expanding hallucination evaluation beyond verbatim factual claims.
-
Exploring Modularity of Agentic Systems for Drug Discovery
On 26 chemistry questions, swapping the LLM, agent type, or prompt in an LLM agent changes its scores so much that the system cannot be treated as modular.
-
A comprehensive taxonomy of hallucinations in Large Language Models
A survey that organizes LLM hallucination types, causes, benchmarks, and mitigations, and restates the theorem that hallucination is inevitable for computable LLMs.
Discussion (0). Sign in to comment.