Pith. sign in

REVIEW 4 major objections 5 minor 27 references

Real-World Summarization: When Evaluation Reaches Its Limits

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In hotel-highlight summaries, the simplest faithfulness check—word overlap—competes with and often beats complex neural metrics, while LLM judges prove unreliable.

desk verdict A useful, honestly-reported case study on metric evaluation for hotel highlights, but the headline r=0.63 is only against coarse human error percentages; against the paper's own span-level error types the simple metrics are near zero, so the general faithfulness-baseline claim is overstated. read the letter →

arxiv 2507.11508 v1 pith:XMWG26CY submitted 2025-07-15 cs.CL

Patrícia Schmidtová , Ondřej Dušek , Saad Mahamood This is my paper
classification cs.CL
keywords faithfulnessevaluationLLMsummarizationhotelhighlightswordoverlapmetricsLLM-as-a-judgecrowdsourcedannotationnaturallanguageinferencereferenceless
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies how to check whether LLM-generated hotel highlights stay faithful to the hotel description they are drawn from. Using human categorical judgments and span-level error annotations, it compares word-overlap and n-gram metrics, trainable similarity methods, and LLM-as-a-judge setups. The paper argues that the simplest and cheapest check—overlap of word forms between highlight and source—achieves a Spearman correlation of $r=0.63$ with human judgments and often outperforms more complex methods when applied to out-of-domain data. It also argues that LLMs are not yet trustworthy as evaluators: GPT-4o over-annotates, o3-mini under-annotates, and none reaches the agreement between two human annotator groups. The practical stakes are that in real business use, non-checkable and incorrect information carry the highest risk, so an inexpensive reliable signal is worth having.

What carries the argument

The load-bearing object is the word-form overlap score—the proportion of words in a generated highlight that appear in the source description—used as a referenceless faithfulness signal. It carries the main result because it is cheap and domain-agnostic, correlating with human hallucination judgments. The comparison apparatus is the categorical human annotation signal (the percentage of annotators marking a hallucination), which turns disagreement into a graded target, and the span-level error typology (non-checkable, misleading, incorrect) that gives metrics and LLM judges a fine-grained benchmark. For LLM-as-a-judge, the mechanism is error-span annotation with a requested rationale, scored against human spans with hard and soft F1.

What would settle it

Re-annotate a random sample of the 496 description-highlight pairs with expert annotators (with adjudication) and recompute the Spearman correlations and LLM F1 scores; if word overlap no longer reaches roughly $r=0.63$, or if any LLM judge matches or exceeds human-human agreement on the clean span set, then the paper's central conclusions would be overturned.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that faithfulness evaluation in this real-world summarization task does not need complicated machinery. The strongest out-of-the-box signal is word-form coverage: how many words of the generated highlight also appear in the source description, reaching a Spearman correlation of $r=0.63$ with the percentage of human annotators who flagged a hallucination. A ModernBERT-based natural-language-inference entailment score performs slightly better ($r=0.67$) on the categorical set, while BERTScore, LaBSE similarity, and older NLI models lag or fail. For error location, the paper finds that LLM-as-a-judge span annotators are unreliable: GPT-4o produces many spurious 'incorrect' errors, o3-mini misses most errors, and Gemma3 matches the human error-type distribution most closely but under-counts non-checkable content. The paper also reports that crowdsourced span annotation is fragile—only 24% of workers passed attention checks and expert review found over half of annotated spans to be spurious—and that incorrect and non-checkable information are the error types with the largest estimated business impact.

Load-bearing premise

The whole comparison rests on treating crowdworkers' span annotations as a trustworthy reference for what counts as an error, even though only 24% passed attention checks and expert review found over half of the annotated spans spurious; if that reference is unreliable, the metric correlations and LLM-judge rankings could change.

Editorial extensions

If this is right

  • Simple lexical overlap should be reported as a baseline in real-world faithfulness evaluations; it is cheap, repeatable, and often competitive with trainable metrics on out-of-domain data.
  • Referenceless evaluation works for this task if the metric is validated against human annotations; NLI entailment on a current model is the best-performing option here.
  • LLM-as-a-judge annotations need validation against a human benchmark before deployment; without it, GPT-4o inflates and o3-mini hides error counts.
  • In a product setting, error-prevention effort should focus on incorrect and non-checkable statements, since these carry the highest business-impact ratings.
  • Crowdsourced span annotation on long inputs should not be assumed reliable; quality filters and expert checks are necessary even after attention-check filtering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same cheap word-overlap result likely transfers to other summarization tasks where the output paraphrases a single source document and hallucinations add new content words; a quick cross-domain test would estimate the transfer.
  • Inference: the poor agreement between crowd annotators suggests the reported correlations may be attenuated by label noise; a cleaner expert ground truth could raise or lower metric rankings rather than leave them unchanged.
  • Inference: LLM judges' systematic under- and over-annotation may be correctable with calibration or confidence thresholds, so the paper's negative verdict is about current off-the-shelf use, not about the approach in principle.
  • Inference: the business-impact result implies a simple taxonomy-based triage: flag non-checkable claims for human review before publication, since they are both common and risky.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies faithfulness evaluation of LLM-generated hotel highlights, combining categorical human judgments from prior work (120 examples, annotator percentages), new span-level crowd annotations (496 examples, two annotators each), and a small business-impact rating (60 spans). It compares simple overlap metrics, n-gram metrics, trained similarity/NLI models, and LLM-as-a-judge span annotators (GPT-4o, o3-mini, Gemma3). The headline finding is that simple word overlap correlates well with the categorical hallucination percentage (Spearman r=0.63) and often beats more complex methods out-of-domain; NLI with ModernBERT reaches r=0.67. LLM judges are reported as unreliable because they over- or under-annotate compared with humans, and the paper draws conclusions about business impact from its error-type distribution. The paper is transparent about annotation quality problems, including a 24% attention-check pass rate and expert review indicating that over half of annotated spans contained no actual error.

Significance. If the main claims held, the paper would provide a practical and inexpensive baseline for faithfulness checking in real-world summarization, and a cautionary result about using LLMs as judges in subjective, longer-input annotation tasks. The study has genuine strengths: it uses a realistic business domain, reports annotation costs and quality-control metrics honestly, makes its limitation about noisy ground truth explicit, and compares a broad set of metrics on the same data. The empirical comparison of human and LLM span annotation (Table 3) with a human-human agreement baseline is a useful contribution. However, as detailed below, the central generalization about simple metrics and faithfulness is not supported by the paper's own span-level results, and the noisy ground truth weakens several of the quantitative comparisons. These issues are fixable with revised claims and additional analysis, so the contribution is potentially valuable but needs substantial revision.

major comments (4)
  1. [§6, abstract, Appendix Table 4] The claim that simple statistical metrics are 'the best out-of-the-box options for measuring faithfulness' overgeneralizes from the r=0.63 result. That correlation is computed against the coarse 'Hallucination %' category from Section 3.1. Against the paper's own span-level faithfulness taxonomy (Section 3.2), Appendix Table 4 shows word-overlap form coverage correlates only -0.16 with non-checkable, -0.09 with misleading, -0.05 with incorrect, and -0.17 with any error. Non-checkable and misleading together account for 39% of highlights. Thus the simple metrics carry essentially no signal for most of the faithfulness error types the paper defines. The conclusion in Section 6 should be narrowed to coarse hallucination detection, or the discrepancy must be analyzed rather than left implicit.
  2. [§3.2, §7, Table 3] The span-level ground truth is too unreliable to support the paper's use of it as the validation basis for both LLM-as-a-judge and metric correlations. Only 24% of annotators passed attention checks, and domain experts found that slightly over half of the annotated spans contained no actual error. Yet Section 4.2 uses these spans to compute LLM precision/recall/F1 (Table 3), and Appendix Table 4 uses them to evaluate all metrics. If the spans are too noisy to trust, the near-zero span-level correlations cannot be explained away as metric failure; if they are trusted, the paper's own quality check undermines the LLM comparison. The authors should re-annotate a reliable subset (or report results on the expert-validated portion) and assess how their conclusions change; Section 7's statement that 'there are still signals to be learned' is not a substitute for this analysis.
  3. [§4.1, Table 2] Correlations are reported without confidence intervals or significance tests. With only 120 examples, the Spearman r=0.63 has a wide confidence interval (roughly 0.50-0.73), and the difference between word overlap (0.63) and ModernBERT NLI entailment (0.67) is not shown to be statistically meaningful. The claim that 'simpler metrics ... often outperform more complex methods' requires either bootstrap confidence intervals or a paired test across examples. Without this, the paper cannot support the comparative ranking that is central to its practical recommendation.
  4. [§3.3, §6] The business-impact conclusion that 'incorrect and non-checkable information pose the greatest risks' is based on only 60 error spans, of which 32 (53%) were judged by experts to have no actual error. With only 2 high-impact and 13 medium-impact spans, and no reported breakdown of impact by error type, the abstract's causal-sounding claim is not supported by the summary statistics given. Please report the contingency table by error type and hedge the conclusion accordingly, or reduce the claim to an observation about the few available examples.
minor comments (5)
  1. [§3.1] The sentence 'For the purposes of this paper, we filtered out all of the annotators who did not pass this attention check' would be clearer with the resulting number of annotators per example stated in the main text; the '19 to 22' figure appears only later in the same paragraph.
  2. [§4.1 / Table 2] Table 2 separates metric types into 'O', 'N', and 'T' but these category labels are not explained in the table caption; please define them, as the text refers to them without a direct mapping.
  3. [Appendix A, Table 4] The sign convention in Table 4 is confusing: for overlap metrics a negative correlation with 'Hallucination %' is good, but for span-level error types the same sign is interpreted as weak signal. Please add a sentence in the appendix explaining that negative correlations for coverage metrics mean 'more overlap is associated with fewer errors,' to prevent misreading.
  4. [§4.2] The reference for o3-mini is given only as a URL in a footnote, while GPT-4o and Gemma3 have full references; please add a citable reference or data sheet for o3-mini.
  5. [§5] The line 'Some LLMs pass attention checks Gemma3 and o3-mini both passed the attention check' is missing a colon or period after 'attention checks', which makes it read as a run-on sentence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline correlations are computed from independent human labels and fixed metric formulas, not from fitted or self-referential inputs.

full rationale

The paper's central claims are empirical comparisons between automatic metrics and human judgments. In Section 4.1, Spearman correlations are computed between fixed metric scores (word overlap, BLEU, ROUGE, NLI, embeddings) and the percentage of annotators who flagged a hallucination in Section 3.1; these labels come from external human annotation, and the metric formulas do not use those labels as inputs. The span-level correlations in Appendix Table 4 are likewise independent computations against human span annotations. No parameter is fitted to the target labels and then renamed as a prediction; the NLI and embedding models are pretrained externally and applied off-the-shelf. Self-citations such as Kasner et al. (2024) for the Factgenie annotation interface, Kasner et al. (2025) for the LLM-as-span-annotator methodology, and Schmidtova et al. (2024) for surveying common evaluation practice are contextual or tooling references, not load-bearing evidence for the paper's measured correlations. The acknowledged noise and low quality of the crowd span annotations (Section 3.2 and Section 7) is a data-quality limitation, not a circularity: the paper does not define its predictions in terms of those annotations. Thus the derivation chain is self-contained with respect to the input data, and no circular step can be exhibited from the paper's own equations or citation structure.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted and no new entities are posited. The central claims rest on the quality and interpretability of human annotations, which the paper itself shows to be fragile.

assumptions (3)
  • domain assumption The percentage of annotators who flagged a hallucination in the categorical data is a valid continuous signal of error presence.
    Used throughout Section 4.1 as the gold standard for Spearman correlations; subjectivity is intentional, but treating disagreement as a signal presupposes it is meaningful.
  • domain assumption Crowd span annotations, despite low attention-check pass rate and expert-identified false positives, remain informative enough to validate LLM-as-a-judge.
    Section 3.2 and Limitations: the authors explicitly say the annotations were used because in-house annotation was not feasible; this premise is load-bearing for the LLM comparison.
  • domain assumption The error typology (non-checkable, misleading, incorrect) is exhaustive and the distinction is reliably applicable by annotators.
    Span annotation campaign is built on these categories (Table 5); if annotators cannot apply them consistently, the agreement numbers and business impact estimates are hard to interpret.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Real-World Summarization: When Evaluation Reaches Its Limits." pith.science (2026). https://pith.science/paper/XMWG26CY

@misc{pith2026250711508,
  author       = {Pith},
  title        = {Pith review of: Real-World Summarization: When Evaluation Reaches Its Limits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XMWG26CY}},
  note         = {Machine review of arXiv:2507.11508}
}
read the original abstract

We examine evaluation of faithfulness to input data in the context of hotel highlights: brief LLM-generated summaries that capture unique features of accommodations. Through human evaluation campaigns involving categorical error assessment and span-level annotation, we compare traditional metrics, trainable methods, and LLM-as-a-judge approaches. Our findings reveal that simpler metrics like word overlap correlate surprisingly well with human judgments (Spearman correlation rank of 0.63), often outperforming more complex methods when applied to out-of-domain data. We further demonstrate that while LLMs can generate high-quality highlights, they prove unreliable for evaluation as they tend to severely under- or over-annotate. Our analysis of real-world business impacts shows incorrect and non-checkable information pose the greatest risks. We also highlight challenges in crowdsourced evaluations.

Figures

Figures reproduced from arXiv: 2507.11508 by the authors.

Figure 1
Figure 1. Distribution of error types based on human [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 6 canonical work pages

  1. [1]

    Miriam Ansch \"u tz, Diego Miguel Lozano, and Georg Groh. 2023. https://doi.org/10.18653/v1/2023.inlg-main.12 This is not correct! negation-aware evaluation of language generation systems . In Proceedings of the 16th International Natural Language Generation Conference, pages 163--175, Prague, Czechia. Association for Computational Linguistics

    doi:10.18653/v1/2023.inlg-main.12 2023

  2. [2]

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fern \' a ndez, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andr \' e F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K. Surikuchi, Ece Takmaz, and Alberto Testoni. ...

    arXiv doi:10.48550/arxiv.2406.18403 2024

  3. [3]

    Ond r ej Du s ek and Zden e k Kasner. 2020. https://doi.org/10.18653/v1/2020.inlg-1.19 Evaluating semantic accuracy of data-to-text generation with natural language inference . In Proceedings of the 13th International Conference on Natural Language Generation, pages 131--137, Dublin, Ireland. Association for Computational Linguistics

    doi:10.18653/v1/2020.inlg-1.19 2020

  4. [4]

    Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. https://doi.org/10.18653/v1/2022.acl-long.62 Language-agnostic BERT sentence embedding . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878--891, Dublin, Ireland. Association for Computational Linguistics

    doi:10.18653/v1/2022.acl-long.62 2022

  5. [5]

    Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2023. Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. Journal of Artificial Intelligence Research, 77:103--166

    2023

  6. [6]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024. https://doi.org/10.48550/ARXIV.2411.15594 A survey on LLM-as-a-Judge . CoRR, abs/2411.15594

    arXiv doi:10.48550/arxiv.2411.15594 2024

  7. [7]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. https://openreview.net/forum?id=XPZIaotutsD Deberta: Decoding-enhanced bert with disentangled attention . In International Conference on Learning Representations

    2021

  8. [8]

    Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. https://doi.org/10.5281/zenodo.1212303 spacy: Industrial-strength natural language processing in python . Zenodo

    doi:10.5281/zenodo.1212303 2020

Show all 27 references
  1. [9]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2023. http://arxiv.org/abs/2202.03629 Survey of Hallucination in Natural Language Generation . ACM Computing Surveys, 55(12):248. ArXiv: 2202.03629

    2023 arXiv

  2. [10]

    Srinivas Ramesh Kamath, Fahime Same, and Saad Mahamood. 2024. https://aclanthology.org/2024.inlg-main.23/ Generating hotel highlights from unstructured text using LLM s . In Proceedings of the 17th International Natural Language Generation Conference, pages 280--288, Tokyo, Ja...

    2024

  3. [11]

    Zden e k Kasner, Ondrej Platek, Patricia Schmidtova, Simone Balloccu, and Ondrej Dusek. 2024. https://aclanthology.org/2024.inlg-demos.5/ factgenie: A framework for span-based evaluation of generated texts . In Proceedings of the 17th International Natural Language Generation ...

    2024

  4. [12]

    Zdeněk Kasner, Vilém Zouhar, Patrícia Schmidtová, Ivan Kartáč, Kristýna Onderková, Ondřej Plátek, Dimitra Gkatzia, Saad Mahamood, Ondřej Dušek, and Simone Balloccu. 2025. https://arxiv.org/abs/2504.08697 Large language models as span annotators . Preprint, arXiv:2504.08697

    2025

  5. [13]

    Tom Kocmi, Vil \'e m Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popovi \'c , Mrinmaya Sachan, and Mariya Shmatova. 2024. https://doi.org/10.18653/v1/2024.wmt-1.131 Error span annotation: A balanced approach for human evaluation of machine transl...

    2024 doi

  6. [14]

    Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.29 Benchmarking cognitive biases in large language models as evaluators . In Findings of the Association for Computational Linguistics: ACL ...

    2024 doi

  7. [15]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  8. [16]

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. https://doi.org/10.18653/v1/2020.acl-main.173 On faithfulness and factuality in abstractive summarization . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1...

    2020 doi

  9. [17]

    Jekaterina Novikova, Ond r ej Du s ek, Amanda Cercas Curry, and Verena Rieser. 2017. https://doi.org/10.18653/v1/D17-1238 Why we need new evaluation metrics for NLG . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241--2252, C...

    2017 doi

  10. [18]

    Chatterji, Faisal Ladhak, and Tatsunori Hashimoto

    Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. 2024. https://openreview.net/forum?id=KS8mIvetg2 Proving Test Set Contamination in Black - Box Language Models . In ICLR , Vienna, Austria

    2024

  11. [19]

    Matt Post. 2018. https://doi.org/10.18653/v1/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Brussels, Belgium. Association for Computational Linguistics

    2018 doi

  12. [20]

    Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S. Yu. 2024. https://arxiv.org/abs/2405.12819 Large language models meet nlp: A survey . arXiv preprint arXiv:2405.12819

    2024 arXiv

  13. [21]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  14. [22]

    Howcroft, Ondrej Platek, and Adarsa Sivaprasad

    Patricia Schmidtova, Saad Mahamood, Simone Balloccu, Ondrej Dusek, Albert Gatt, Dimitra Gkatzia, David M. Howcroft, Ondrej Platek, and Adarsa Sivaprasad. 2024. https://aclanthology.org/2024.inlg-main.44/ Automatic metrics in natural language generation: A survey of current eva...

    2024

  15. [23]

    Damien Sileo. 2024. https://aclanthology.org/2024.lrec-main.1361/ tasksource: A large collection of NLP tasks with a structured dataset preprocessing framework . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Eval...

    2024

  16. [24]

    Gemma Team. 2025. https://arxiv.org/abs/2503.19786 Gemma 3 technical report . Preprint, arXiv:2503.19786

    2025 arXiv

  17. [25]

    OpenAI Team. 2024. https://arxiv.org/abs/2410.21276 Gpt-4o system card . Preprint, arXiv:2410.21276

    2024 arXiv

  18. [26]

    Dai, and Quoc V

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. https://openreview.net/forum?id=gEZrGCozdqR Finetuned Language Models are Zero - Shot Learners . Online

    2022

  19. [27]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.