Pith. sign in

REVIEW 4 major objections 5 minor 23 references

Evaluate Summarization in Fine-Granularity: Auto Evaluation with LLM

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that summarization quality is best measured by scoring individual facts, and that the resulting entity-level scores agree with human experts more closely than n-gram, embedding, or holistic LLM baselines.

desk verdict Entity-level evaluation framework is plausible, but the SummEval evidence is built on a filtered subset and post-hoc outlier removal, making the headline claim of better human correlation circular. read the letter →

arxiv 2412.19906 v1 pith:YAREQGD7 submitted 2024-12-27 cs.CL cs.AI

Dong Yuan , Eti Rastogi , Fen Zhao , Sagar Goyal , Gautam Naik , Sree Prasanna Rajagopal This is my paper · ORCID
classification cs.CLcs.AI
keywords summarizationevaluationLLM-as-judgeentity-levelmetricscompletenesscorrectnessalignmentreadabilitymedicalnote
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SumAutoEval aims to replace subjective 1–5 summary ratings with objective, interpretable scores computed at the level of individual facts. The paper's proposal is to break a ground-truth summary into atomic entities, check each entity against the candidate summary with several LLM prompts, and aggregate the checks into four ratios: completeness, correctness, alignment, and readability. The evidence is that on a 30-note medical dataset the completeness score agrees with expert labels at Cohen's kappa 0.81, and on a curated SummEval subset the method beats G-Eval on consistency and coherence after outlier removal. If the result holds, a summary score stops being an opaque number and becomes an audit trail of exactly which facts are missing, wrong, misplaced, or poorly written.

What carries the argument

The load-bearing object is the entity, defined as a short phrase with exactly one key concept. The pipeline extracts entities from both the ground-truth and target notes in three steps — extraction, self-verification, and reference sourcing — and then runs three differently-phrased prompts per entity, with majority voting and partial credit (0.5 for 'partial', 1 for 'yes'). Per-entity answers are aggregated into the ratio scores, so every dimension reduces to counts over entities. This is what makes the evaluation auditable: a low completeness score can be traced to specific missing ground-truth entities.

What would settle it

A decisive check is to recompute Table 3 on the full SummEval data with no outlier deletion: the paper states that roughly ten flagged examples shift correlations by 20–23%, so the consistency advantage (0.70 vs 0.62) surviving that recomputation would settle the claim.

Watch

Extended reading notes

Core claim

The paper claims that summarization quality is measurable as ratios over atomic entities. With $\text{Completeness} = \frac{\text{total entity score in target}}{\#\text{ground-truth entities}}$ and $\text{Correctness} = \frac{\text{total entity score in target}}{\#\text{target entities}}$, each dimension becomes a count that can be audited. Alignment is the share of target entities placed under the correct section, and readability is the share of sentences without writing issues. Using GPT-4o as the backend, the framework reaches Cohen's kappa 0.81 against expert entity labels for completeness on 30 medical notes, and on the curated SummEval subset it reaches Spearman 0.70 for consistency and 0.75 for coherence after removing erroneous expert scores, surpassing G-Eval's 0.62 and 0.60. The author's intended conclusion is that fine-grained entity-level evaluation gives a more understandable and more human-aligned assessment than holistic scores.

Load-bearing premise

The entire method depends on the automatic fact-pulling step being accurate and complete: if the LLM merges, splits, or misses a key fact, every score changes even when the summaries themselves are the same.

Editorial extensions

If this is right

  • Each score can be decomposed: a low completeness score names the specific ground-truth entities that the summary omitted.
  • Correctness and alignment scores identify the exact entities that are unsupported or placed in the wrong section, turning hallucination detection into a checklist.
  • Because the prompts ask for step-by-step reasoning and use majority voting, the scores are less sensitive to prompt tone than holistic LLM ratings, which the paper argues are easily manipulated.
  • In the medical setting, the method gives section-aware quality control, penalizing information placed under the wrong heading.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An unstated but natural product extension is an evidence-highlighting view: the reference-sourcing step already returns the supporting phrase from the source, so a clinician-facing tool could display why each entity was scored as present, missing, or contradicted.
  • Because the four scores are unit-free ratios over entity sets, the framework should transfer to other structured summarization domains, such as radiology or discharge summaries, wherever a reliable entity extraction step can be built.
  • The paper's own readability results—strong agreement on the SummEval subset with outlier removal, weak on whole-note medical flow—point to a testable extension: score readability at the passage or transition level rather than per sentence, and compare correlation with human judgments again.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SumAutoEval, an LLM-based summarization evaluation method that decomposes summaries into atomic entities, then computes four objective scores (completeness, correctness, alignment, readability) from entity-level judgments obtained via multiple prompts. The method is evaluated on a 30-note medical summarization dataset and on a modified SummEval subset (SummEval†), with correlations against human expert scores compared to ROUGE, BARTScore, and G-Eval. The authors claim that SumAutoEval gives more interpretable and objective scores and empirically achieves better human correlation.

Significance. If validated, SumAutoEval would be a useful interpretable alternative to holistic LLM evaluators, particularly for medical note summarization where missing details and hallucinations matter. The design choices of entity-level scoring, multiple verification prompts, and explicit partial/contradict categories are reasonable. However, the current evidence does not support the headline claim. The SummEval comparison is compromised by a non-standard subset and post-hoc outlier exclusion, and the entity-extraction layer, which is the foundation of all scores, is never validated against human annotations. The paper's own medical results show lower readability correlation than G-Eval, so the claim of 'better human correlation' is overstated. The method may have promise, but it requires a properly controlled evaluation and independent validation of the entity extraction.

major comments (4)
  1. [§4.3, Table 3] The SummEval† subset is constructed by selecting, for each source article, the machine summary whose expert scores were most consistent. This changes the test distribution, so published G-Eval scores computed on the full SummEval dataset are not directly comparable. The paper does not state that G-Eval was re-run on SummEval†, and therefore the reported wins in consistency (0.70 vs. 0.62) and coherence (0.75 vs. 0.60) do not establish an advantage over G-Eval.
  2. [§4.3.1] The paper removes 'at least 10 instances' per dimension because the expert scores were deemed inaccurate after analysis that appears to be motivated by autoEval scores. The Appendix examples (A.1–A.4) show expert averages of 4 or 5 with autoEval scores of 1–2.6, and these are used as evidence of expert error. Removing human labels that disagree with the metric under evaluation is circular and inflates reported correlations; the paper states these exclusions improve correlations by 20–23%. Correlations without outlier removal must be reported, and outlier criteria must be pre-specified or independently adjudicated.
  3. [§3.1, Eqs. (1)–(2)] All four scores are ratios over entity sets extracted by GPT-4-family models, but the extraction step is not validated against human entity annotations. No analysis measures how entity merging, splitting, or omission affects completeness and correctness. Because the same model family performs both extraction and entity-level classification, the high agreement in Table 1 may reflect model self-consistency rather than fidelity to the true information content. Human entity annotations or an independent extraction model are needed to support the claim that the scores are objective.
  4. [§4.2, Table 2] On the 30-note medical dataset, SumAutoEval's readability correlation (ρ=0.05) is far below G-Eval (0.27), and its organization correlation is only marginally higher (0.98 vs. 0.92). The abstract claims 'better human correlation' without qualification. The paper should either narrow the claim to the dimensions where the method wins or provide an explanation for the readability failure.
minor comments (5)
  1. [Introduction] In the Introduction, the text contains a garbled fragment 'hard tYesah o identify'; please fix this typo.
  2. [§3.1] Section 3.1 defines 'Entity' informally; please include a worked extraction example and a description of how the reference sourcing step handles unsupported entities.
  3. [Table 2] Table 2 labels a column 'G-Eval*' but the asterisk is never explained in the caption or text.
  4. [§3.2.1] The aggregation rules in Section 3.2.1 ('partial + partial + yes' -> yes; no agreement -> no) are ad hoc; please justify them or provide a sensitivity analysis.
  5. [References] Some references contain encoding artifacts (e.g., 'Kry´sci´nski') and inconsistent capitalization (e.g., 'ROUGE' vs. 'Rouge'); please standardize.

Circularity Check

1 steps flagged · score 6.0 of 10

SummEval 'better human correlation' is obtained by letting SumAutoEval veto disagreeing human scores as outliers; the reported correlations are partly self-agreement.

  1. fitted input called prediction [Section 4.3 (Metrics) and 4.3.1 (Data Error Analysis), Table 3]
    "We also report the scores after excluding the outliers where we observed that the human score was erroneous as described in the Data Error Analysis section below. ... Coherence We observed a pattern similar to what we found in relevance assessment: a few data samples had ratings that did not match the autoEval results. ... Notably, a small number of data points (around 10) were sufficient to reduce the correlation by 20%."

    The reference labels used to validate SumAutoEval are filtered by SumAutoEval itself. The paper reports correlations with and without outliers, where the outliers are precisely the expert scores that 'did not match the autoEval results'; the appendix examples (A.1, A.2, A.4) show expert ratings of 4 or 5 discarded because AutoEvalScore was 2, 1, or 2.6. Computing 'Ours -w/o outliers' on the remaining points therefore removes the disagreements that could lower the correlation, so the 0.70 consistency and 0.75 coherence numbers are partly a self-agreement artifact rather than evidence of alignment with a fixed human gold standard. G-Eval's w/o-outliers row is also evaluated on exclusions chosen by SumAutoEval, not by an independent criterion.

full rationale

The medical-note evaluation (Tables 1-2) provides some external grounding: Table 1 compares SumAutoEval's entity-level judgments against human entity labels using Cohen's Kappa without the outlier veto used on SummEval, and Table 2 honestly reports a low readability correlation (0.05) that does not favor the method. So the paper is not wholly circular. However, the abstract's 'better human correlation' claim is most strongly supported by Table 3 on SummEval, and that support is partially circular: the metric is used to declare disagreeing human expert scores erroneous, exclude them, and then the remaining subset is used to compute the headline correlations. The additional SummEval† subset selection ('chose a machine summary ... where expert evaluations were more consistent') is a data-selection concern rather than a circular reduction, but it compounds the difficulty of comparing against published G-Eval numbers. Overall, the central comparative claim over G-Eval reduces in part to a self-selected evaluation set, so a moderate circularity score of 6 is appropriate.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The framework rests on the validity of LLM-based entity extraction and LLM judgments. Several hand-set parameters (aggregation rules, partial weight, rubric frequency threshold, outlier removal) directly influence the reported correlations and are not tested for sensitivity.

free parameters (5)
  • Consensus aggregation exception rules = 'partial + partial + yes' -> yes; no agreement -> no
    Hand-set rules in Section 3.2.1 that determine entity-level labels and therefore all downstream scores; no ablation or validation is provided.
  • Partial entity weight = 0.5
    Partial responses are awarded 0.5 points in the entity score (Section 3.2.1), an arbitrary choice affecting completeness/correctness ratios.
  • Rubric entity frequency threshold = entities present in at least 5 of the human summaries
    This filter is used to define the SummEval rubric (Section 4.3) and changes which information counts as relevant; no sensitivity analysis is reported.
  • Subset selection for SummEval = only summaries with consistent expert scores are kept
    Section 4.3: 'we chose a machine summary... where expert evaluations were more consistent,' removing the noisiest comparisons before computing correlations.
  • Outlier exclusion count = ~10 examples per task
    Section 4.3.1 removes examples where human scores are deemed erroneous; the paper says these were 'substantial enough to reduce our metrics by 23%.'
assumptions (4)
  • domain assumption Summaries decompose into discrete, single-concept entities that are faithfully extractable by LLMs.
    Section 3.1 defines an Entity and assumes the three-step extraction captures the salient content; no human inter-annotator agreement on entity extraction is reported.
  • domain assumption LLM judgments on entity presence, correctness, and alignment are reliable proxies for truth.
    The whole metric is computed from LLM answers to prompts in Section 3.2; the LLM's own annotation consistency is not independently verified.
  • domain assumption Human-curated ground truth notes are accurate and complete enough to serve as the reference.
    Section 2 defines the ground truth note as 'accurate, detailed, well-organized, and curated by expert humans,' but Section 4.3.1 later concedes human labels contain errors.
  • standard math Cohen's Kappa, Spearman's rho, and Kendall tau statistics are appropriate for the agreement comparison.
    Used in Section 4 without derivation; standard statistical tools.
invented entities (1)
  • Entity (atomic information unit)
    purpose: A unit of information used to decompose summaries into comparable pieces for completeness, correctness, and alignment scoring.
    The paper defines the entity in Section 3.1 but provides no external validation that entities are stable, unique, or aligned with human intuition; extraction quality is not measured separately.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluate Summarization in Fine-Granularity: Auto Evaluation with LLM." pith.science (2026). https://pith.science/paper/YAREQGD7

@misc{pith2026241219906,
  author       = {Pith},
  title        = {Pith review of: Evaluate Summarization in Fine-Granularity: Auto Evaluation with LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YAREQGD7}},
  note         = {Machine review of arXiv:2412.19906}
}
read the original abstract

Due to the exponential growth of information and the need for efficient information consumption the task of summarization has gained paramount importance. Evaluating summarization accurately and objectively presents significant challenges, particularly when dealing with long and unstructured texts rich in content. Existing methods, such as ROUGE (Lin, 2004) and embedding similarities, often yield scores that have low correlation with human judgements and are also not intuitively understandable, making it difficult to gauge the true quality of the summaries. LLMs can mimic human in giving subjective reviews but subjective scores are hard to interpret and justify. They can be easily manipulated by altering the models and the tones of the prompts. In this paper, we introduce a novel evaluation methodology and tooling designed to address these challenges, providing a more comprehensive, accurate and interpretable assessment of summarization outputs. Our method (SumAutoEval) proposes and evaluates metrics at varying granularity levels, giving objective scores on 4 key dimensions such as completeness, correctness, Alignment and readability. We empirically demonstrate, that SumAutoEval enhances the understanding of output quality with better human correlation.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 2 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201

    arXiv 2023

  4. [4]

    Sihao Chen, Hongming Zhang, Tong Chen, Ben Zhou, Wenhao Yu, Dian Yu, Baolin Peng, Hongwei Wang, Dan Roth, and Dong Yu. 2023. http://arxiv.org/abs/2311.04335 Sub-sentence encoder: Contrastive learning of propositional semantic representations

    work page Pith review arXiv 2023

  5. [5]

    Alexander R Fabbri, Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391--409

    2021

  6. [6]

    Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, and Xiaojun Wan. 2024. Llm-based nlg evaluation: Current status and challenges. arXiv preprint arXiv:2402.01383

    arXiv 2024

  7. [7]

    Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. http://arxiv.org/abs/2304.02554 Human-like summarization evaluation with chatgpt

    arXiv 2023

  8. [8]

    Wojciech Kry \'s ci \'n ski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. arXiv preprint arXiv:1908.08960

    arXiv 2019

Show all 23 references
  1. [9]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  2. [10]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  3. [11]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251

    2023 arXiv

  4. [12]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  5. [13]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023 a . http://arxiv.org/abs/2303.04048 Is chatgpt a good nlg evaluator? a preliminary study

    2023 arXiv

  6. [14]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023 b . http://arxiv.org/abs/2305.17926 Large language models are not fair evaluators

    2023 arXiv

  7. [15]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

  8. [16]

    Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, et al. 2024. Long-form factuality in large language models. arXiv preprint arXiv:2403.18802

    2024 arXiv

  9. [17]

    Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A critical evaluation of evaluations for long-form question answering. arXiv preprint arXiv:2305.18201

    2023 arXiv

  10. [18]

    Seonghyeon Ye, Yongrae Jo, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, and Minjoon Seo. 2023. https://kaistai.github.io/SelFee/ Selfee: Iterative self-revising llm empowered by self-feedback generation . Blog post

    2023

  11. [19]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263--27277

    2021

  12. [20]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. http://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert

    2020 arXiv

  13. [21]

    Meyer, and Steffen Eger

    Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. http://arxiv.org/abs/1909.02622 Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance

    2019 arXiv

  14. [22]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. http://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena

    2023 arXiv

  15. [23]

    Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi-dimensional evaluator for text generation. arXiv preprint arXiv:2210.07197

    2022 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.