Pith. sign in

REVIEW 3 major objections 5 minor 36 references

Facet-Aware Evaluation for Extractive Summarization

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Extractive summaries should be judged by semantic facet coverage, not token overlap; a new recall metric matches human ranking better than ROUGE.

desk verdict Useful metric and dataset, but the human-correlation claim is a 0.017 Spearman gap without significance testing—worth reviewing, not worth accepting as-is. read the letter →

arxiv 1908.10383 v2 pith:PMKT5XOU submitted 2019-08-27 cs.CL cs.IR

Yuning Mao , Liyuan Liu , Qi Zhu , Xiang Ren , Jiawei Han This is my paper · ORCID
classification cs.CLcs.IR
keywords extractivesummarizationevaluationmetricsfacet-awarerecallROUGEsupportsentencesinformationcoverageCNN/DailyMailsentenceregression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that information coverage in extractive summarization should be measured by whether the extracted sentences cover the semantic facets of the reference summary, not by token-level n-gram overlap. It defines each reference sentence as a facet, annotates which document sentences express each facet as support sentences grouped into support groups, and proposes Facet-Aware Recall (FAR), the fraction of facets for which at least one complete support group is covered by the extracted summary. To test this, the authors hand-annotate facet-aware mappings on a sample of CNN/Daily Mail and report that FAR correlates better with human rankings than ROUGE, supports fine-grained analysis by facet category, and can be approximated automatically via sentence regression. If these claims hold, summarization evaluation would move from rewarding superficial lexical similarity to rewarding genuine information coverage, and current benchmark results would need to be re-read in that light.

What carries the argument

The central object is the Facet-Aware Mapping (FAM): a relation from each reference sentence $r_i$, treated as one facet, to one or more support groups $S_i^j$, where each support group is a set of document sentences that together express that facet's semantics. The evaluation metric that runs on FAMs is Facet-Aware Recall (FAR), defined as the fraction of facets for which at least one complete support group is a subset of the extracted sentence set $E$. FAR counts a facet as covered exactly once, so it neither penalizes long extracted sentences that carry extra details nor rewards covering the same facet multiple times. A companion metric, Support-Aware Recall (SAR), merges all support sentences and measures the proportion of salient document sentences extracted, letting the framework separate salience from redundancy in comparative analysis.

What would settle it

Collect human preference rankings on a new set of document-summary pairs, especially pairs where the reference summary shares few words with the document, and compute the Spearman correlation of both FAR and ROUGE with those rankings; if FAR does not beat ROUGE, or if re-annotating the support sentences with different annotators changes which system FAR ranks first, the paper's central claim would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that an extractive summary should be scored by whether it conveys the reference summary's information, and that this can be operationalized at the sentence-facet level. The paper treats each reference sentence $r_i$ as a facet, annotates one or more support groups $S_i^j$ for it, where each support group is a set of document sentences that together express the facet's semantics, and defines Facet-Aware Recall as $FAR = \frac{1}{R}\sum_{i=1}^R \mathrm{Any}(I(S_i^1,E),\ldots,I(S_i^N,E))$, with $E$ the set of extracted sentences and $I(X,Y)=1$ if $X \subseteq Y$. The authors report that on their annotated extractive CNN/Daily Mail set, FAR ranks the three systems compared with human judges with Spearman $\rho = 0.457$, slightly above ROUGE's $0.44$, and the system FAR ranks first also receives the most first-place human ranks. They also show that facet-aware evaluation can be broken down by facet category, exposing that both extractive and abstractive systems perform markedly worse on noisy and high-abstraction facets, and that machine-created support mappings built by sentence regression correlate strongly with human annotations for system ranking, making automatic large-scale facet-aware evaluation feasible.

Load-bearing premise

The argument rests on the assumption that each reference sentence is one self-contained facet and that the annotated support sentences fully capture that facet's semantics; if the human annotations are incomplete or inconsistent, the FAR scores and the claimed correlation with human judgment would not hold.

Editorial extensions

If this is right

  • On CNN/Daily Mail, choosing FAR instead of ROUGE changes which extractive system is ranked best: UnifiedSum(E) leads under FAR, while ROUGE-1 F1 gives almost no discrimination among four compared methods.
  • Because FAR treats every facet equally and only checks whether a facet is covered, extractive systems are no longer penalized for outputting long sentences that add detail beyond the facet's core meaning.
  • Facet-category analysis provides a diagnostic: all tested extractive and abstractive systems score far worse on high-abstraction and noisy facets, suggesting that current strong performance on CNN/Daily Mail is largely driven by low-abstraction, copyable content.
  • Machine-created FAMs via sentence regression, especially when three support groups are allowed, correlate strongly with human-annotated FAMs in system ranking, so facet-aware evaluation can be scaled to the full test set without additional human annotation.
  • Because FAMs allow multiple alternative support groups for the same facet, the evaluation setup is ready to extend to multi-document summarization, where the same facet may be covered by redundant sentences across documents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely unstated consequence is that leaderboards built on ROUGE may have rewarded systems for high lexical overlap without verifying semantic coverage; switching to facet-aware metrics could change model selection and even training objectives, for example by training extractive models to maximize expected FAR rather than ROUGE.
  • On datasets where summaries are more abstractive, or where documents and summaries share fewer words, the correlation advantage of FAR over ROUGE should be larger than the observed $0.457$ vs $0.44$, because token overlap is even less informative there; annotating FAMs on such a dataset would test this.
  • The sentence-level facet definition likely undercounts sub-sentence semantic units; splitting facets below the sentence level, as Pyramid does with content units, might match human judgment even better at the cost of more annotation effort.
  • The reported pairwise Jaccard agreement of $0.714$ on support-sentence annotations suggests the ground truth is not perfectly stable; a sensitivity analysis that perturbs support sets and measures how stable FAR's system rankings are would quantify how much annotation noise the metric tolerates.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a facet-aware evaluation setup for extractive summarization. Each reference summary sentence is treated as a facet, human annotators mark document sentences that support each facet, and the Facet-Aware Recall (FAR) metric measures how many facets an extracted summary covers by comparing extracted sentence indices with these support groups. The authors construct a 150-sample extractive CNN/Daily Mail benchmark with Facet-Aware Mappings (FAMs), evaluate six extractive systems under FAR, ROUGE, and Support-Aware Recall, report a human ranking study claiming that FAR correlates better with human judgment than ROUGE, and investigate automatic FAM creation via sentence regression and a linear-regression extension (AutoFAR-L). The paper also provides fine-grained analysis by category (noise, low abstraction, high abstraction) and comparative analysis of extractive systems.

Significance. If the central claims hold, the paper makes a useful contribution: it releases a new annotation resource for extractive summarization, proposes a simple index-matching metric that is interpretable and cheap to compute once FAMs exist, and provides category-level analysis that could inform future dataset curation. The concrete strengths are the public release of the annotated data, the clear formalization of FAR/SAR, and the candid caveats about AutoFAR-L. However, the headline claim of better human correlation than ROUGE rests on a very small evaluation, and the annotation agreement is moderate; these issues affect the main scientific contribution, not just the presentation.

major comments (3)
  1. [§3.2, Table 4] The abstract's claim that facet-aware evaluation 'manifests better correlation with human judgment than ROUGE' rests on a single Spearman's rho comparison of 0.457 versus 0.44. The observed difference is 0.017, no confidence interval, bootstrap, or significance test is reported, and the text does not state which ROUGE variant is used. Since only three extractive systems were ranked by human annotators, the rank space is extremely coarse and a difference of 0.017 cannot be distinguished from noise. This evidence is insufficient to support the headline claim in its current strong form. Please add uncertainty quantification and a clearly stated ROUGE variant, or rephrase the claim as suggestive/preliminary.
  2. [§2.2, Table 2; §3.2] The validity of FAR depends on reliable and complete annotation of support sentences and support groups. The reported pairwise Jaccard index of 0.714 is moderate, and the main FAR comparison in §3.2 is computed only over the low-abstraction category, excluding noisy and high-abstraction facets. The paper does not analyze how annotation disagreement or the exclusion of these categories affects system-level FAR rankings. Please report agreement on support-group boundaries (not only overall sentence-level overlap), and include a sensitivity analysis of the system ranking under FAR on alternative annotation subsets or with relaxed support-group matching.
  3. [§4.2, Table 8] The AutoFAR-L extension fits a linear regression to ground-truth FAR on the small annotated set and then applies it to predict FAR on the entire CNN/Daily Mail test set. The text appropriately cautions that the reliability of this extrapolation is not guaranteed; however, this caution does not repair the insufficient evidence for the primary human-correlation claim in §3.2. The large-scale system ranking under AutoFAR-L should be presented only as an unvalidated pilot, and the abstract's wording should not imply fully automatic validation of FAR without that caveat.
minor comments (5)
  1. [§3.2, Table 4] Please specify which ROUGE measure (ROUGE-1, ROUGE-2, ROUGE-L, and which F1/recall/precision variant) was used for the correlation with human rankings.
  2. [§2.2] The pairwise Jaccard index of 0.714 is described as 'high agreement'; please report the number of annotators per sample, the range of agreement values, and whether the final consensus labels were produced jointly or by majority vote.
  3. [§3.3, Table 5] The fine-grained analysis in Table 5 uses ROUGE-1 F1 rather than FAR; the text should clarify that the category breakdown comes from annotated FAMs, while the metric used in the table is still ROUGE.
  4. [§4.2, Table 8] The parenthetical values in columns 'Pearson's r', 'Spearman's ρ', and 'Kendall's τ' (e.g., '97.6 (42.9)') are not clearly labeled; please separate the fit on the human-annotated subset from the extrapolated AutoFAR-L results.
  5. [§2.2, Table 6] The statement that FAM annotation was 'easier than previously believed' should be reconciled with the low recall of sentence regression approaches in Table 6; human effort and automatic support-sentence discovery are distinct and the contrast should be stated explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central correlation claim rests on an independent human ranking, not on the FAM annotations by construction.

full rationale

The paper's central claim is that FAR correlates better with human judgment than ROUGE. FAR is defined from human-annotated facet-aware mappings (FAMs), while the validation in Sec. 3.2 uses a separate human ranking of three extractive systems (Table 4). The Spearman comparison (0.457 vs 0.44) is an empirical benchmark result, not a quantity forced by FAR's definition. The FAMs are human labels, not fitted parameters; the metric is evaluated, not derived from the evaluation. The AutoFAR-L portion (Sec. 4.2) does fit a linear regressor to ground-truth FAR, but the paper explicitly labels it a pilot study and cautions that 'the reliability of such extrapolation is not guaranteed', so the extrapolated scores are not used as load-bearing evidence for the headline claim. The only self-citation (Gu et al., 2020) appears in a related-work remark about ROUGE usage and is not load-bearing. No uniqueness theorem is invoked, and no ansatz is smuggled in via citation. The weakness of the human-correlation claim is statistical (a 0.017 gap with no significance test on a three-system ranking), which is a correctness risk, not circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 2 invented entities

The central claim rests on the facet annotation scheme and the representativeness of the annotated sample. No physical constants or external parameters are involved; the only fitted quantity is the selected number of support groups in the automatic FAM creation experiment.

free parameters (1)
  • Number of support groups N for machine-created FAMs = 3
    Selected post hoc in Table 7 as the value that maximizes correlation with human-annotated FAR; not justified independently of the data.
assumptions (4)
  • domain assumption Each reference sentence corresponds to exactly one facet.
    Sec 2.1 hypothesizes that each reference sentence ri corresponds to one facet, which is the basis for FAM annotation and FAR calculation.
  • domain assumption Support groups fully cover the semantics of their facet; extracting any one support group suffices for facet coverage.
    Sec 2.1 defines support groups as sets of document sentences that fully cover the facet information, and FAR uses this to determine coverage.
  • domain assumption Human annotations are of sufficient quality as ground truth despite moderate inter-annotator agreement.
    Sec 2.2 reports pairwise Jaccard index 0.714, and the paper proceeds to use these FAMs as ground truth without further validation.
  • domain assumption The 150 annotated document-summary pairs are representative of the entire CNN/Daily Mail test set.
    Sec 2.2 samples from the test set and the paper generalizes conclusions to the whole dataset, especially in Sec 4.2.
invented entities (2)
  • Facet
    purpose: A semantic unit in the reference summary, used as the target of coverage evaluation.
    A conceptual construct defined by the paper; no external falsifiable handle.
  • Support group
    purpose: A set of document sentences that together cover a facet.
    Annotation construct; no external evidence outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Facet-Aware Evaluation for Extractive Summarization." pith.science (2026). https://pith.science/paper/PMKT5XOU

@misc{pith2026190810383,
  author       = {Pith},
  title        = {Pith review of: Facet-Aware Evaluation for Extractive Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PMKT5XOU}},
  note         = {Machine review of arXiv:1908.10383}
}
read the original abstract

Commonly adopted metrics for extractive summarization focus on lexical overlap at the token level. In this paper, we present a facet-aware evaluation setup for better assessment of the information coverage in extracted summaries. Specifically, we treat each sentence in the reference summary as a \textit{facet}, identify the sentences in the document that express the semantics of each facet as \textit{support sentences} of the facet, and automatically evaluate extractive summarization methods by comparing the indices of extracted sentences and support sentences of all the facets in the reference summary. To facilitate this new evaluation setup, we construct an extractive version of the CNN/Daily Mail dataset and perform a thorough quantitative investigation, through which we demonstrate that facet-aware evaluation manifests better correlation with human judgment than ROUGE, enables fine-grained evaluation as well as comparative analysis, and reveals valuable insights of state-of-the-art summarization methods. Data can be found at https://github.com/morningmoni/FAR.

Figures

Figures reproduced from arXiv: 1908.10383 by the authors.

Figure 1
Figure 1. An illustration of facet-aware evaluation. Two of three support groups of facet 1 (r1) are covered. Facet 2 (r2) cannot be covered as document sentence 4 (d4) is missing in the extracted summary. The illustra￾tion corresponds to the example in Sec. 3.1. acteristics of the CNN/Daily Mail dataset. 2 Dataset Creation In this section, we describe the process of creat￾ing an extractive summarization dataset to facili￾tat… view at source ↗
Figure 2
Figure 2. Performance of extractive methods under ROUGE, FAR, and SAR. The results under ROUGE-1/2/L often disagree with each other. UnifiedSum(E) generally performs the best in the facet-aware evaluation. 0.34 0.35 0.36 0.37 0.38 0.39 0.40 0.41 SAR 0.46 0.48 0.50 0.52 0.54 FAR Lead-3 FastRL(E) BanditSum NeuSum Refresh UnifiedSum(E) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Comparison of extractive methods under FAR and SAR reflects their capability of extracting salient and non-redundant sentences. ically transform abstractive reference summaries to extractive labels heuristically using ROUGE. Previ￾ously, one could only estimate the quality of these labels by evaluating the extractive models trained using such labels, i.e., comparing their extracted summaries with the reference summa… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of summarization metrics. Support sentences are marked in the same color as their corresponding facets. SCUs have to be annotated for each extracted summary during evaluation, while facet￾aware evaluation can be conducted automatically by comparing sentence …
Figure 5
Figure 5. Figure 5: The first three figures show the ground-truth and estimated FAR scores via human-annotated FAMs and [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 19 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. https://doi.org/10.18653/v1/P16-1223 A thorough examination of the CNN /daily mail reading comprehension task . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2358--2367, Berlin, Germany. Association for Computational Linguistics

    work page doi:10.18653/v1/p16-1223 2016

  4. [4]

    Yen-Chun Chen and Mohit Bansal. 2018. https://doi.org/10.18653/v1/P18-1063 Fast abstractive summarization with reinforce-selected sentence rewriting . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675--686, Melbourne, Australia. Association for Computational Linguistics

    doi:10.18653/v1/p18-1063 2018

  5. [5]

    Hoa Trang Dang and Karolina Owczarzak. 2008. Overview of the tac 2008 update summarization task. In TAC

    work page 2008

  6. [6]

    Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. https://doi.org/10.18653/v1/D18-1409 B andit S um: Extractive summarization as a contextual bandit . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3739--3748, Brussels, Belgium. Association for Computational Linguistics

    doi:10.18653/v1/d18-1409 2018

  7. [7]

    Kavita Ganesan. 2018. Rouge 2.0: Updated and improved measures for evaluation of summarization tasks. arXiv preprint arXiv:1803.01937

    arXiv 2018

  8. [8]

    Xiaotao Gu, Yuning Mao, Jiawei Han, Jialu Liu, Hongkun Yu, You Wu, Cong Yu, Daniel Finnie, Jiaqi Zhai, and Nicholas Zukoski. 2020. Generating representative headlines for news stories. WWW

    work page 2020

Show all 36 references
  1. [9]

    Hardy Hardy, Shashi Narayan, and Andreas Vlachos. 2019. https://doi.org/10.18653/v1/P19-1330 H igh RES : Highlight-based reference-less evaluation of summarization . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3381--3392, F...

    2019 doi

  2. [10]

    Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. https://doi.org/10.18653/v1/P18-1013 A unified model for extractive and abstractive summarization using inconsistency loss . In Proceedings of the 56th Annual Meeting of the Association for Co...

    2018 doi

  3. [11]

    Taehee Jung, Dongyeop Kang, Lucas Mentch, and Eduard Hovy. 2019. https://doi.org/10.18653/v1/D19-1327 Earlier isn ' t always better: Sub-aspect analysis on corpus and system biases in summarization . In Proceedings of the 2019 Conference on Empirical Methods in Natural Languag...

    2019 doi

  4. [12]

    Chris Kedzie, Kathleen McKeown, and Hal Daum \'e III. 2018. https://doi.org/10.18653/v1/D18-1208 Content selection in deep learning models of summarization . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1818--1828, Brussels, ...

    2018 doi

  5. [13]

    Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. https://doi.org/10.18653/v1/D19-1051 Neural text summarization: A critical evaluation . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing ...

    2019 doi

  6. [14]

    Chin-Yew Lin. 2004. https://www.aclweb.org/anthology/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  7. [15]

    Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI, pages 3075--3081

    2017

  8. [16]

    Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, C a g lar Gu̇l c ehre, and Bing Xiang. 2016. https://doi.org/10.18653/v1/K16-1028 Abstractive text summarization using sequence-to-sequence RNN s and beyond . In Proceedings of The 20th SIGNLL Conference on Computational Natural...

    2016 doi

  9. [17]

    Cohen, Mirella Lapata, Jiangsheng Yu, and Yi Chang

    Shashi Narayan, Ronald Cardenas, Nikos Papasarantopoulos, Shay B. Cohen, Mirella Lapata, Jiangsheng Yu, and Yi Chang. 2018 a . https://doi.org/10.18653/v1/P18-1188 Document modeling with external attention for sentence extraction . In Proceedings of the 56th Annual Meeting of ...

    2018 doi

  10. [18]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018 b . https://doi.org/10.18653/v1/D18-1206 Don ' t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization . In Proceedings of the 2018 Conference on Empirical Methods in N...

    2018 doi

  11. [19]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018 c . https://doi.org/10.18653/v1/N18-1158 Ranking sentences for extractive summarization with reinforcement learning . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computation...

    2018 doi

  12. [20]

    Ani Nenkova and Rebecca Passonneau. 2004. https://www.aclweb.org/anthology/N04-1019 Evaluating content selection in summarization: The pyramid method . In Proceedings of the Human Language Technology Conference of the North A merican Chapter of the Association for Computationa...

    2004

  13. [21]

    Jun-Ping Ng and Viktoria Abrecht. 2015. https://doi.org/10.18653/v1/D15-1222 Better summarization evaluation with word embeddings for ROUGE . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1925--1930, Lisbon, Portugal. Associat...

    2015 doi

  14. [22]

    Over Paul and Yen James. 2004. An introduction to duc-2004. In Proceedings of the 4th Document Understanding Conference (DUC 2004)

    2004

  15. [23]

    Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304

    2017 arXiv

  16. [24]

    Natalie Schluter. 2017. https://www.aclweb.org/anthology/E17-2007 The limits of automatic summarisation according to ROUGE . In Proceedings of the 15th Conference of the E uropean Chapter of the Association for Computational Linguistics: Volume 2, Short Papers , pages 41--45, ...

    2017

  17. [25]

    Liu, and Christopher D

    Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. https://doi.org/10.18653/v1/P17-1099 Get to the point: Summarization with pointer-generator networks . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2017 doi

  18. [26]

    Elaheh ShafieiBavani, Mohammad Ebrahimi, Raymond Wong, and Fang Chen. 2018. https://doi.org/10.18653/v1/D18-1085 A graph-theoretic summary evaluation for ROUGE . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 762--767, Brussels...

    2018 doi

  19. [27]

    Ori Shapira, David Gabay, Hadar Ronen, Judit Bar-Ilan, Yael Amsterdamer, Ani Nenkova, and Ido Dagan. 2018. https://doi.org/10.18653/v1/D18-1087 Evaluating multiple system summary lengths: A case study . In Proceedings of the 2018 Conference on Empirical Methods in Natural Lang...

    2018 doi

  20. [28]

    Simeng Sun and Ani Nenkova. 2019. https://doi.org/10.18653/v1/D19-1116 The feasibility of embedding based automatic evaluation for single document summarization . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internationa...

    2019 doi

  21. [29]

    Stratos Xenouleas, Prodromos Malakasiotis, Marianna Apidianaki, and Ion Androutsopoulos. 2019. https://doi.org/10.18653/v1/D19-1618 SUM - QE : a BERT -based summary quality estimation model . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Proces...

    2019 doi

  22. [30]

    Qian Yang, Rebecca J Passonneau, and Gerard De Melo. 2016. Peak: Pyramid evaluation via automated knowledge extraction. In Thirtieth AAAI Conference on Artificial Intelligence

    2016

  23. [31]

    Semih Yavuz, Izzeddin Gur, Yu Su, and Xifeng Yan. 2018. https://doi.org/10.18653/v1/D18-1197 What it takes to achieve 100 \ In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1702--1711, Brussels, Belgium. Association for Computati...

    2018 doi

  24. [32]

    Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2019 a . Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint arXiv:1912.08777

    2019 arXiv

  25. [33]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019 b . Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  26. [34]

    Meyer, and Steffen Eger

    Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. https://doi.org/10.18653/v1/D19-1053 M over S core: Text generation evaluating with contextualized embeddings and earth mover distance . In Proceedings of the 2019 Conference on Empirical ...

    2019 doi

  27. [35]

    Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. https://doi.org/10.18653/v1/P18-1061 Neural document summarization by jointly learning to score and select sentences . In Proceedings of the 56th Annual Meeting of the Association for Computation...

    2018 doi

  28. [36]

    Markus Zopf, Eneldo Loza Menc \' a, and Johannes F \"u rnkranz. 2018. https://doi.org/10.18653/v1/N18-1161 Which scores to predict in sentence regression for text summarization? In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computa...

    2018 doi

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.