Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Better Rewards Yield Better Summaries: Learning to Summarise Without References

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Human-rated rewards beat ROUGE for training summarisers.

desk verdict Learning a reference-free reward from human ratings genuinely works for RL summarization, though the human evals are small and the abstract oversells the supervised comparison. read the letter →

arxiv 1909.01214 v1 pith:474AIVG2 submitted 2019-09-03 cs.CL

Florian Böhm , Yang Gao , Christian M. Meyer , Ori Shapira , Ido Dagan , Iryna Gurevych This is my paper · ORCID
classification cs.CL
keywords reinforcementlearningtextsummarizationrewardhumanevaluationreference-freeROUGEBERTpreference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard practice of using ROUGE as the reward signal in reinforcement-learning summarisers is flawed, because ROUGE's correlation with human judgement at the level of individual summaries is weak. The authors learn a reward function from human ratings on 2,500 summaries, taking only the source document and the generated summary as input, so it never needs a reference summary. They report that this learned reward correlates significantly better with human ratings than ROUGE and other metrics, and that both an extractive and an abstractive RL summariser trained with it produce summaries that human readers rate higher than those produced by state-of-the-art systems. If this holds, reference-free reward learning can replace ROUGE as the training objective for summarisation, using far fewer human annotations than the document-reference pairs currently required.

What carries the argument

The central object is a learned reward function $R(x,y;\theta)$ that scores a summary $y$ for document $x$ without any reference summary. Its key component is a pairwise preference loss: for every pair of summaries of the same document, the model is trained to prefer the one with the higher human rating, using a softmax over reward scores. The best-performing implementation encodes the document and summary with BERT, applied with a sliding window for long texts, and feeds the concatenated embeddings into a multilayer perceptron; a second architecture, SimRed, explicitly computes sentence-similarity and redundancy matrices. The same reward is then used both as the optimisation target for RL training and as the scoring function during inference.

What would settle it

Take a held-out set of articles, generate summaries with a policy trained on the learned reward, collect human ratings for those summaries, and measure the reward's Spearman correlation with the ratings; if it is no better than ROUGE's correlation, the central claim that the learned reward is a better training signal than ROUGE fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that a reward function $R(x,y;\theta)$ trained on 2,500 human overall-quality ratings of summaries for 500 CNN/DailyMail articles can stand in for reference-based metrics in RL summarisation. The best configuration, BERT+MLP with a pairwise preference loss, reaches a Spearman correlation of $0.505$ with human ratings, substantially above ROUGE-1's $0.290$, and improves the precision and recall of identifying 'good' summaries. In human evaluation, the extractive NeuralTD system trained on the learned reward scores $2.52$ on a 3-point scale, versus $2.27$ for Refresh and $1.66$ for ExtAbsRL, and the abstractive ExtAbsRL system rises from $1.75$ with ROUGE rewards to $2.20$ with the learned reward. The authors conclude that a small set of human ratings can replace the two orders of magnitude more document-reference pairs used by current systems, and that the idea extends naturally to other natural-language generation tasks.

Load-bearing premise

The load-bearing premise is that a reward function trained on human ratings of summaries produced by four fixed systems will also give accurate scores to the different kinds of summaries an RL agent generates when optimising against it; the paper only tests this indirectly on 30 and 20 new articles in two human evaluations.

Editorial extensions

If this is right

  • Summarisation systems can be trained end-to-end without any reference summaries, as long as a reward function has been learned from human ratings.
  • Human ratings collected once on a few thousand summaries can be reused to train many different summarisers, since the reward model is independent of the policy.
  • ROUGE should not be trusted as a summary-level reward for RL; the paper's correlation study shows it can barely distinguish good from bad summaries.
  • Both extractive and abstractive RL frameworks benefit from swapping in the learned reward, suggesting the result is not tied to one architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reward generalises beyond CNN/DailyMail, the same recipe could replace ROUGE in other generation tasks such as machine translation and dialogue, where references are also scarce; the authors leave this as future work.
  • A natural next test is iterative training: use the reward to train a policy, collect new human ratings on that policy's summaries, retrain the reward, and see whether correlation with human judgement and downstream quality keep improving.
  • Because the reward is trained on summaries from only four fixed systems, it may be biased towards their error patterns; an out-of-distribution check on summaries from a very different summariser would clarify its robustness.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a reference-free reward function for reinforcement-learning-based document summarization. The reward is learned from 2,500 human ratings of summaries produced by four fixed systems on 500 CNN/DailyMail articles, using either a regression loss or a preference (cross-entropy) loss over summary pairs, with several text encoders (CNN-RNN, PMeans-RNN, BERT) and two reward architectures (MLP and SimRed). The authors report that BERT+MLP trained with the preference loss has significantly higher summary-level correlation with human judgments than ROUGE and other reference-based metrics. They then use this learned reward to train an extractive RL system (NeuralTD) and an abstractive system (ExtAbsRL), and report human evaluations on 30 and 20 test documents, respectively, in which summaries trained with the learned reward receive higher average human ratings than comparisons trained with ROUGE-based rewards.

Significance. If the central claim holds, the paper would show that a reward function learned from a small set of human ratings (2,500 summaries) can replace ROUGE as an RL training signal and yield summaries that humans prefer, without requiring reference summaries at either training or inference time. The strengths of the paper include a clean problem formulation, a careful cross-validated evaluation of the reward itself, an explicit comparison of regression versus preference learning, the release of code and models, and the inclusion of both extractive and abstractive RL systems. The authors also directly document the poor summary-level correlation of ROUGE, which is a useful negative result. However, the final human evaluations are small, the comparisons are partly confounded by architecture differences, and the out-of-distribution validity of the learned reward on RL-generated outputs is not directly tested; these issues limit the strength of the conclusions that can currently be drawn.

major comments (3)
  1. [§3 and §6, Table 2] The reward model is trained on human ratings of summaries produced by four fixed systems (See et al. 2017; Paulus et al. 2018), and Table 2 evaluates it only on that same distribution via 5-fold cross-validation. The RL agents trained in §7 optimize this reward over outputs that are deliberately different from those training summaries, so the central claim that "better rewards yield better summaries" requires the reward to remain aligned with human quality on RL-generated outputs. This alignment is not directly tested: no correlation or agreement analysis is reported on the actual outputs of NeuralTD or ExtAbsRL. I request either a direct evaluation of the learned reward against fresh human ratings on RL-generated summaries, or a clearly stated limitation with appropriately weakened claims.
  2. [§7.2, Tables 3-4] The extractive human evaluation compares NeuralTD+LearnedReward with Refresh and ExtAbsRL, which differ from NeuralTD not only in reward but also in architecture, training procedure, and extractor design. Because a NeuralTD model trained with ROUGE as the reward is not included, the observed human-rating differences cannot be attributed solely to the reward function. In addition, the abstract claims superiority over "the state-of-the-art supervised-learning systems," but none of the supervised systems listed in Table 3 is included in the human evaluation in §7.2 or §7.3. Please add the missing controlled comparison (e.g., NeuralTD with ROUGE reward) or revise the claims to match the experiments actually performed.
  3. [§7.2 and §7.3, Tables 4-5] The human evaluation is based on only 30 documents with 5 raters (extractive) and 20 documents with 3 raters (abstractive). The reported t-tests are not corrected for multiple comparisons and are applied to ordinal Likert-scale ratings. Consequently, the reported differences (e.g., 2.52 vs. 2.27 and 2.20 vs. 1.75) have wide confidence intervals, and the "Best%" and "Pref%" proportions are based on very small counts. Please report confidence intervals (bootstrap or Bayesian), effect sizes, and the number of statistical tests performed, and interpret the results accordingly.
minor comments (6)
  1. [§8, Conclusion] "2,500 human ratings on 500 summaries" should read "2,500 summaries from 500 articles" or "2,500 human ratings of summaries for 500 articles."
  2. [§6] The statement that "p-values between each pair of metrics/rewards can be found in the supplementary material" is not verifiable from the arXiv version; please include the full pairwise significance table in the paper or in an accessible appendix.
  3. [§5.1, BERT encoder] The sliding-window description gives an offset size of 128 tokens but not the window length; please specify both the window size and the stride for reproducibility.
  4. [§5.2, Eq. (4)] The choice of α = 0.85 is reported without a sensitivity analysis; a small study over α values would increase confidence that the SimRed results are not tuned to a single fold.
  5. [§7.1] The lead-sentence bonus of 0.5 and the length factor of 1.2 are motivated by pilot studies, but the range of values tried and the selection criterion are not reported; please add these details for reproducibility.
  6. [Table 3] The phrase "full-length ROUGE F-scores" could be misread; clarify that these are standard ROUGE F1 scores computed on the complete generated summaries.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reward is cross-validated and the RL results are confirmed by fresh human evaluations.

full rationale

The paper's central chain is empirical and out-of-sample. The reward model is trained on 2,500 human ratings compiled by Chaganty et al. (2018), and its correlation with human judgment is measured by 5-fold cross-validation in Table 2, so the reward-quality claim is not a fit-to-data-as-prediction artifact. The RL experiments in Section 7 are then evaluated with fresh human ratings on new articles from the CNN/DailyMail test set, not on the training summaries; the paired comparison in Table 5 holds the summariser fixed (ExtAbsRL) and varies only the reward (ROUGE-L versus learned), directly testing the paper's title claim. The self-citations to Gao et al. (2019) for the NeuralTD system and to Peyrard and Gurevych (2018) as a baseline are not load-bearing: NeuralTD's base algorithm is attributed to external prior work (Ryang and Abekawa, 2012), the Table 5 comparison does not depend on NeuralTD, and the baseline is outperformed rather than used as support. The remaining concerns, such as the small human-evaluation sample sizes and the possible distribution shift from the four training summarisers to the RL policy, are external-validity and statistical-power risks rather than circularity, because the final judgment is still measured against independent human opinions rather than being reconstructed from the training labels.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The ledger captures the hand-tuned constants and domain assumptions beyond the neural network parameters learned from data. The three free parameters directly affect the RL training setup, while the axioms state the premises that must hold for the central claim to transfer beyond the specific experiments.

free parameters (3)
  • SimRed alpha = 0.85
    Weight in Eq. (4) balancing informativeness against redundancy; set manually in Section 5.2, not selected per fold.
  • NeuralTD lead-sentence bonus = 0.5
    Extra reward for selecting one of the first three sentences; chosen in a pilot study over values 0.1 to 0.9 (Section 7.1).
  • ExtAbsRL length factor = 1.2
    Stopping threshold set to 1.2 times reference summary length, based on observed lengths of the original ExtAbsRL system (Section 7.1).
assumptions (4)
  • domain assumption Human ratings provided by Chaganty et al. (averaged over five workers) are a reliable ground-truth reward R*.
    Used throughout Sections 3 and 4 as the target for reward learning.
  • domain assumption A reward function with higher summary-level correlation with human ratings will improve RL training outcomes.
    Core premise of the paper; tested in Section 7, not proven a priori.
  • domain assumption Pre-trained embeddings (GloVe, BERT) adequately represent sentences and documents for reward scoring without fine-tuning.
    Section 5.1 uses these embeddings; BERT is kept fixed due to the small dataset size.
  • domain assumption The test articles sampled for human evaluation (30 and 20) are representative of the CNN/DailyMail distribution.
    Sections 7.2 and 7.3 base the main human-level conclusions on these samples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Better Rewards Yield Better Summaries: Learning to Summarise Without References." pith.science (2026). https://pith.science/paper/474AIVG2

@misc{pith2026190901214,
  author       = {Pith},
  title        = {Pith review of: Better Rewards Yield Better Summaries: Learning to Summarise Without References},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/474AIVG2}},
  note         = {Machine review of arXiv:1909.01214}
}
read the original abstract

Reinforcement Learning (RL) based document summarisation systems yield state-of-the-art performance in terms of ROUGE scores, because they directly use ROUGE as the rewards during training. However, summaries with high ROUGE scores often receive low human judgement. To find a better reward function that can guide RL to generate human-appealing summaries, we learn a reward function from human ratings on 2,500 summaries. Our reward function only takes the document and system summary as input. Hence, once trained, it can be used to train RL-based summarisation systems without using any reference summaries. We show that our learned rewards have significantly higher correlation with human ratings than previous approaches. Human evaluation experiments show that, compared to the state-of-the-art supervised-learning systems and ROUGE-as-rewards RL summarisation systems, the RL systems using our learned rewards during training generate summarieswith higher human ratings. The learned reward function and our source code are available at https://github.com/yg211/summary-reward-no-reference.

Figures

Figures reproduced from arXiv: 1909.01214 by the authors.

Figure 1
Figure 1. Distributions of some metrics/rewards for summaries with different human ratings. Among the four [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DrugImproverGPT: A Large Language Model for Drug Optimization with Fine-Tuning via Structured Policy Optimization

    cs.LG 2025-02 conditional novelty 5.0 of 10

    SPO fine-tunes an LLM to rewrite drug molecules into analogs that score higher on docking, drug-likeness, solubility, and synthesizability, beating baselines on two protein targets.

Reference graph

Works this paper leans on

43 extracted references · 32 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Chris Alberti , Kenton Lee , and Michael Collins . 2019. http://arxiv.org/abs/1901.08634 A BERT Baseline for the Natural Questions . arXiv e-prints

    arXiv 2019

  4. [4]

    Kristjan Arumae and Fei Liu. 2019. https://doi.org/http://dx.doi.org/10.18653/v1/N19-1264 Guiding extractive summarization with question-answering rewards . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2566--2577, Minneapolis, USA

    work page doi:10.18653/v1/n19-1264 2019

  5. [5]

    Bowman, Gabor Angeli, Christopher Potts, and Christopher D

    Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. http://aclweb.org/anthology/D/D15/D15-1075.pdf A large annotated corpus for learning natural language inference . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632--642, Lisbon, Portugal

    work page 2015

  6. [6]

    Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. http://www.aclweb.org/anthology/P18-1060 The price of debiasing automatic metrics in natural language evalaution . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers, pages 643--653, Melbourne, Australia

    work page 2018

  7. [7]

    Yen - Chun Chen and Mohit Bansal. 2018. https://aclanthology.info/papers/P18-1063/p18-1063 Fast abstractive summarization with reinforce-selected sentence rewriting . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers, pages 675--686, Melbourne, Australia

    work page 2018

  8. [8]

    Alexis Conneau, Douwe Kiela, Holger Schwenk, Lo \" c Barrault, and Antoine Bordes. 2017. https://aclanthology.info/papers/D17-1070/d17-1070 Supervised learning of universal sentence representations from natural language inference data . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670--680, Copenhagen, Denmark

    work page 2017

Show all 43 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  2. [10]

    Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. https://aclanthology.info/papers/D18-1409/d18-1409 Banditsum: Extractive summarization as a contextual bandit . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language...

    2018

  3. [11]

    Meyer, and Iryna Gurevych

    Yang Gao, Christian M. Meyer, and Iryna Gurevych. 2018. https://aclanthology.info/papers/D18-1445/d18-1445 APRIL: interactively learning to summarise by combining active preference learning and reinforcement learning . In Proceedings of the 2018 Conference on Empirical Methods...

    2018

  4. [12]

    Meyer , and Iryna Gurevych

    Yang Gao , Christian M. Meyer , and Iryna Gurevych . 2019. http://arxiv.org/abs/1906.02923 Preference-based Interactive Multi-Document Summarisation . arXiv e-prints

    2019 arXiv

  5. [13]

    Meyer, Mohsen Mesgar, and Iryna Gurevych

    Yang Gao, Christian M. Meyer, Mohsen Mesgar, and Iryna Gurevych. 2019. https://doi.org/10.24963/ijcai.2019/326 Reward learning for efficient reinforcement learning in extractive document summarisation . In Proceedings of the Twenty-Eighth International Joint Conference on Arti...

    2019 doi

  6. [14]

    Max Grusky, Mor Naaman, and Yoav Artzi. 2018. https://aclanthology.info/papers/N18-1065/n18-1065 Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa...

    2018

  7. [15]

    Karl Moritz Hermann, Tom \' a s Kocisk \' y , Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend Teaching machines to read and comprehend . In Advances in Neural Inf...

    2015

  8. [16]

    McKeown, and Hal Daum \' e III

    Chris Kedzie, Kathleen R. McKeown, and Hal Daum \' e III. 2018. https://aclanthology.info/papers/D18-1208/d18-1208 Content selection in deep learning models of summarization . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1818...

    2018

  9. [17]

    Yoon Kim. 2014. http://aclweb.org/anthology/D/D14/D14-1181.pdf Convolutional neural networks for sentence classification . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1746--1751, Doha, Qatar

    2014

  10. [18]

    Julia Kreutzer, Artem Sokolov, and Stefan Riezler. 2017. https://doi.org/10.18653/v1/P17-1138 Bandit structured prediction for neural sequence-to-sequence learning . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Pape...

    2017 doi

  11. [19]

    Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. 2018. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers, pa...

    2018

  12. [20]

    Wojciech Kryscinski, Romain Paulus, Caiming Xiong, and Richard Socher. 2018. https://aclanthology.info/papers/D18-1207/d18-1207 Improving abstraction in text summarization . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1808--...

    2018

  13. [21]

    Denkowski

    Alon Lavie and Michael J. Denkowski. 2009. https://doi.org/10.1007/s10590-009-9059-4 The meteor metric for automatic evaluation of machine translation . Machine Translation, 23(2-3):105--115

    2009 doi

  14. [22]

    Chin-Yew Lin. 2004 a . Looking for a few good metrics: Rouge and its evaluation. In NTCIR Workshop

    2004

  15. [23]

    Chin-Yew Lin. 2004 b . http://aclweb.org/anthology/W04-1013 ROUGE : A package for automatic evaluation of summaries . In ACL Workshop ``Text Summarization Branches Out''

    2004

  16. [24]

    Annie Louis and Ani Nenkova. 2013. https://doi.org/10.1162/COLI\_a\_00123 Automatically assessing machine summary content without a gold standard . Computational Linguistics, 39(2):267--300

    2013 doi

  17. [25]

    Cohen, Mirella Lapata, Jiangsheng Yu, and Yi Chang

    Shashi Narayan, Ronald Cardenas, Nikos Papasarantopoulos, Shay B. Cohen, Mirella Lapata, Jiangsheng Yu, and Yi Chang. 2018 a . https://aclanthology.info/papers/P18-1188/p18-1188 Document modeling with external attention for sentence extraction . In Proceedings of the 56th Annu...

    2018

  18. [26]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018 b . https://aclanthology.info/papers/N18-1158/n18-1158 Ranking sentences for extractive summarization with reinforcement learning . In Proceedings of the 2018 Conference of the North American Chapter of the Association fo...

    2018

  19. [27]

    Ng and Stuart J

    Andrew Y. Ng and Stuart J. Russell. 2000. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 663--670, Stanford University, Stanford, CA, USA

    2000

  20. [28]

    Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. https://aclanthology.info/papers/D17-1238/d17-1238 Why we need new evaluation metrics for NLG . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 224...

    2017

  21. [29]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei - Jing Zhu. 2002. http://www.aclweb.org/anthology/P02-1040.pdf BLEU : a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages...

    2002

  22. [30]

    Ramakanth Pasunuru and Mohit Bansal. 2018. https://aclanthology.info/papers/N18-2102/n18-2102 Multi-reward reinforced summarization with saliency and entailment . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisti...

    2018

  23. [31]

    Romain Paulus, Caiming Xiong, and Richard Socher. 2018. https://openreview.net/forum?id=HkAClQgA- A deep reinforced model for abstractive summarization . In 6th International Conference on Learning Representations, Conference Track Proceedings, Vancouver, BC, Canada

    2018

  24. [32]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. https://doi.org/10.3115/v1/D14-1162 Glove: Global vectors for word representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532--1543, Doha, Qatar

    2014 doi

  25. [33]

    Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. https://aclanthology.info/papers/W17-4510/w17-4510 Learning to score system summaries for better content selection evaluation . In Proceedings of the Workshop on New Frontiers in Summarization, pages 74--84, Copenhagen...

    2017

  26. [34]

    Maxime Peyrard and Iryna Gurevych. 2018. https://aclanthology.info/papers/N18-2103/n18-2103 Objective function learning to match human judgements for optimization-based summarization . In Proceedings of the 2018 Conference of the North American Chapter of the Association for C...

    2018

  27. [35]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. http://aclweb.org/anthology/D/D16/D16-1264.pdf SQuAD: 100, 000+ Questions for Machine Comprehension of Text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pag...

    2016

  28. [36]

    Hasan, and Yllias Chali

    Cody Rioux, Sadid A. Hasan, and Yllias Chali. 2014. http://aclweb.org/anthology/D/D14/D14-1075.pdf Fear the REAPER: A system for automatic multi-document summarization with reinforcement learning . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language ...

    2014

  29. [37]

    Andreas R \"u ckl \'e , Steffen Eger , Maxime Peyrard , and Iryna Gurevych . 2018. http://arxiv.org/abs/1803.01400 Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations . arXiv e-prints

    2018 arXiv

  30. [38]

    Seonggi Ryang and Takeshi Abekawa. 2012. http://www.aclweb.org/anthology/D12-1024 Framework of automatic text summarization using reinforcement learning . In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural ...

    2012

  31. [39]

    Liu, and Christopher D

    Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. https://doi.org/10.18653/v1/P17-1099 Get to the point: Summarization with pointer-generator networks . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers...

    2017 doi

  32. [40]

    Simeng Sun, Ori Shapira, Ido Dagan, and Ani Nenkova. 2019. https://doi.org/10.18653/v1/W19-2303 How to Compare Summarizers without Target Length? Pitfalls, Solutions and Re-Examination of the Neural Summarization Literature . In Proceedings of the Workshop on Methods for Optim...

    2019 doi

  33. [41]

    Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. https://aclanthology.info/papers/N18-1101/n18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North American Chapter of the Association...

    2018

  34. [42]

    Xingxing Zhang, Mirella Lapata, Furu Wei, and Ming Zhou. 2018. https://aclanthology.info/papers/D18-1088/d18-1088 Neural latent extractive document summarization . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 779--784, Brusse...

    2018

  35. [43]

    Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. https://aclanthology.info/papers/P18-1061/p18-1061 Neural document summarization by jointly learning to score and select sentences . In Proceedings of the 56th Annual Meeting of the Association f...

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.