Pith. sign in

REVIEW 4 major objections 7 minor 70 references

Multi-document summaries improve when a topic-F1 reward trains a small LLM to match the sources' key topics

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A topic-F1 reward measuring alignment between summary and source-document topics, combined with GRPO training, improves multi-document summarization over several baselines.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A plausible topic-reward for MDS with real empirical gains, but the 'consistent improvements' claim is undercut by missing significance tests and a partially circular evaluation. the 4 major comments →

arxiv 2509.09852 v1 pith:6F3ZPQHJ submitted 2025-09-11 cs.CL

Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization

Chuyuan Li , Austin Xu , Shafiq Joty , Giuseppe Carenini This is my paper
classification cs.CL
keywords multi-document summarizationreinforcement learningGRPOtopic modelingtopic rewardLLM summarizationRLHFreference-free reward
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes using explicit topic phrases—short labels extracted from each source document—as a training signal for multi-document summarization. It first shows that prompting a small LLM with these topic labels improves summary informativeness. Then it introduces a reference-free 'topic-F1' reward, the harmonic mean of coverage and precision computed from cosine similarities between topic-phrase embeddings of source and summary, and trains a 0.5B policy model with Group Relative Policy Optimization (GRPO) using this reward. The paper reports consistent gains over standard, RLHF-guided, and ROUGE-reward baselines on Multi-News and Multi-XScience, and finds the small trained model matches an untuned 7B model. If correct, this suggests topic-level supervision is a cheap, interpretable alternative to human preference data for summarization.

Core claim

The central claim is that explicitly guiding a summarizer with topics—both at prompt time and as an RL reward—improves multi-document summarization quality. The topic-F1 reward is computed by extracting topic phrases from each source document and from the generated summary, embedding them, forming a cosine-similarity matrix, and taking the harmonic mean of coverage (how well each source topic is matched by some summary topic) and precision (how well each summary topic is matched by some source topic). This reward, combined with a length penalty and inverse-standard-deviation weighting, is optimized with GRPO. The paper reports that this topic-guided RL outperforms baselines including RLHF an

What carries the argument

The topic-F1 reward: topic phrases are extracted from each source document and from the generated summary (by a teacher LLM for the documents, and the same teacher for the summary); phrase embeddings are computed with a sentence-transformer; the cosine similarity matrix between document topics and summary topics yields coverage (mean max similarity over source topics) and precision (mean max similarity over summary topics), whose harmonic mean is the reward. The reward is averaged over all source documents, weighted against a length reward using inverse standard deviations, and fed into GRPO to compute group advantages for policy updates.

Load-bearing premise

The key assumption is that making the generated summary more topically similar to the source documents—as measured by embedding overlap of topic phrases—makes the summary better; if that overlap is a weak or gameable proxy for true summary quality, the reward will not generalize.

What would settle it

Run an experiment where summaries are produced by simply concatenating the extracted topic phrases from the source documents, or by copying the first paragraph of each source verbatim; compute the topic-F1 reward, ROUGE against gold summaries, and human quality scores. If the topic-F1 reward ranks these shallow summaries above or near the RL-trained summaries while human judges rank them clearly worse, the reward is not faithfully capturing summary quality.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Topic-level supervision can improve multi-document summarization without any human preference labels, using a frozen teacher LLM's topic extractions.
  • A much smaller policy model (0.5B) can match an untuned 7B model on summarization quality, implying cheaper inference for comparable quality.
  • Combining the reference-free topic reward with a reference-based ROUGE reward yields further gains, showing the two signals are complementary.
  • The topic-F1 score works as a test-time selection metric: best-of-n reranking with topic-F1 improves both base and RL-trained models.
  • Topic-guided RL-trained models stay stable as the number of source documents grows, while SFT degrades, suggesting robustness to input length and volume.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the topic reward is computed from embedding overlap of extracted topic phrases, one testable extension is whether adversarial summaries that merely echo the topic labels receive high reward but low human quality ratings—this would reveal surface-overlap gaming.
  • The teacher-student setup implies a scaling path: better topic extraction (via larger or more specialized teachers) should transfer to better summaries; comparing teachers of increasing size would quantify this.
  • A likely boundary is that topic-F1 rewards may favor broad topical coverage over critical synthesis, especially on scientific MDS where the query paper's contribution must be related to cited works; a targeted study on abstractive synthesis quality would probe this.
  • The method's reliance on a fixed embedding model and a teacher LLM introduces two extra components; ablating each (e.g., using different embedding models or topic extractors) would test how much of the gain depends on these choices.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes a topic-guided reinforcement learning approach for multi-document summarization (MDS). It first shows that prompting small LLMs (Qwen2.5-0.5B) with topic labels extracted by a larger teacher model (Qwen2.5-7B) improves ROUGE scores. It then introduces a reference-free Topic-F1 reward, computed as the harmonic mean of coverage and precision between topic phrases extracted from source documents and generated summaries, and integrates it into GRPO training with a length penalty and inverse-standard-deviation reward weighting. Experiments on Multi-News and Multi-XScience compare the resulting RL TOPIC-7B model against zero-shot baselines, an RLHF-guided baseline, SFT, and ROUGE-reward RL, reporting ROUGE, BERTScore, LLM2V, and topic-alignment COVRATIO/PRERATIO. The paper claims consistent improvements over baselines, and shows that combining topic reward with ROUGE reward yields further gains. An LLM-as-a-judge evaluation and a best-of-n analysis are also included.

Significance. If the central claim holds, the paper makes a useful contribution: an interpretable, reference-free topic-alignment reward that enables a 0.5B policy to approach or exceed an untuned 7B baseline, and a knowledge-distillation-style mechanism for transferring topic structure from a larger teacher to a smaller policy. Strengths include the release of code, the use of several complementary automatic metrics, an LLM-as-a-judge evaluation with position-order control, and ablations that isolate coverage versus precision rewards. The topic-F1 reward is a natural and cheap training signal that could be reused beyond MDS. However, the force of the empirical claim depends on statistical reliability and on the independence of the evaluation from the reward construction; both are currently only partially addressed. The contribution is worthwhile but needs revision to establish the claimed consistency of improvement.

major comments (4)
  1. [§6.1, Table 1] The central reference-free claim of 'consistent improvements' is not supported by significance testing or error bars. The caption says scores are averaged over six runs, but no standard deviations or confidence intervals are reported, and the Wilcoxon test is only applied to reference-based comparisons with SFT in §6.2. Moreover, the point estimates are not uniformly higher for the reference-free comparison: on Multi-News, RL TOPIC-7B has Rouge-L 18.97 versus BASE(7B) 19.77, and BERTScore is tied at .845. Given the small 0.5B policy and the known variance of RL training, the observed ROUGE-M and LLM2V gains may be within noise. Please provide per-instance paired significance tests (or CIs) for the reference-free comparisons, especially RL TOPIC-7B vs. BASE(7B) and RL TOPIC-7B vs. RL HUMAN-FEEDBACK, and discuss the inconsistent point estimates.
  2. [§4.1 Eqs. (3)–(4); §5.2] The topic-alignment evaluation is partly circular with the training reward. The COVRATIO and PRERATIO metrics in §5.2 use the same coverage and precision construction as the Topic-F1 reward in Eqs. (3)–(4). Thus the reported 2–7 point gains in Coverage and 4–8 point gains in Precision are partly by construction: the policy is explicitly trained to maximize this exact objective. The external metrics (ROUGE, BERTScore, LLM2V) provide partial grounding, and the LLM-as-a-judge results help, but the paper's claim about 'topic alignment evaluation' should be framed with this dependency acknowledged, or an independent topic-alignment metric (e.g., a different topic extraction/embedding pipeline) should be reported.
  3. [§4.1; §6.5] The key assumption—that increasing topical similarity between summary and source documents improves summary quality—remains unsupported by analysis. The Topic-F1 reward is a cosine-similarity-based matching of LLM-extracted topic phrases, and it is vulnerable to reward hacking via superficial lexical/embedding overlap rather than genuine synthesis. The manual check in the Ethical Statement (200 samples) is reassuring but limited. The LLM-as-a-judge prompt in Appendix E asks judges to select the 'more topically aligned' summary, which shares the same conceptual bias. A concrete robustness test would be to generate summaries that maximize Topic-F1 without RL (e.g., by copying topic phrases into the summary) and show that such degenerate outputs are not preferred by the independent metrics; alternatively, report human preference judgments on the final summaries, not only on extracted topics
  4. [§4.1, §4.3, Appendix C] The manuscript relies on several hyperparameters that are fitted on validation sets (n=10/5 topics, m=5 summary topics, factor_topic=2, length target L_exp), but no sensitivity analysis is provided. Appendix C explicitly states that 'extensive hyperparameter optimization was impractical.' Since the central comparison includes RL TOPIC-7B versus RL HUMAN-FEEDBACK and BASE(7B), the robustness of the claimed gains to these choices should be demonstrated, at least for n and factor_topic on one dataset. Without this, it is difficult to know whether the results are intrinsic to topic-guided RL or artifacts of a particular validation-set selection.
minor comments (7)
  1. [Table 1 caption] The caption says 'Scores are averaged over six runs' but the table gives no standard deviation or variance. Even for the reference-based rows, the reader cannot see the spread that motivates the significance tests. Please add error bars or a supplementary table with per-metric variances.
  2. [§6.1] The sentence 'Interestingly, our RL-trained model, despite being much smaller, performed comparably with the much larger 7B model (BASE(7B))' is accurate, but the abstract and conclusion say 'consistently outperforms strong baselines.' 'Comparably' and 'consistently outperforms' are different claims; please align the wording throughout.
  3. [§6.2, Table 1] In Multi-XScience, RL TOPIC-7B+ROUGE has Rouge-2 8.96, which is lower than SFT's 9.25, and it is not marked as significantly better. This is another instance where 'consistently surpasses' overstates the results. Please qualify the claim with the actual metric-by-metric pattern.
  4. [Appendix E, Table 13] The judge prompt asks about 'topical alignment' rather than overall summary quality. This is fine as a complement, but it does not directly validate the 'improving summary quality' claim. Consider also reporting a judge prompt that asks about overall quality, or noting that the judge only measures topic alignment.
  5. [Appendix C, Table 10] There is a typo 'deta point' in the Time Consumption paragraph. Also, Table 9's 'Nb. refs per summ' column is confusingly named; the number 2.8 for Multi-News is the average number of source documents per summary, not references per summary. Please rename to 'Average source docs per summary'.
  6. [§6.6 and Ethical Statement] The failure case frequency table (Appendix G) is useful, but the main text only mentions the qualitative finding. Since the SFT model's higher failure rate is invoked to explain variance, report the quantitative failure rates in the main text or include the table in the main paper.
  7. [Equation (7)] The notation R_r in Eq. (6)–(7) is used for both reward signals and weights; consider distinguishing the reward value from the weight variable to avoid confusion.

Circularity Check

2 steps flagged

Topic-alignment evaluation largely reuses the reward objective, but external lexical and semantic metrics keep the central claim non-circular.

specific steps
  1. self definitional [Section 4.1 (Eqs. 3-4) and Section 5.2 (evaluation metrics); Table 1]
    "Then, we define Coverage as the average of the maximum similarity scores between each source topic and its most similar summary topic. Conversely, Precision is defined as the average of the maximum similarity scores between each summary topic and its most similar source topic. ... Additionally, we examine topical alignment via COVRATIO and PRERATIO, reflecting respectively the coverage and precision of extracted topics between the summary and source documents."

    The RL reward is the harmonic mean of Coverage and Precision defined in Eqs. (3)-(4), and the evaluation metrics COVRATIO/PRERATIO are the same coverage/precision construction. A policy trained to maximize this reward will therefore mechanically raise COVRATIO/PRERATIO relative to untrained baselines. The reported 2-7 point Coverage and 4-8 point Precision gains are thus a consequence of training on the same objective, not an independent confirmation that topical similarity improves summary quality. The external metrics (ROUGE, BERTScore, LLM2V) do provide independent grounding, so the circularity is partial.

  2. self definitional [Section 6.3 and Table 2]
    "To evaluate the effectiveness of this strategy, we generate eight responses using different random seeds from the BASE/ RL-trained model, calculate all evaluation metrics, and select the response with the highest topic F1 score. ... The performance trend is clear: RL + best-of-n > RL > Base + best-of-n > Base."

    The best-of-n selection criterion is the same topic-F1 score used to compute the F1/COVRATIO/PRERATIO columns in Table 2. Choosing the summary with the highest topic-F1 and then reporting that those topic-alignment columns improve is guaranteed by the selection rule, not by genuine summary-quality improvement. The non-topic metrics are the meaningful test here, and they improve only partially (e.g., BERTScore on Multi-News drops from .845 to .842), so presenting the whole trend as uniformly positive overstates the evidence.

full rationale

The paper's central claim is that topic-guided RL improves multi-document summarization. That claim is supported by external metrics: ROUGE, BERTScore, and LLM2V score are computed against gold summaries and are not part of the training reward used for the main RL TOPIC variants. These metrics give independent evidence, so the core result is not circular. The main circularity is confined to the topic-alignment evaluation: COVRATIO/PRERATIO are essentially the same Coverage/Precision construction that defines the topic-F1 reward, as are the F1 and alignment columns in the best-of-n experiment. This makes those specific reported improvements partly by construction. The paper's own Limitations section candidly discusses evaluation challenges and proposes the topic-coverage/precision scores, but that candor does not remove the construction overlap. The self-citations to Xu et al. (2025) for the judge prompt and Cao et al. (2025) for best-of-n are implementation details and are not load-bearing; they do not raise the score. No uniqueness theorem or ansatz is imported from the authors' prior work. Overall, the derivation chain is not wholly circular because the headline improvements are anchored in independent metrics, but the topic-alignment evidence should be interpreted with the reward-evaluation overlap in mind.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The method introduces no new entities, but it relies on several fitted parameters (topic counts, emphasis factor, target length) and domain assumptions about the validity of LLM-based topic extraction and embedding similarity as proxies for summary quality.

free parameters (4)
  • n (number of source-document topics) = 10 for Multi-News, 5 for Multi-XScience
    Chosen because the best results were achieved when the number of topics matched the number of sentences in the gold summary (Section 4.1). This is a dataset-specific fitted choice.
  • m (number of summary topics) = 5 for both datasets
    Fixed by the authors for the generated summary topics; not justified by data but affects the reward computation.
  • factor_topic = 2
    Emphasis factor applied to the topic reward, selected based on development set performance (Section 4.3).
  • L_exp (target summary length) = Computed on a small validation set
    Used in the length-penalty reward (Eq. 5); tuned to reflect dataset-specific target lengths.
axioms (4)
  • domain assumption GRPO (Shao et al., 2024) provides a stable and effective RL algorithm for policy optimization
    The paper adopts GRPO without re-deriving or validating its convergence properties in this setting; it relies on prior success.
  • domain assumption all-mpnet-base-v2 sentence embeddings capture semantic similarity between topic phrases
    The topic-F1 reward (Eq. 2) uses cosine similarity between these embeddings as the core alignment measure; no validation of this specific choice for topic phrases is provided.
  • domain assumption Qwen2.5-7B extracts high-quality, relevant topic labels
    The entire reward pipeline depends on the 7B model's topic extraction being accurate; a small human evaluation (Section 6.6) supports this for 10 documents, but it is a limited check.
  • ad hoc to paper The harmonic mean of coverage and precision is an appropriate way to combine the two alignment signals
    The choice of harmonic mean versus arithmetic mean or other combination is justified only by the ablation results in Appendix D, not by a principled derivation.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization." pith.science (2026). https://pith.science/paper/6F3ZPQHJ

@misc{pith2026250909852,
  author       = {Pith},
  title        = {Pith review of: Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6F3ZPQHJ}},
  note         = {Machine review of arXiv:2509.09852}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

A key challenge in Multi-Document Summarization (MDS) is effectively integrating information from multiple sources while maintaining coherence and topical relevance. While Large Language Models have shown impressive results in single-document summarization, their performance on MDS still leaves room for improvement. In this paper, we propose a topic-guided reinforcement learning approach to improve content selection in MDS. We first show that explicitly prompting models with topic labels enhances the informativeness of the generated summaries. Building on this insight, we propose a novel topic reward within the Group Relative Policy Optimization (GRPO) framework to measure topic alignment between the generated summary and source documents. Experimental results on the Multi-News and Multi-XScience datasets demonstrate that our method consistently outperforms strong baselines, highlighting the effectiveness of leveraging topical cues in MDS.

Figures

Figures reproduced from arXiv: 2509.09852 by Austin Xu, Chuyuan Li, Giuseppe Carenini, Shafiq Joty.

Figure 1
Figure 1. Figure 1: Performance on Multi-News (Fabbri et al., 2019) using prompting (Base) and topic-incorporated prompting (Tn; n means number of topic labels) with Qwen2.5-series model (Qwen et al., 2025). The geomet￾ric mean of Rouge-1/2/L scores are reported. Topic key words are previously generated using a teacher model: Qwen2.5-7B. We see that topic-enhanced instruction (T5 and T10) improves small LLMs’ (0.5B and 1.5B) … view at source ↗
Figure 2
Figure 2. Figure 2: Multi-Document Summarization training (a) using our proposed Topic-Guided reward (b), with GRPO [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Model performance under different number [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Model performance under different number [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 14 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2024. https://arxiv.org/pdf/2303.08774 Gpt-4 technical report . arXiv preprint arXiv:2303.08774

    Pith/arXiv arXiv 2024

  4. [4]

    Shmuel Amar, Liat Schiff, Ori Ernst, Asi Shefer, Ori Shapira, and Ido Dagan. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.121 O pen A sp: A benchmark for multi-document open aspect-based summarization . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1967--1991, Singapore. Association for Computational...

    work page doi:10.18653/v1/2023.emnlp-main.121 2023

  5. [5]

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. https://arxiv.org/pdf/2404.05961 Llm2vec: Large language models are secretly powerful text encoders . In First Conference on Language Modeling

    Pith/arXiv arXiv 2024

  6. [6]

    Catarina G Bel \'e m, Pouya Pezeshkpour, Hayate Iso, Seiji Maekawa, Nikita Bhutani, and Estevam Hruschka. 2025. https://doi.org/10.18653/v1/2025.findings-naacl.293 From single to multi: How LLM s hallucinate in multi-document summarization . In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5276--5309, Albuquerque, New Mexico...

    doi:10.18653/v1/2025.findings-naacl.293 2025

  7. [7]

    Federico Bianchi, Silvia Terragni, and Dirk Hovy. 2021. https://doi.org/10.18653/v1/2021.acl-short.96 Pre-training is a hot topic: Contextualized document embeddings improve topic coherence . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing...

    doi:10.18653/v1/2021.acl-short.96 2021

  8. [8]

    David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993--1022

    2003

  9. [9]

    Juntai Cao, Xiang Zhang, Raymond Li, Chuyuan Li, Chenyu You, Shafiq Joty, and Giuseppe Carenini. 2025. https://arxiv.org/abs/2502.20592 Multi2: Multi-agent test-time scalable framework for multi-document processing . arXiv preprint arXiv:2502.20592

    Pith/arXiv arXiv 2025

  10. [10]

    Cheng-Han Chiang and Hung-yi Lee. 2023. https://doi.org/10.18653/v1/2023.acl-long.870 Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607--15631, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.acl-long.870 2023

  11. [11]

    Roberto Cipolla, Yarin Gal, and Alex Kendall. 2018. https://www.computer.org/csdl/proceedings-article/cvpr/2018/642000h482/17D45XeKgxA Multi-task learning using uncertainty to weigh losses for scene geometry and semantics . In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7482--7491. IEEE Computer Society

    2018

  12. [12]

    Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. https://doi.org/10.18653/v1/N18-2097 A discourse-aware attention model for abstractive summarization of long documents . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human...

    doi:10.18653/v1/n18-2097 2018

  13. [13]

    Peng Cui and Le Hu. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.126 Topic-guided abstractive multi-document summarization . In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1463--1472, Punta Cana, Dominican Republic. Association for Computational Linguistics

    work page doi:10.18653/v1/2021.findings-emnlp.126 2021

  14. [14]

    Peng Cui, Le Hu, and Yuanchao Liu. 2020. https://doi.org/10.18653/v1/2020.coling-main.468 Enhancing extractive text summarization with topic-aware graph neural networks . In Proceedings of the 28th International Conference on Computational Linguistics, pages 5360--5371, Barcelona, Spain (Online). International Committee on Computational Linguistics

    work page doi:10.18653/v1/2020.coling-main.468 2020

  15. [15]

    Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2021. https://doi.org/10.18653/v1/2021.naacl-main.384 GS um: A general framework for guided neural abstractive summarization . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4...

    work page doi:10.18653/v1/2021.naacl-main.384 2021

  16. [16]

    G \"u nes Erkan and Dragomir R Radev. 2004. https://www.jair.org/index.php/jair/article/download/10396/24901/ Lexrank: Graph-based lexical centrality as salience in text summarization . Journal of artificial intelligence research, 22:457--479

    2004

  17. [17]

    Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. https://doi.org/10.18653/v1/P19-1102 Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074--1084, Florence, Italy. Association for C...

    doi:10.18653/v1/p19-1102 2019

  18. [18]

    Fabbri, Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev

    Alexander R. Fabbri, Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. https://doi.org/10.1162/tacl_a_00373 S umm E val: Re-evaluating summarization evaluation . Transactions of the Association for Computational Linguistics, 9:391--409

    doi:10.1162/tacl_a_00373 2021

  19. [19]

    Zheng Fang, Yulan He, and Rob Procter. 2024. https://aclanthology.org/2024.lrec-main.382/ CWTM : Leveraging contextualized word embeddings from BERT for neural topic modeling . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4273--4286, Torino, Italia. ELRA...

    2024

  20. [20]

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.552 S im CSE : Simple contrastive learning of sentence embeddings . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894--6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics

    doi:10.18653/v1/2021.emnlp-main.552 2021

  21. [21]

    Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. https://arxiv.org/abs/2209.12356 News summarization and evaluation in the era of gpt-3 . arXiv preprint arXiv:2209.12356

    Pith/arXiv arXiv 2022

  22. [22]

    Aaron Grattafiori , Abhimanyu Dubey , Abhinav Jauhri , Abhinav Pandey , Abhishek Kadian , Ahmad Al-Dahle , Aiesha Letman , Akhil Mathur , Alan Schelten , Alex Vaughan , Amy Yang , Angela Fan , Anirudh Goyal , Anthony Hartshorn , Aobo Yang , Archi Mitra , Archie Sravankumar , Artem Korenev , Arthur Hinsvark , and 181 others. 2024. https://doi.org/10.48550/...

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783 2024

  23. [23]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. https://arxiv.org/pdf/2501.12948? Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . arXiv preprint arXiv:2501.12948

    Pith/arXiv arXiv 2025

  24. [24]

    Aria Haghighi and Lucy Vanderwende. 2009. https://aclanthology.org/N09-1041/ Exploring content models for multi-document summarization . In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North A merican Chapter of the Association for Computational Linguistics , pages 362--370, Boulder, Colorado. Association for Computational...

    2009

  25. [25]

    Sanda Harabagiu and Finley Lacatusu. 2005. https://dl.acm.org/doi/abs/10.1145/1076034.1076071 Topic themes for multi-document summarization . In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 202--209

    arXiv 2005

  26. [26]

    Junxian He, Wojciech Kryscinski, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.396 CTRL sum: Towards generic controllable text summarization . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5879--5915, Abu Dhabi, United Arab Emirates. Association for Com...

    work page doi:10.18653/v1/2022.emnlp-main.396 2022

  27. [27]

    Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. https://doi.org/10.18653/v1/D19-1051 Neural text summarization: A critical evaluation . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP...

    doi:10.18653/v1/d19-1051 2019

  28. [28]

    Philippe Laban, Alexander Fabbri, Caiming Xiong, and Chien-Sheng Wu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.552 Summary of a haystack: A challenge to long-context LLM s and RAG systems . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9885--9903, Miami, Florida, USA. Association for Computational...

    doi:10.18653/v1/2024.emnlp-main.552 2024

  29. [29]

    Bennett, and Marti A

    Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. https://doi.org/10.1162/tacl_a_00453 S umma C : Re-visiting NLI -based models for inconsistency detection in summarization . Transactions of the Association for Computational Linguistics, 10:163--177

    doi:10.1162/tacl_a_00453 2022

  30. [30]

    Chuyuan Li. 2023. http://docnum.univ-lorraine.fr/public/DDOC_T_2023_0107_LI.pdf Facing Data Scarcity in Dialogues for Discourse Structure Discovery and Prediction . Ph.D. thesis, Universit \'e de Lorraine

    2023

  31. [31]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  32. [32]

    Gili Lior, Avi Caciularu, Arie Cattan, Shahar Levy, Ori Shapira, and Gabriel Stanovsky. 2024. https://arxiv.org/abs/2406.16086 Seam: A stochastic benchmark for multi-document tasks . arXiv preprint arXiv:2406.16086

    Pith/arXiv arXiv 2024

  33. [33]

    Ran Liu, Ming Liu, Min Yu, Jianguo Jiang, Gang Li, Dan Zhang, Jingyuan Li, Xiang Meng, and Weiqing Huang. 2024 a . https://arxiv.org/abs/2408.10115 Glimmer: Incorporating graph and lexical features in unsupervised multi-document summarization . arXiv preprint arXiv:2408.10115

    Pith/arXiv arXiv 2024

  34. [34]

    Yang Liu and Mirella Lapata. 2019. https://doi.org/10.18653/v1/P19-1500 Hierarchical transformers for multi-document summarization . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070--5081, Florence, Italy. Association for Computational Linguistics

    work page doi:10.18653/v1/p19-1500 2019

  35. [35]

    Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan. 2024 b . https://doi.org/10.18653/v1/2024.findings-naacl.280 Benchmarking generation and evaluation capabilities of large language models for instruction controllable summarization . In Findings of the Association fo...

    doi:10.18653/v1/2024.findings-naacl.280 2024

  36. [36]

    Adian Liusie, Potsawee Manakul, and Mark Gales. 2024. https://aclanthology.org/2024.eacl-long.8/ LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 139--1...

    2024

  37. [37]

    Yao Lu, Yue Dong, and Laurent Charlin. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.648 Multi- XS cience: A large-scale dataset for extreme multi-document summarization of scientific articles . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8068--8074, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.emnlp-main.648 2020

  38. [38]

    Congbo Ma, Wei Emma Zhang, Mingyu Guo, Hu Wang, and Quan Z Sheng. 2022. https://dl.acm.org/doi/abs/10.1145/3529754 Multi-document summarization via deep learning techniques: A survey . ACM Computing Surveys, 55(5):1--37

    work page doi:10.1145/3529754 2022

  39. [39]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. https://doi.org/10.18653/v1/D18-1206 Don ' t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797--1807, Brussels, Belgium. Association for Co...

    doi:10.18653/v1/d18-1206 2018

  40. [40]

    Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, Kuzman Ganchev, Annie Louis, Fantine Huot, Anders Sandholm, Dipanjan Das, and Mirella Lapata. 2023. https://doi.org/10.1162/tacl_a_00583 Conditional generation with a question-answering blueprint . Transactions of the Association for Computational Linguistics, 11:974--996

    work page doi:10.1162/tacl_a_00583 2023

  41. [41]

    Shashi Narayan, Yao Zhao, Joshua Maynez, Gon c alo Sim \ o es, Vitaly Nikolaev, and Ryan McDonald. 2021. https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl_a_00438/1979348/tacl_a_00438.pdf Planning with learned entity prompts for abstractive summarization . Transactions of the Association for Computational Linguistics, 9:1475--1492

    work page doi:10.1162/tacl_a_00438/1979348/tacl_a_00438.pdf 2021

  42. [42]

    Ani Nenkova and Lucy Vanderwende. 2005. https://www.academia.edu/download/42163071/sumbasic.pdf The impact of frequency on summarization . Microsoft Research, Redmond, Washington, Tech. Rep. MSR-TR-2005, 101

    arXiv 2005

  43. [43]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf Training language models to follow instructions with human feedback . Advances in...

    2022

  44. [44]

    You Ouyang, Sujian Li, and Wenjie Li. 2007. https://dl.acm.org/doi/abs/10.1145/1321440.1321454 Developing learning strategies for topic-based summarization . In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, pages 79--86

    arXiv 2007

  45. [45]

    Ramakanth Pasunuru and Mohit Bansal. 2018. https://doi.org/10.18653/v1/N18-2102 Multi-reward reinforced summarization with saliency and entailment . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , pages 646--653, New Orleans, Louis...

    work page doi:10.18653/v1/n18-2102 2018

  46. [46]

    Romain Paulus, Caiming Xiong, and Richard Socher. 2018. https://arxiv.org/abs/1705.04304 A deep reinforced model for abstractive summarization . In International Conference on Learning Representations

    Pith/arXiv arXiv 2018

  47. [47]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115

    Pith/arXiv arXiv 2025

  48. [48]

    Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. https://arxiv.org/pdf/1511.06732 Sequence level training with recurrent neural networks . In ICLR

    Pith/arXiv arXiv 2016

  49. [49]

    Nils Reimers and Iryna Gurevych. 2019. https://aclanthology.org/D19-1410/ Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982--3992

    2019

  50. [50]

    DM Roijers, P Vamplew, S Whiteson, and R Dazeley. 2013. https://www.jair.org/index.php/jair/article/download/10836/25862 A survey of multi-objective sequential decision-making . Journal of Artificial Intelligence Research, 48:67--113

    2013

  51. [51]

    Sangwon Ryu, Heejin Do, Yunsu Kim, Gary Lee, and Jungseul Ok. 2024. https://doi.org/10.18653/v1/2024.acl-long.319 Multi-dimensional optimization for text summarization via reinforcement learning . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5858--5871, Bangkok, Thailand. Associa...

    doi:10.18653/v1/2024.acl-long.319 2024

  52. [52]

    Nihar B Shah and Martin J Wainwright. 2018. https://www.jmlr.org/papers/volume18/16-206/16-206.pdf Simple, robust and optimal ranking from pairwise comparisons . Journal of machine learning research, 18(199):1--38

    2018

  53. [53]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024. https://arxiv.org/pdf/2402.03300 Deepseekmath: Pushing the limits of mathematical reasoning in open language models . arXiv preprint arXiv:2402.03300

    Pith/arXiv arXiv 2024

  54. [54]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf Learning to summarize with human feedback . Advances in neural information processing systems, 33:3008--3021

    2020

  55. [55]

    Harm Van Seijen, Mehdi Fatemi, Joshua Romoff, Romain Laroche, Tavian Barnes, and Jeffrey Tsang. 2017. https://proceedings.neurips.cc/paper/2017/hash/1264a061d82a2edae1574b07249800d6-Abstract.html Hybrid reward architecture for reinforcement learning . Advances in neural information processing systems, 30

    2017

  56. [56]

    Li Wang, Junlin Yao, Yunzhe Tao, Li Zhong, Wei Liu, and Qiang Du. 2018. https://dl.acm.org/doi/abs/10.5555/3304222.3304389 A reinforced topic-aware convolutional sequence-to-sequence model for abstractive text summarization . In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4453--4460

    arXiv 2018

  57. [57]

    Noah Wang, Feiyu Duan, Yibo Zhang, Wangchunshu Zhou, Ke Xu, Wenhao Huang, and Jie Fu. 2024 a . https://doi.org/10.18653/v1/2024.findings-emnlp.983 P osition ID : LLM s can control lengths, copy and paste with explicit positional awareness . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16877--16915, Miami, Florida, USA. A...

    work page doi:10.18653/v1/2024.findings-emnlp.983 2024

  58. [58]

    Pancheng Wang, Shasha Li, Kunyuan Pang, Liangliang He, Dong Li, Jintao Tang, and Ting Wang. 2022. https://aclanthology.org/2022.coling-1.543/ Multi-document scientific summarization from a knowledge graph-centric view . In Proceedings of the 29th International Conference on Computational Linguistics, pages 6222--6233, Gyeongju, Republic of Korea. Internat...

    2022

  59. [59]

    Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, and Shafiq Joty. 2024 b . https://arxiv.org/abs/2409.14664 Direct judgement preference optimization . arXiv preprint arXiv:2409.14664

    Pith/arXiv arXiv 2024

  60. [60]

    Zhengjue Wang, Zhibin Duan, Hao Zhang, Chaojie Wang, Long Tian, Bo Chen, and Mingyuan Zhou. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.35 Friendly topic assistant for transformer based abstractive summarization . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 485--497, Online. Association fo...

    work page doi:10.18653/v1/2020.emnlp-main.35 2020

  61. [61]

    Yang Wei. 2012. https://ieeexplore.ieee.org/abstract/document/6234047/?casa_token=qTeTZeMFM1sAAAAA:29hTwhR4cIjqd6W0yj9MeOcvz27jcwKlbgiwR-28vLqdDQn_kc2r5o02oQ4OJQNZuxcTl6ODng Document summarization method based on heterogeneous graph . In 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery, pages 1285--1289. IEEE

    arXiv 2012

  62. [62]

    Yuxiang Wu and Baotian Hu. 2018. https://ojs.aaai.org/index.php/AAAI/article/view/11987/11846 Learning to extract coherent summary via deep reinforcement learning . In Proceedings of the AAAI conference on artificial intelligence, volume 32

    2018

  63. [63]

    Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2022. https://doi.org/10.18653/v1/2022.acl-long.360 PRIMERA : Pyramid-based masked sentence pre-training for multi-document summarization . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5245--5263, Dublin, Ireland. Associat...

    work page doi:10.18653/v1/2022.acl-long.360 2022

  64. [64]

    Austin Xu, Yilun Zhou, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. 2025. https://arxiv.org/pdf/2505.13346 J4r: Learning to judge with equivalent initial state group relative policy optimization . arXiv preprint arXiv:2505.13346

    Pith/arXiv arXiv 2025

  65. [65]

    Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. http://proceedings.mlr.press/v119/zhang20ae/zhang20ae.pdf Pegasus: Pre-training with extracted gap-sentences for abstractive summarization . In International conference on machine learning, pages 11328--11339. PMLR

    2020

  66. [66]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2019

  67. [67]

    Chao Zhao, Tenghao Huang, Somnath Basu Roy Chowdhury, Muthu Kumar Chandrasekaran, Kathleen McKeown, and Snigdha Chaturvedi. 2022. https://doi.org/10.18653/v1/2022.findings-acl.51 Read top news first: A document reordering approach for multi-document news summarization . In Findings of the Association for Computational Linguistics: ACL 2022, pages 613--621...

    work page doi:10.18653/v1/2022.findings-acl.51 2022

  68. [68]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, and 1 others. 2023. https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena . In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 46595--46623

    Pith/arXiv arXiv 2023

  69. [69]

    Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. https://doi.org/10.18653/v1/2021.naacl-main.472 QMS um: A new benchmark for query-based multi-domain meeting summarization . In Proceedings of the 2021 Conference of the North American Chapter of th...

    doi:10.18653/v1/2021.naacl-main.472 2021

  70. [70]

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. https://arxiv.org/abs/1909.08593 Fine-tuning language models from human preferences . arXiv preprint arXiv:1909.08593

    Pith/arXiv arXiv 2019

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.