Topic Augmented Generator for Abstractive Summarization

Bowen Zhang; Fei Sha; Melissa Ailem

REVIEW 4 major objections 4 minor 21 references

Topic Augmented Generator for Abstractive Summarization

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Conditioning an abstractive summarization decoder on the document's LDA topic vector improves summary quality, yielding higher ROUGE scores and closer topic coherence to the source text on CNN/Daily Mail and WikiHow.

desk verdict A clean, simple extension of pointer-generator with an LDA topic component, but the empirical case rests on weaker in-house baselines and no significance tests. read the letter →

arxiv 1908.07026 v1 pith:RYUOTVH3 submitted 2019-08-19 cs.LG stat.ML

Melissa Ailem , Bowen Zhang , Fei Sha This is my paper · ORCID

classification cs.LGstat.ML

keywords abstractivesummarizationtopicmodellatentDirichletallocationpointer-generatornetworkdecoderconditioningCNN/DailyMailWikiHowROUGE

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the output summaries of attention-based sequence-to-sequence models improve when the decoder is conditioned not only on the input text but also on the document's latent topics, extracted with LDA. The proposed Topic Augmented Generator replaces the pure pointer-generator distribution at each step with a mixture of that distribution and a topic-conditioned word distribution, so the decoder can draw on corpus-level word co-occurrence patterns beyond the local attention context. The authors report higher ROUGE-1, ROUGE-2, and ROUGE-L scores for TAG and TAG+Cov over the pointer-generator baselines on both CNN/Daily Mail and WikiHow. They additionally report that their summaries have lower KL divergence to the source document's LDA topic distribution than the baseline summaries, which they take as evidence that the generated text better preserves what the document is about.

What carries the argument

The load-bearing mechanism is a per-step mixture between the pointer-generator decoder and a topic-conditioned softmax. LDA supplies the document-level summary $\theta^*$: each document is a distribution over $K$ latent topics and each topic a distribution over words, and $\theta^*$ is inferred from the input document. The topic term $q(\mu_{y_t}^\top \theta^*)$ then lets the decoder assign probability to words according to how well they fit the document's inferred topic mixture, overriding the narrow view of the attention context when the mixture weight $\lambda_t$ favors it.

What would settle it

Retrain TAG on CNN/Daily Mail with the topic switch fixed at $\lambda_t = 1$, so the topic term never contributes; if ROUGE scores stay at TAG levels, the reported gains do not come from topic conditioning.

Watch

Extended reading notes

Core claim

The paper's central claim is that a decoder gains from a global, document-level semantic vector that local attention does not expose. Formally, TAG writes the per-word generation probability as $p(y_t\mid x,y_{<t}) = \lambda_t p^{\mathrm{PG}}_t + (1-\lambda_t) q(\mu_{y_t}^\top \theta^*)$, where $p^{\mathrm{PG}}_t$ is the pointer-generator probability, $\theta^*$ is the LDA topic vector inferred for the input document, $\mu$ is a topic-word matrix initialized from the LDA topic-word matrix $\beta$ and then fine-tuned end-to-end, and $\lambda_t$ is a learned switching weight. The topic term biases generation toward words that are coherent with the document's overall themes. Empirically, the paper reports that this added term improves ROUGE scores on both CNN/Daily Mail and WikiHow, and that generated summaries are closer to the original document's topic distribution than those of the pointer-generator baseline; on CNN/Daily Mail, the TAG+Cov summaries are even closer to the source topic distribution than the ground-truth summaries.

Load-bearing premise

The load-bearing assumption is that the topic-word matrix fine-tuned inside the decoder remains compatible with the original LDA parameters used to infer each document's topic vector, so the topic signal the decoder receives still matches the topics that describe the document.

Editorial extensions

If this is right

Topic conditioning helps on both a news corpus and a more abstractive how-to corpus, so the benefit is not tied to one genre of summaries.
The topic component is trainable with ordinary maximum-likelihood sequence learning, so it can be added to an existing pointer-generator decoder without changing the training objective or requiring extra supervision.
Because test documents are not used to fit LDA, the model can infer a fresh topic vector for any new input at inference and use it in the same decoder.
The reported improvement over the baseline on more than half of the CNN/Daily Mail test documents suggests the effect is systematic rather than driven by a few examples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not ablate the end-to-end fine-tuning of $\mu$; a follow-up could freeze $\mu$ at LDA's $\beta$ and test whether the ROUGE gain survives, which would isolate the value of the topic signal from the value of aligning it with the decoder.
Because the KL-coherence analysis treats the LDA topic space as ground truth, one could test whether ROUGE gains concentrate on documents with confidently inferred topic vectors; if they do, topic fidelity is an evaluation axis that ROUGE only partially captures.
The same mixture-of-distributions idea could apply to other conditional generation tasks with a document-level latent variable, such as style or domain conditioning, but the paper only demonstrates the mechanism for topic-conditioned summarization.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Brief take: this is a clear and simple proposal—add the document-level LDA topic vector as a third mixture component in the pointer-generator decoder. The specific design (Eq. 6) is new relative to Wang et al. (2019), who inject topic-derived embeddings into attention but discard θ*. The paper is honest that attention injection gave only minor gains in their early experiments. That is useful context.

What the paper does well: the idea is easy to explain, the model is a natural extension, and the authors report results on two datasets including WikiHow, which is less common. The qualitative examples show TAG+Cov producing more complete summaries. The coherence boxplots (Figure 1) are suggestive.

Soft spots, in order of seriousness. First, the empirical claim is not yet established. The headline says "strongly improved ROUGE" over state-of-the-art, but the comparisons are against the authors' own PG and PG+Cov implementations, which score below the published numbers (PG 35.73 vs 36.44; PG+Cov 39.12 vs 39.53 on ROUGE-1). Against the published PG+Cov, TAG+Cov improves ROUGE-1 by 0.53 and ROUGE-L by 0.14 on CNN/DM, and it still trails Lead-3 on ROUGE-1 and ROUGE-L. No error bars, confidence intervals, or paired tests. The per-document win count (5888/11490) is 51.2%, not tested against chance. The "strongly improved" phrasing is out of proportion to the measured effect relative to published baselines. Second, the coherence evaluation is partly circular: the same LDA model is used to condition the decoder and to compute the KL divergences, so lower KL for TAG is baked in. Third, details are deferred to a supplementary file that is not present; as a reader I can't reproduce the setup without more specifics (e.g., LDA training details, hyperparameters, exact inference procedure). The potential μ/β mismatch after end-to-end fine-tuning is worth noting but I don't think it's fatal—the model still works as a learned mixture. Minor: no analysis of output diversity or repetition beyond the qualitative examples.

Who is this for? Researchers working on topic-informed text generation or summarization. The method is simple enough to try in a follow-up, but I'd want code or significance tests before building on the numbers.

My recommendation: send it to peer review, but the authors should be pushed to release code, add significance tests, compare against published baseline numbers, and temper the "strongly improved" claim. The core idea is sound.

Referee Report

4 major / 4 minor

Summary. The paper proposes Topic Augmented Generator (TAG), a decoder modification for abstractive summarization that adds a mixture component based on the document-level LDA topic vector θ*. The generative probability in Eq. (6) interpolates between the pointer-generator distribution and a softmax over the topic-word matrix µ applied to θ*. The method is evaluated on CNN/Daily Mail and WikiHow, with reported ROUGE improvements over in-house pointer-generator (PG) and PG+Cov baselines, and with a topic-coherence analysis using KL divergence between topic vectors of documents and generated summaries. The paper claims that conditioning on latent topics yields 'strongly improved ROUGE scores compared to state-of-the-art models.'

Significance. If the empirical claims were established, the proposed idea would be a simple and potentially useful inductive bias for abstractive summarization, and the paper provides a clean mathematical formulation. The approach is plausible and worth further study. However, the current evidence is not sufficient: the ROUGE gains are measured against re-implementations of the baselines that are weaker than the published numbers, no significance testing or error bars are provided, and the secondary coherence evaluation is circular because it uses the same LDA model that conditions the decoder. The manuscript also defers key implementation details to an unavailable supplementary file. The paper has strengths in its clear presentation of the method and the use of two datasets, but the headline claim is not yet substantiated.

major comments (4)

[Table 1, Section 4.2] The central empirical claim rests on comparisons to the authors' own PG and PG+Cov baselines, which are below the published numbers in See et al. (2017). For CNN/DM, TAG+Cov improves over the published PG+Cov by only 0.53 ROUGE-1 (40.06 vs 39.53) and 0.14 ROUGE-L (36.52 vs 36.38), and it remains below the Lead-3 extractive baseline on ROUGE-1 (40.06 vs 40.34) and ROUGE-L (36.52 vs 36.57). No confidence intervals, bootstrap intervals, or paired significance tests are reported, and the per-document win rate of 5888/11490 (51.2%) is not tested against chance. The claim of 'strongly improved ROUGE scores' is therefore not established by the evidence presented.
[Section 4.2, Figure 1] The KL-divergence coherence evaluation is circular: the same LDA model used to condition the TAG decoder is also used to infer the topic vectors θ* for both documents and generated summaries. Since TAG is explicitly trained to make q(µ^T θ*) large for generated words, its summaries will naturally have topic vectors closer to the document's θ* than summaries from a model not conditioned on θ*. The lower KL divergence for TAG is thus partly an artifact of the evaluation protocol, not independent evidence of better topic preservation. The authors should evaluate topic coherence with a held-out topic model, an independently trained topic model, or human judgments.
[Section 2.2, Eq. (6)] The topic vector θ* is inferred using the original LDA parameters, while the decoder's topic-word matrix µ is initialized from β and then fine-tuned end-to-end. If fine-tuning moves µ away from β, the topic component q(µ^T θ*) may become mismatched with the topic vector used at inference, potentially weakening the mechanism or making the interpolation in Eq. (6) less meaningful. The paper should quantify this drift (for example, by reporting the distance between µ and β after training) or include an ablation where µ is kept fixed at β.
[Section 4.1 and Supplementary Material] Key implementation details are deferred to a 'Supplementary Material' that is not available with the manuscript. This includes the number of LDA topics K, the LDA training procedure, hyperparameters, vocabulary size, optimization schedule, and the specific setup for the WikiHow baselines. Without these details, the experiments cannot be reproduced, and it is impossible to assess whether the in-house PG and PG+Cov baselines are faithful implementations of See et al. (2017). The per-document win rate also lacks any statistical test; a sign test or bootstrap would be needed to support the claim that TAG+Cov is better on more than half of the test documents.

minor comments (4)

[Abstract] The sentence 'The latent topics ... reveals more global semantic information' has a subject-verb agreement error: the plural subject 'topics' should take the plural verb 'reveal.'
[Section 4.2] The phrase 'TAG+Covturn out to be' is missing a space between 'Cov' and 'turn'.
[Figure 1] The boxplots would be easier to interpret if the caption stated the number of documents used in each box and whether outliers are shown or omitted.
[Section 4.2] The speculation that ground-truth news summaries are 'too concise' for topic inference is presented without any supporting evidence; either provide a quantitative analysis or move this remark to future work.

Circularity Check

1 steps flagged · score 4.0 of 10

ROUGE claim is externally grounded, but the secondary topic-coherence evaluation is partly self-confirming because it measures proximity to the same LDA topic vector used as the decoder's conditioning input.

self definitional [Section 4.2, 'Quantitative results' (KL-divergence/coherence evaluation) vs. Eq. (6) and Section 2.2 'LDA'/'TAG']
"for each test document x, we compute the Kullback-Leibler divergence between the topic distributions θ∗ inferred on the original document and on the reference as well as the generated summaries. As shown by the boxplots ... TAG+Cov tends to generate summaries that are more coherent with those of the original documents."

The coherence metric compares θ* of the original document with θ* of the generated summary, both inferred with the same LDA model. But the decoder in Eq. (6) is explicitly conditioned on θ* of the original document via the topic component q(µ^T θ*), with µ initialized from the LDA topic-word matrix β. A summary produced by drawing words from this component is structurally biased to have an inferred LDA topic vector close to the conditioning θ*, so lower KL divergence for TAG+Cov is partly a byproduct of the model definition rather than an independent measure of summary quality. The ROUGE evaluation, by contrast, is against fixed gold summaries and is not circular.

full rationale

The central derivation and main empirical claim are not circular. Eq. (6) is a new mixture distribution p(yt|x,y<t) = λt pPG_t + (1-λt) q(µ^T θ*); its parameters are learned by maximum likelihood against document-summary training pairs, and the ROUGE scores in Table 1 are computed against fixed reference summaries that do not depend on the model's fitted quantities. The authors also explicitly avoid information leak by not fitting LDA on test samples. The only significant circularity is the secondary coherence evaluation in Section 4.2: the quantity being measured (KL divergence in LDA topic space) is the same quantity used to condition the decoder, so the TAG models' apparent topic coherence is partially self-confirming. The weakened reproduced baselines and missing significance tests are correctness concerns, not circularity, and do not raise the circularity score further.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard seq2seq assumptions, an empirical bet that LDA topics help generation, an ad-hoc mixture architecture, and an unexamined compatibility assumption between the LDA-inferred topic vector and the fine-tuned topic-word matrix. No new entities are introduced.

free parameters (1)

Number of LDA topics K = 100
Chosen by hand for both datasets with no sensitivity analysis; it determines the dimensionality of the topic vector that conditions the decoder.

assumptions (5)

standard math The summary distribution factorizes as a Markovian product (Eq. 1).
Standard autoregressive factorization used by all seq2seq summarization models; not unique to this paper.
domain assumption LDA topics fitted on the training corpus capture semantic structure useful for summarization.
Central inductive bias of the paper; empirically tested via ROUGE and coherence, but not proven.
ad hoc to paper The mixture in Eq. (6) of pointer-generator and topic softmax is a valid probability distribution and a beneficial inductive bias.
The specific linear mixture and learned switching is chosen by the authors; no theoretical justification is given.
ad hoc to paper The document topic vector θ* inferred with the original LDA parameters remains compatible with the fine-tuned topic-word matrix µ in the decoder.
The paper initializes µ from β and fine-tunes it, while θ* is inferred with the original β; the compatibility of the two is never analyzed.
domain assumption ROUGE scores are a reliable proxy for summary quality.
Standard evaluation in summarization, though known to correlate imperfectly with human judgment.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Topic Augmented Generator for Abstractive Summarization." pith.science (2026). https://pith.science/paper/RYUOTVH3

@misc{pith2026190807026,
  author       = {Pith},
  title        = {Pith review of: Topic Augmented Generator for Abstractive Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RYUOTVH3}},
  note         = {Machine review of arXiv:1908.07026}
}

read the original abstract

Steady progress has been made in abstractive summarization with attention-based sequence-to-sequence learning models. In this paper, we propose a new decoder where the output summary is generated by conditioning on both the input text and the latent topics of the document. The latent topics, identified by a topic model such as LDA, reveals more global semantic information that can be used to bias the decoder to generate words. In particular, they enable the decoder to have access to additional word co-occurrence statistics captured at document corpus level. We empirically validate the advantage of the proposed approach on both the CNN/Daily Mail and the WikiHow datasets. Concretely, we attain strongly improved ROUGE scores when compared to state-of-the-art models.

Figures

Figures reproduced from arXiv: 1908.07026 by the authors.

**Figure 2.** From CNN/DM. Our TAG+Cov model generates 3 main sentences instead of 2 for PG+Cov. Ground Truth: acquire a pot. gather the ingredients needed to make the curry. walk to the pot on your kitchen counter. choose the ingredients that you need for the recipe. PG+Cov: go to the kitchen counter. go to your kitchen counter. look for the ingredients you want to cook. finished. take care of your health. TAG+Cov: acquire a pot… view at source ↗

**Figure 3.** From WikiHow (“How to Make Vegetable Curry in Harvest Moon Animal Parade”). 5 Conclusion We have shown that by conditioning on the topics underlying the input documents, the decoder generates noticeably improved summaries. This suggests that the supervised learning of represen- [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 7 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR

work page 2015
[4]

David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993--1022

2003
[5]

Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93--98

2016
[6]

Bonnie Dorr, David Zajic, and Richard Schwartz. 2003. Hedge trimmer: A parse-and-trim approach to headline generation. In Proceedings of the HLT-NAACL 03 on Text summarization workshop-Volume 5, pages 1--8. Association for Computational Linguistics

work page 2003
[7]

Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693--1701

2015
[8]

Mahnaz Koupaee and William Yang Wang. 2018. Wikihow: A large scale text summarization dataset. arXiv preprint arXiv:1810.09305

arXiv 2018

Show all 21 references

[9]

Julian Kupiec, Jan Pedersen, and Francine Chen. 1999. A trainable document summarizer. Advances in Automatic Summarization, pages 55--60

1999
[10]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out

2004
[11]

Linqing Liu, Yao Lu, Min Yang, Qiang Qu, Jia Zhu, and Hongyan Li. 2018. Generative adversarial network for abstractive text summarization. In Thirty-Second AAAI Conference on Artificial Intelligence

2018
[12]

Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Thirty-First AAAI Conference on Artificial Intelligence

2017
[13]

Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. Computational Natural Lan-guage Learning

2016
[14]

Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304

2017 arXiv
[15]

Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. EMNLP

2015
[16]

Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368

2017 arXiv
[17]

Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. arXiv preprint arXiv:1601.04811

2016 arXiv
[18]

Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692--2700

2015
[19]

Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. 2019. Topic-guided variational autoencoders for text generation. arXiv preprint arXiv:1903.07137

2019 arXiv
[20]

Yau-Shian Wang and Hung-Yi Lee. 2018. Learning to encode text as human-readable summaries using generative adversarial networks. arXiv preprint arXiv:1810.02851

2018 arXiv
[21]

Wenyuan Zeng, Wenjie Luo, Sanja Fidler, and Raquel Urtasun. 2016. Efficient summarization with read-again and copy mechanism. arXiv preprint arXiv:1611.03382

2016 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR

work page 2015

[4] [4]

David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993--1022

2003

[5] [5]

Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93--98

2016

[6] [6]

Bonnie Dorr, David Zajic, and Richard Schwartz. 2003. Hedge trimmer: A parse-and-trim approach to headline generation. In Proceedings of the HLT-NAACL 03 on Text summarization workshop-Volume 5, pages 1--8. Association for Computational Linguistics

work page 2003

[7] [7]

Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693--1701

2015

[8] [8]

Mahnaz Koupaee and William Yang Wang. 2018. Wikihow: A large scale text summarization dataset. arXiv preprint arXiv:1810.09305

arXiv 2018