REVIEW 4 major objections 6 minor 39 references
An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation
T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Imitation learning beats reinforcement learning for paraphrase generation.
desk verdict Useful, reproducible IL-vs-RL comparison for paraphrase generation; the fixed-alpha DAgger result is credible, but 'constantly better' overstates the paper's own tables. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a unified online-learning objective parameterized by two schedule rates. DAgger (dataset aggregation) is an imitation-learning algorithm that, at each decoding step, mixes ground-truth words and the model's own argmax predictions as inputs while always supervising against the ground-truth output; in this paper it is treated as equivalent to scheduled sampling. The rate $\alpha$ controls the probability of feeding the ground-truth previous word as decoder input, and $\beta$ controls the probability of using the ground-truth word as the training target. Setting $\alpha = \beta = 1$ gives maximum-likelihood training, $\alpha = \beta = 0$ with random-sampling rollouts gives REINFORCE, and $\beta = 1$ with $0 < \alpha < 1$ gives DAgger. This machinery turns the comparison between RL and IL into a comparison of schedule-rate choices, so the observed performance differences are attributable to the learning signal rather than to the architecture.
What would settle it
Re-run the comparison with identical schedule-rate budgets: train DAgger with $\alpha$ decayed to zero and REINFORCE-SIO with fixed $\alpha$ (and comparable $\beta$), then compare average scores on the same Quora and Twitter splits. If REINFORCE-SIO with fixed rates matches or exceeds DAgger, the claimed constant superiority of IL over RL would be falsified; if DAgger stays ahead under both tunings, the claim holds.
Extended reading notes
Core claim
The central claim is that for paraphrase generation with a pointer-generator model, imitation learning outperforms reinforcement learning on both datasets tested, and that a DAgger variant with a fixed schedule rate (0.5 on Quora, 0.2 on Twitter) sets a new state of the art, beating prior systems by about 13 percent on average score. The paper reaches this claim through a unified objective in which the schedule rates ($\alpha$, $\beta$) and the decoding function interpolate continuously among MLE, REINFORCE, and DAgger, so that comparing special cases isolates the effect of the training signal. It finds that replacing ground-truth inputs with sampled outputs during training, and weighting the gradient by a ROUGE-2 reward, gives only small gains, whereas always supervising against ground-truth outputs while mixing the decoder input between expert and decoded words gives the largest gain. The authors interpret this as evidence that imitation learning's more stable supervision is better suited to this task than reinforcement learning's high-variance reward.
Load-bearing premise
The conclusion that imitation learning is consistently better depends on the chosen schedule rates: the winning DAgger setting keeps $\alpha$ fixed, while the closest RL variant, REINFORCE-SIO, decays both rates to zero, and when DAgger's $\alpha$ is also decayed to zero its average score dips just below that RL variant.
Editorial extensions
If this is right
- If DAgger is reliably better, practitioners fine-tuning sequence-to-sequence paraphrasers should prefer scheduled-sampling-style imitation learning over policy-gradient reinforcement learning by default.
- Keeping some expert input throughout fine-tuning appears to matter: on Quora, DAgger with $\alpha$ decayed to zero scores 57.22, while fixed $\alpha = 0.5$ scores 58.02.
- Imitation-learning fine-tuning adds a few average-score points over MLE pre-training on both datasets, whereas reinforcement learning's margin over MLE is small, about 0.18 average points on Quora.
- The unified $(\alpha, \beta)$ family provides a practical two-parameter grid for diagnosing whether a text-generation task benefits more from expert supervision or from reward-driven exploration.
Reading between the lines
- The schedule-rate dependence suggests the measured IL-versus-RL gap may be partly a proxy for the gap between fixed and decayed mixing rates; testing REINFORCE-SIO with fixed nonzero $\alpha$ would clarify whether the family, not the label, drives the gain.
- The same $(\alpha, \beta)$ grid could be applied to summarization or machine translation as a cheap diagnostic before designing a custom reward.
- A testable extension would be to give every algorithm the same schedule-rate budget: tune REINFORCE-SIO with the fixed rates that win for DAgger and decay DAgger's $\alpha$ the way REINFORCE-SIO decays its rates; if the ordering reverses, the constant-superiority conclusion is tuning-dependent.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an empirical comparison of imitation learning (IL) and reinforcement learning (RL) methods for paraphrase generation, using a pointer-generator base model and proposing a unified objective (Eq. 2) in which REINFORCE, DAgger, MLE, and several intermediate variants arise as special cases through schedule rates (alpha, beta) and a decode function. Experiments on the Quora and Twitter paraphrase datasets compare these fine-tuning algorithms against each other and against prior systems. The paper claims that (1) IL is constantly better than RL and (2) the best IL setting, DAgger* with a fixed alpha, outperforms prior state-of-the-art methods by about 13% on average score.
Significance. If the central claims held, the paper would provide a practically useful comparison of training objectives for paraphrase generation and a concrete recommendation in favor of DAgger-style scheduled sampling over REINFORCE-style policy gradient fine-tuning. The paper has clear strengths: it releases code and data, formulates several algorithms in one framework, and reports results on two benchmark datasets with a consistent base model and evaluation protocol. The comparison is, however, weakened by an asymmetry in hyperparameter tuning, small performance margins without variance estimates, and an abstract-level claim that is contradicted by the paper's own Table 1. With corrected claims and additional targeted experiments, the finding that a fixed-alpha DAgger setting improves over the pre-trained MLE baseline would be a useful empirical contribution.
major comments (4)
- [Abstract and §3, Table 1] The abstract's claim that 'imitation learning is constantly better than reinforcement learning' is not supported by Table 1. Under comparable schedule-rate settings, DAgger (line 10, alpha decays to 0, beta=1) scores 57.22 on average, while REINFORCE-SIO (line 9, alpha and beta decay to 0) scores 57.26. The only IL configuration that beats all RL variants is DAgger* (line 11) with a fixed alpha=0.5. The claim should be revised to state that a fixed-alpha DAgger setting achieved the best results, and that the IL/RL ranking depends on schedule-rate choices.
- [§2.2, §3, and Appendix B] The IL-vs-RL comparison is confounded with hyperparameter tuning. RL variants (REINFORCE, REINFORCE-GTI, REINFORCE-SO, REINFORCE-SIO) use schedule rates that decay to 0 by default, and no fixed-alpha or otherwise wide sweep over schedule rates is reported for these RL methods. In contrast, Appendix B reports schedule-rate sweeps for DAgger only (Tables 3-4). The conclusion that IL is better than RL therefore compares a tuned IL method against default RL configurations. The authors should either report schedule-rate sweeps for the RL variants or explicitly scope the conclusion to the tested configurations.
- [§3, Tables 1-2 and Appendix A] The reported differences between methods are small in several key comparisons, and no error bars, multiple-seed runs, or significance tests are provided. For example, on Quora the gap between REINFORCE-SIO (57.26) and DAgger with decayed alpha (57.22) is 0.04 points, and on Twitter the best DAgger* (47.44) is only 0.19 points ahead of REINFORCE-SIO (47.25). Given these margins, the claim that IL is 'constantly better' needs variance estimates or statistical testing. Relatedly, ROUGE-2 is used both as the training reward (Appendix A, Eq. 5) and as one of the three evaluation metrics, so part of the observed ROUGE-2 gain may reflect reward optimization; the authors should state this explicitly and give primary weight to the metrics not used for training.
- [§3, Result Analysis and Tables 1-2] The 'large margin' state-of-the-art claim on Quora is partly an artifact of a stronger MLE baseline. All fine-tuned models are compared to competitive systems reprinted from prior work, but the pre-trained MLE baseline itself already averages 56.14, well above RbM (48.68). The paper notes that the development set was run on-the-fly, which may explain this gap. To support the SOTA claim, the comparison should be controlled: either reproduce prior systems under the same training and evaluation protocol, or explicitly frame the result as an improvement over the reproduced pointer-generator baseline rather than over published numbers from other setups.
minor comments (6)
- [Abstract] 'constantly better' should be 'consistently better' or 'better in all tested settings'.
- [§2.1] The sentence 'That implies taking an action based on the current observation' is missing a period and should be merged with the preceding sentence.
- [§2.1 and §2.2] The phrase 'the the policy function' contains a duplicated 'the' and should be corrected.
- [§2] The pointer-generator is described as 'state-of-the-art model on paragraph generation' but the paper is about paraphrase generation; this should be corrected.
- [Appendix B, Tables 3-4] Table numbering is inconsistent: both Tables 3 and 4 are labeled, but the row numbering is duplicated (both contain rows numbered 1, 2, 4, 4), and the header 'k alpha' is not explained in the table caption.
- [§3, Table 1] For Dis-LSTM, ROUGE-1 is missing and the average score (45.30) does not obviously correspond to the two reported values; the source and exact metric configuration should be clarified in a footnote.
Circularity Check
No circularity: the paper is an empirical comparison with no derivation chain that reduces to its own inputs.
full rationale
I examined the abstract, method section, unified objective in Equation 2, Algorithm 1, the schedule-rate definitions, and the results tables in Sections 3, Appendix A, and Appendix B. The paper makes no formal derivation from first principles; it defines a unified learning objective and shows that REINFORCE, DAgger, and MLE correspond to particular choices of (alpha, beta) and decoding functions. That is a taxonomic unification of existing algorithms, not a claim that one algorithm is derived from another, so it cannot be circular by construction. The main stated results are empirical: DAgger* with a fixed alpha outperforms the RL variants and previous systems on Quora and Twitter. These numbers come from actual experiments against external baselines, and the best DAgger* alpha values are selected in the experiments reported in Appendix B, not renamed from the training reward or from a fitted parameter. The use of ROUGE-2 as both the RL reward and one of the evaluation metrics is a potential reward-overlap concern, but it is not circularity under the criteria here: the ROUGE-2 gains are measured on held-out test sets and are accompanied by simultaneous gains on ROUGE-1 and BLEU, which are not optimized directly. The skeptical observation that the IL-vs-RL ranking depends on schedule-rate choices and that decayed DAgger slightly trails REINFORCE-SIO in Table 1 is a substantive experimental-validity criticism, not a reduction of the conclusion to its own inputs. No load-bearing self-citations, imported uniqueness theorems, or ansatz-smuggling citations appear. Therefore the paper deserves a circularity score of 0.
Assumptions & free parameters
free parameters (5)
- Schedule rate alpha =
0.5 (Quora), 0.2 (Twitter); decay k_alpha=0.9999 for DAgger
- Schedule rate beta =
1 for DAgger; 0 for REINFORCE; inverse-sigmoid decay with k=3000 for variants
- Baseline sample size N =
4
- Beam size =
8
- Model hyperparameters =
hidden 256, embedding 128, vocab 5k, max length 20
assumptions (4)
- domain assumption The pointer-generator is a strong base model for paraphrase generation
- domain assumption ROUGE-2 is an appropriate reward for training a paraphraser
- domain assumption Scheduled sampling can be treated as a form of DAgger imitation learning
- domain assumption MLE pre-training is necessary for RL and IL fine-tuning to work
Cite this review
Pith. "Pith review of An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation." pith.science (2026). https://pith.science/paper/D2ROLFUL
@misc{pith2026190810835,
author = {Pith},
title = {Pith review of: An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/D2ROLFUL}},
note = {Machine review of arXiv:1908.10835}
}
read the original abstract
Generating paraphrases from given sentences involves decoding words step by step from a large vocabulary. To learn a decoder, supervised learning which maximizes the likelihood of tokens always suffers from the exposure bias. Although both reinforcement learning (RL) and imitation learning (IL) have been widely used to alleviate the bias, the lack of direct comparison leads to only a partial image on their benefits. In this work, we present an empirical study on how RL and IL can help boost the performance of generating paraphrases, with the pointer-generator as a base model. Experiments on the benchmark datasets show that (1) imitation learning is constantly better than reinforcement learning; and (2) the pointer-generator models with imitation learning outperform the state-of-the-art methods with a large margin.
Figures
Reference graph
Works this paper leans on
-
[1]
JA Bagnell, Joel Chestnutt, David M Bradley, and Nathan D Ratliff. 2007. Boosting structured prediction for imitation learning. In Advances in Neural Information Processing Systems, pages 1153--1160
work page 2007
-
[2]
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016. An actor-critic algorithm for sequence prediction. arXiv preprint arXiv:1607.07086
arXiv 2016
-
[3]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473
arXiv 2014
-
[4]
Regina Barzilay and Lillian Lee. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL '03, pages 16--23, Stroudsburg, PA, USA. Association for Computational...
work page 2003
-
[5]
Regina Barzilay and Kathleen R McKeown. 2001. Extracting paraphrases from a parallel corpus. In Proceedings of the 39th annual meeting of the Association for Computational Linguistics
work page 2001
-
[6]
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, pages 1171--1179
2015
-
[7]
Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Wojciech Gajewski, Andrea Gesmundo, Neil Houlsby, and Wei Wang. 2017. Ask the right questions: Active question reformulation with reinforcement learning. arXiv preprint arXiv:1705.07830
arXiv 2017
-
[8]
Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daum \'e III, and John Langford. 2015. Learning to search better than your teacher. arXiv preprint arXiv:1502.02206
work page Pith review arXiv 2015
Show all 39 references
-
[9]
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. arXiv preprint arXiv:1805.11080
2018 arXiv
-
[10]
Kyunghyun Cho, Aaron Courville, and Yoshua Bengio. 2015. Describing multimedia content using attention-based encoder-decoder networks. IEEE Transactions on Multimedia, 17(11):1875--1886
2015
-
[11]
Kyunghyun Cho, Bart Van Merri \"e nboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078
2014 arXiv
-
[12]
Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie...
2016
-
[13]
Hal Daum \'e , John Langford, and Daniel Marcu. 2009. Search-based structured prediction. Machine learning, 75(3):297--325
2009
-
[14]
Peter Dayan and Yael Niv. 2008. Reinforcement learning: the good, the bad and the ugly. Current opinion in neurobiology, 18(2):185--196
2008
-
[15]
Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017. Learning to paraphrase for question answering. arXiv preprint arXiv:1708.06022
2017 arXiv
-
[16]
Ali Ibrahim, Boris Katz, and Jimmy Lin. 2003. Extracting structural paraphrases from aligned monolingual corpora. In Proceedings of the second international workshop on Paraphrasing-Volume 16, pages 57--64
2003
-
[17]
Yaser Keneshloo, Tian Shi, Naren Ramakrishnan, and Chandan K. Reddy. 2018. Deep reinforcement learning for sequence to sequence models. arXiv preprint arXiv:1805.09461
2018 arXiv
-
[18]
Wuwei Lan, Siyu Qiu, Hua He, and Wei Xu. 2017. A continuously growing dataset of sentential paraphrases. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1224--1234, Copenhagen, Denmark. Association for Computational Linguistics
2017
-
[19]
Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li. 2018. Paraphrase generation with deep reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
2018
-
[20]
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out
2004
-
[21]
Ming Liu, Wray Buntine, and Gholamreza Haffari. 2018. Learning how to actively learn: A deep imitation learning approach. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1874--1883
2018
-
[22]
Nitin Madnani, Necip Fazil Ayan, Philip Resnik, and Bonnie J Dorr. 2007. Using paraphrases for parameter tuning in statistical machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 120--127. Association for Computational Linguistics
2007
-
[23]
Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing revisited with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 881--893
2017
-
[24]
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730
2018 arXiv
-
[25]
Bo Pang, Kevin Knight, and Daniel Marcu. 2003. Syntax-based alignment of multiple translations: Extracting paraphrases and generating new sentences. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human La...
2003
-
[26]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311--318. Association for Computational Linguistics
2002
-
[27]
Badri N Patro, Vinod K Kurmi, Sandeep Kumar, and Vinay P Namboodiri. 2018. Learning semantic sentence embeddings using pair-wise discriminator. arXiv preprint arXiv:1806.00807
2018 arXiv
-
[28]
Aaditya Prakash, Sadid A Hasan, Kathy Lee, Vivek Datla, Ashequl Qadir, Joey Liu, and Oladimeji Farri. 2016. Neural paraphrase generation with stacked residual LSTM networks. arXiv preprint arXiv:1610.03098
2016 arXiv
-
[29]
Chris Quirk, Chris Brockett, and William Dolan. 2004. Monolingual machine translation for paraphrase generation. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 142--149
2004
-
[30]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1:8
2019
-
[31]
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732
2015 arXiv
-
[32]
St \'e phane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627--635
2011
-
[33]
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368
2017 arXiv
-
[34]
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2015. Minimum risk training for neural machine translation. arXiv preprint arXiv:1512.02433
2015 arXiv
-
[35]
Andreas Vlachos. 2013. An investigation of imitation learning algorithms for structured prediction. In European Workshop on Reinforcement Learning, pages 143--154
2013
-
[36]
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229--256
1992
-
[37]
Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2018. A study of reinforcement learning for neural machine translation. arXiv preprint arXiv:1808.08866
2018 arXiv
-
[38]
Qingyu Yin, Yu Zhang, Weinan Zhang, Ting Liu, and William Yang Wang. 2018. Deep reinforcement learning for chinese zero pronoun resolution. arXiv preprint arXiv:1806.03711
2018 arXiv
-
[39]
Shiqi Zhao, Xiang Lan, Ting Liu, and Sheng Li. 2009. Application-driven statistical paraphrase generation. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volum...
2009
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.