{"id":"c6958fb1-03bf-4d1a-a3ee-6d60fd681f0b","arxiv_id":"1908.10835","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DAgger-style imitation learning outperforms REINFORCE reinforcement learning for paraphrase generation with a pointer-generator, and the best model reaches state-of-the-art scores on Quora.","lead":"This paper compares imitation learning and reinforcement learning for training a neural paraphraser. It reports that imitation learning works best, and that the best model beats earlier systems on the Quora paraphrase benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'constantly better' IL claim is confounded with schedule-rate choice: in Table 1, decayed DAgger (57.22) trails REINFORCE-SIO (57.26), and the best IL result uses a fixed alpha not swept for RL.","rationale":"The paper is a useful empirical study and the released code/data are a real asset, but the headline conclusion overreaches. The reader's weakest assumption correctly identifies schedule-rate selection as the core confound: the RL variants are run with alpha and beta decaying to 0 by default, while the winning IL variant (DAgger*) uses a fixed nonzero alpha chosen per dataset. The internal comparison in Table 1 (line 10 vs line 9) shows that when both use a decay-to-zero schedule, RL actually edges out IL. So 'constantly better' is false under the paper's own protocol, and the only way to support it is to allow IL a hyperparameter search that RL is not given. The claimed 13% SOTA margin is also largely inherited from the pre-trained MLE base model (56.14 already exceeds RbM's 48.68), and the text itself concedes the dev-set protocol differs from prior work. These are addressable with a matched schedule-rate sweep and significance testing, so a conditional verdict is appropriate rather than rejection.","tokens_in":9338,"tokens_out":7225,"duration_ms":72452,"concrete_test":"Re-run the Appendix B schedule-rate sweep with REINFORCE-SIO instead of DAgger: use fixed alpha in {0.2, 0.5} and beta in {0.2, 0.5, 0.8} (plus the existing decay-to-zero baseline), the same pre-trained pointer-generator checkpoint, the same 30K Quora / 3K Twitter test sets, and the same beam search. If any fixed-alpha,beta REINFORCE-SIO configuration reaches or exceeds DAgger*'s reported averages (58.02 Quora, 47.44 Twitter), the 'IL is constantly better' claim is an artifact of schedule-rate selection. If no configuration does, report per-seed means and standard deviations across at least 3 seeds to confirm the 0.76/0.19 margins are outside noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that imitation learning is constantly better than reinforcement learning is not supported by the paper's own Table 1. DAgger under the same decay-to-zero schedule rate used for the RL variants (line 10: alpha->0, beta=1) averages 57.22, below REINFORCE-SIO (line 9: alpha->0, beta->0) at 57.26. The only IL configuration that beats the best RL configuration is DAgger* (line 11) with a fixed alpha=0.5 on Quora and alpha=0.2 on Twitter, while Section 2.2 fixes the RL variants' schedule rates to decay to 0 by default and no fixed-alpha sweep is reported for REINFORCE-SIO/SO. Appendix B sweeps schedule rates for DAgger only (Tables 3-4), so the IL-vs-RL ranking is confounded with the schedule-rate hyperparameter: a tuned IL method is compared against an untuned/default RL method. Because the quoted margins are small (0.76 avg on Quora; 0.19 on Twitter) and no error bars or significance tests are reported, the 'constantly better' conclusion is fragile. This does not invalidate the finding that a fixed-alpha DAgger can improve on pre-trained MLE, but it removes the support for the general IL-over-RL claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical comparison of imitation learning (IL) and reinforcement learning (RL) methods for paraphrase generation, using a pointer-generator base model and proposing a unified objective (Eq. 2) in which REINFORCE, DAgger, MLE, and several intermediate variants arise as special cases through schedule rates (alpha, beta) and a decode function. Experiments on the Quora and Twitter paraphrase datasets compare these fine-tuning algorithms against each other and against prior systems. The paper claims that (1) IL is constantly better than RL and (2) the best IL setting, DAgger* with a fixed alpha, outperforms prior state-of-the-art methods by about 13% on average score.","tokens_in":9625,"tokens_out":2156,"duration_ms":23861,"significance":"If the central claims held, the paper would provide a practically useful comparison of training objectives for paraphrase generation and a concrete recommendation in favor of DAgger-style scheduled sampling over REINFORCE-style policy gradient fine-tuning. The paper has clear strengths: it releases code and data, formulates several algorithms in one framework, and reports results on two benchmark datasets with a consistent base model and evaluation protocol. The comparison is, however, weakened by an asymmetry in hyperparameter tuning, small performance margins without variance estimates, and an abstract-level claim that is contradicted by the paper's own Table 1. With corrected claims and additional targeted experiments, the finding that a fixed-alpha DAgger setting improves over the pre-trained MLE baseline would be a useful empirical contribution.","major_comments":[{"comment":"The abstract's claim that 'imitation learning is constantly better than reinforcement learning' is not supported by Table 1. Under comparable schedule-rate settings, DAgger (line 10, alpha decays to 0, beta=1) scores 57.22 on average, while REINFORCE-SIO (line 9, alpha and beta decay to 0) scores 57.26. The only IL configuration that beats all RL variants is DAgger* (line 11) with a fixed alpha=0.5. The claim should be revised to state that a fixed-alpha DAgger setting achieved the best results, and that the IL/RL ranking depends on schedule-rate choices.","section":"Abstract and §3, Table 1"},{"comment":"The IL-vs-RL comparison is confounded with hyperparameter tuning. RL variants (REINFORCE, REINFORCE-GTI, REINFORCE-SO, REINFORCE-SIO) use schedule rates that decay to 0 by default, and no fixed-alpha or otherwise wide sweep over schedule rates is reported for these RL methods. In contrast, Appendix B reports schedule-rate sweeps for DAgger only (Tables 3-4). The conclusion that IL is better than RL therefore compares a tuned IL method against default RL configurations. The authors should either report schedule-rate sweeps for the RL variants or explicitly scope the conclusion to the tested configurations.","section":"§2.2, §3, and Appendix B"},{"comment":"The reported differences between methods are small in several key comparisons, and no error bars, multiple-seed runs, or significance tests are provided. For example, on Quora the gap between REINFORCE-SIO (57.26) and DAgger with decayed alpha (57.22) is 0.04 points, and on Twitter the best DAgger* (47.44) is only 0.19 points ahead of REINFORCE-SIO (47.25). Given these margins, the claim that IL is 'constantly better' needs variance estimates or statistical testing. Relatedly, ROUGE-2 is used both as the training reward (Appendix A, Eq. 5) and as one of the three evaluation metrics, so part of the observed ROUGE-2 gain may reflect reward optimization; the authors should state this explicitly and give primary weight to the metrics not used for training.","section":"§3, Tables 1-2 and Appendix A"},{"comment":"The 'large margin' state-of-the-art claim on Quora is partly an artifact of a stronger MLE baseline. All fine-tuned models are compared to competitive systems reprinted from prior work, but the pre-trained MLE baseline itself already averages 56.14, well above RbM (48.68). The paper notes that the development set was run on-the-fly, which may explain this gap. To support the SOTA claim, the comparison should be controlled: either reproduce prior systems under the same training and evaluation protocol, or explicitly frame the result as an improvement over the reproduced pointer-generator baseline rather than over published numbers from other setups.","section":"§3, Result Analysis and Tables 1-2"}],"minor_comments":[{"comment":"'constantly better' should be 'consistently better' or 'better in all tested settings'.","section":"Abstract"},{"comment":"The sentence 'That implies taking an action based on the current observation' is missing a period and should be merged with the preceding sentence.","section":"§2.1"},{"comment":"The phrase 'the the policy function' contains a duplicated 'the' and should be corrected.","section":"§2.1 and §2.2"},{"comment":"The pointer-generator is described as 'state-of-the-art model on paragraph generation' but the paper is about paraphrase generation; this should be corrected.","section":"§2"},{"comment":"Table numbering is inconsistent: both Tables 3 and 4 are labeled, but the row numbering is duplicated (both contain rows numbered 1, 2, 4, 4), and the header 'k alpha' is not explained in the table caption.","section":"Appendix B, Tables 3-4"},{"comment":"For Dis-LSTM, ROUGE-1 is missing and the average score (45.30) does not obviously correspond to the two reported values; the source and exact metric configuration should be clarified in a footnote.","section":"§3, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an NLP/empirical-methods venue and the code/data release is a positive feature. The main risk is that the headline claims overstate what the experiments show. A revision that narrows the claims, adds variance information and symmetric schedule-rate sweeps, and controls the state-of-the-art comparison would make the contribution publishable. I would not reject on the basis of the current evidence, since the central practical finding is plausible and reproducible from the released artifacts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The practical finding holds up: if you're fine-tuning a pointer-generator for paraphrasing, a fixed-schedule DAgger is a solid, easy choice and beats the REINFORCE variants they tested. The unified (alpha, beta) framing is a neat way to see MLE, REINFORCE, and scheduled sampling as points in the same space. Code and data are public, and the appendix actually documents the schedule-rate search for DAgger. That's real credit.\n\nThe paper's own Table 1 undercuts the abstract's 'constantly better' claim. With the same decay-to-zero schedule used for the RL variants, DAgger scores 57.22 average on Quora, just under REINFORCE-SIO's 57.26. The best IL number comes from a fixed alpha=0.5, and the appendix sweeps alpha only for DAgger, not for the RL variants. So the IL-over-RL ranking is partly a tuning asymmetry. The margins are small (0.76 avg on Quora, 0.19 on Twitter) and there are no error bars or significance tests. I'd trust the direction—fixed alpha DAgger being reliable—but not the categorical claim.\n\nThe 'large margin' over prior SOTA is also mostly the pre-trained MLE baseline. Line 5 (pre-trained MLE) is already at 56.14 average, well above Dis-LSTM's 45.30 and RbM's 48.68; DAgger* adds 1.88 points. That's a legitimate improvement, but the headline oversells it as 13% over SOTA when the base model does most of the work.\n\nMinor: ROUGE-2 is both reward and metric; the paper acknowledges this only implicitly. Simultaneous gains on ROUGE-1 and BLEU make it a non-issue, but worth a sentence.\n\nWho is this for: people doing text generation with MLE pre-training who wonder whether to reach for REINFORCE or scheduled sampling. It gives a concrete data point and a simple framework. It's not a breakthrough, but it's an honest, reproducible comparison with a useful takeaway. A serious referee should engage with it; the conditional verdict is right. I'd ask for softened claims, a sweep of the RL variants' schedule rates, and any measure of variance. Then it's solid.","headline":"Useful, reproducible IL-vs-RL comparison for paraphrase generation; the fixed-alpha DAgger result is credible, but 'constantly better' overstates the paper's own tables.","tokens_in":10158,"tokens_out":2273,"would_cite":true,"duration_ms":22551,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Imitation learning beats reinforcement learning for paraphrase generation.","keywords":["paraphrase generation","imitation learning","reinforcement learning","DAgger","REINFORCE","pointer-generator","scheduled sampling","exposure bias"],"falsifier":"Re-run the comparison with identical schedule-rate budgets: train DAgger with $\\alpha$ decayed to zero and REINFORCE-SIO with fixed $\\alpha$ (and comparable $\\beta$), then compare average scores on the same Quora and Twitter splits. If REINFORCE-SIO with fixed rates matches or exceeds DAgger, the claimed constant superiority of IL over RL would be falsified; if DAgger stays ahead under both tunings, the claim holds.","tokens_in":9117,"feed_emoji":"📝","tokens_out":7899,"duration_ms":67254,"temperature":0.7,"pith_summary":"This paper sets out to settle, in one controlled setting, whether reinforcement learning or imitation learning is the better cure for exposure bias in neural paraphrase generation. Using a pointer-generator decoder fine-tuned on two benchmark datasets, it reports that imitation learning in the form of DAgger (scheduled sampling) is consistently better than REINFORCE and its variants. The best DAgger configuration, which keeps a fixed mixing rate between ground-truth and decoded inputs, improves the average ROUGE-1, ROUGE-2, and BLEU score by about 13 percent over the previous state of the art on Quora. The practical payoff would be a simpler and more reliable recipe for fine-tuning text generators.","feed_headline":"Imitation learning beats RL for paraphrase generation","feed_subtitle":"Scheduled-sampling fine-tuning beats REINFORCE on two paraphrase datasets and tops prior art by ~13 percent.","key_machinery":"The central object is a unified online-learning objective parameterized by two schedule rates. DAgger (dataset aggregation) is an imitation-learning algorithm that, at each decoding step, mixes ground-truth words and the model's own argmax predictions as inputs while always supervising against the ground-truth output; in this paper it is treated as equivalent to scheduled sampling. The rate $\\alpha$ controls the probability of feeding the ground-truth previous word as decoder input, and $\\beta$ controls the probability of using the ground-truth word as the training target. Setting $\\alpha = \\beta = 1$ gives maximum-likelihood training, $\\alpha = \\beta = 0$ with random-sampling rollouts gives REINFORCE, and $\\beta = 1$ with $0 < \\alpha < 1$ gives DAgger. This machinery turns the comparison between RL and IL into a comparison of schedule-rate choices, so the observed performance differences are attributable to the learning signal rather than to the architecture.","core_discovery":"The central claim is that for paraphrase generation with a pointer-generator model, imitation learning outperforms reinforcement learning on both datasets tested, and that a DAgger variant with a fixed schedule rate (0.5 on Quora, 0.2 on Twitter) sets a new state of the art, beating prior systems by about 13 percent on average score. The paper reaches this claim through a unified objective in which the schedule rates ($\\alpha$, $\\beta$) and the decoding function interpolate continuously among MLE, REINFORCE, and DAgger, so that comparing special cases isolates the effect of the training signal. It finds that replacing ground-truth inputs with sampled outputs during training, and weighting the gradient by a ROUGE-2 reward, gives only small gains, whereas always supervising against ground-truth outputs while mixing the decoder input between expert and decoded words gives the largest gain. The authors interpret this as evidence that imitation learning's more stable supervision is better suited to this task than reinforcement learning's high-variance reward.","pith_inferences":["The schedule-rate dependence suggests the measured IL-versus-RL gap may be partly a proxy for the gap between fixed and decayed mixing rates; testing REINFORCE-SIO with fixed nonzero $\\alpha$ would clarify whether the family, not the label, drives the gain.","The same $(\\alpha, \\beta)$ grid could be applied to summarization or machine translation as a cheap diagnostic before designing a custom reward.","A testable extension would be to give every algorithm the same schedule-rate budget: tune REINFORCE-SIO with the fixed rates that win for DAgger and decay DAgger's $\\alpha$ the way REINFORCE-SIO decays its rates; if the ordering reverses, the constant-superiority conclusion is tuning-dependent."],"forward_implications":["If DAgger is reliably better, practitioners fine-tuning sequence-to-sequence paraphrasers should prefer scheduled-sampling-style imitation learning over policy-gradient reinforcement learning by default.","Keeping some expert input throughout fine-tuning appears to matter: on Quora, DAgger with $\\alpha$ decayed to zero scores 57.22, while fixed $\\alpha = 0.5$ scores 58.02.","Imitation-learning fine-tuning adds a few average-score points over MLE pre-training on both datasets, whereas reinforcement learning's margin over MLE is small, about 0.18 average points on Quora.","The unified $(\\alpha, \\beta)$ family provides a practical two-parameter grid for diagnosing whether a text-generation task benefits more from expert supervision or from reward-driven exploration."],"supporting_citations":[{"why":"Supplies the RbM baseline, the pointer-generator-with-RL setup this work extends, and the Quora data split.","marker":"Li et al., 2018"},{"why":"Supplies the DAgger algorithm that the paper's best-performing configuration instantiates.","marker":"Ross et al., 2011"},{"why":"Supplies scheduled sampling, treated here as the deep-learning form of DAgger, plus the decay schedule used for $\\alpha$ and $\\beta$.","marker":"Bengio et al., 2015"},{"why":"Supplies the pointer-generator base model used in all experiments.","marker":"See et al., 2017"},{"why":"Supplies the REINFORCE policy-gradient rule used as the RL baseline.","marker":"Williams, 1992"},{"why":"Supplies the ROUGE metrics, including ROUGE-2, which serves as the reward function.","marker":"Lin, 2004"},{"why":"Provides the Twitter URL paraphrase dataset used for the second benchmark.","marker":"Lan et al., 2017"}],"fun_headline_variants":["Imitation learning beats RL for paraphrasing","IL outperforms RL in paraphrase generation","Paraphrase generation: IL better than RL","Scheduled-sampling IL tops RL for paraphrasing","IL beats RL for paraphrase generation by 13%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that imitation learning is consistently better depends on the chosen schedule rates: the winning DAgger setting keeps $\\alpha$ fixed, while the closest RL variant, REINFORCE-SIO, decays both rates to zero, and when DAgger's $\\alpha$ is also decayed to zero its average score dips just below that RL variant.","fun_headline_variants_meta":{"raw":{"variants":["Imitation learning beats RL for paraphrasing","IL outperforms RL in paraphrase generation","Paraphrase generation: IL better than RL","Scheduled-sampling IL tops RL for paraphrasing","IL beats RL for paraphrase generation by 13%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000606,"raw_usage":{"total_tokens":2789,"prompt_tokens":873,"completion_tokens":1916,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":1845}},"tokens_in":489,"tokens_out":1916,"duration_ms":14505,"temperature":1.0,"reasoning_tokens":1845,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:32:18.930780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison with identical schedule-rate budgets: train DAgger with $\\alpha$ decayed to zero and REINFORCE-SIO with fixed $\\alpha$ (and comparable $\\beta$), then compare average scores on the same Quora and Twitter splits. If REINFORCE-SIO with fixed rates matches or exceeds DAgger, the claimed constant superiority of IL over RL would be falsified; if DAgger stays ahead under both tunings, the claim holds.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DAgger algorithm that the paper's best-performing configuration instantiates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Twitter URL paraphrase dataset used for the second benchmark."}],"review_version":1}