{"id":"08450b0b-5644-4fc2-afd6-dbe371ca98c1","arxiv_id":"1908.07831","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Outputting the input unchanged beats published BLEU and METEOR scores on Twitter and METEOR on Quora paraphrase benchmarks, showing standard evaluation can reward copying.","lead":"This paper shows that a trivial baseline, copying the input sentence unchanged, beats published paraphrase-generation models on standard metrics, including BLEU and METEOR on the Twitter dataset and METEOR on the Quora dataset. The result suggests these benchmark scores can reward copying rather than genuine paraphrasing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Parroting scores in Tables 1-3 lack a stated metric aggregation protocol: if BLEU is per-sentence averaged while SOTA numbers are corpus-level, the claimed 42.67% Twitter gain is an artifact; Table 4 does not fix this.","rationale":"The paper's qualitative finding is important and likely robust: in paraphrase datasets constructed from pairs, copying the input is a strong baseline, and BLEU/METEOR overreward lexical overlap. The authors also include a useful limitation discussion and recommend multiple references. However, the quantitative claim of beating SOTA is only as strong as the comparability of metric pipelines. The text and Figures 1-4 imply per-sentence scoring, whereas published SOTA papers in this area typically report corpus-level BLEU via multi-bleu.perl. Because the paper does not ship code or specify the aggregation, a reader cannot verify whether the +42.67% on Twitter reflects a real finding or a scoring-protocol mismatch. The reader's conditional verdict is therefore appropriate; my concern adds a concrete technical mechanism beyond the already-identified test-split issue, but does not change the verdict.","tokens_in":6960,"tokens_out":9092,"duration_ms":97001,"concrete_test":"Recompute the Twitter full-dataset parrot BLEU in Table 2 using corpus-level BLEU (multi-bleu.perl) with the same tokenization and reference files; if the corpus-level score falls to or below Li et al.'s 45.74, the claimed 42.67% improvement is an aggregation artifact rather than evidence that parroting outperforms SOTA.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—zero-parameter parroting beats SOTA on BLEU/METEOR for Quora and Twitter—depends on Tables 1-3 being comparable to published numbers. The paper never states whether its headline BLEU is corpus-level BLEU (the standard MT-style computation, e.g., multi-bleu.perl) or the average of per-sentence BLEU scores. Figures 1-4 bucket metric scores by reference count, which requires per-sentence scoring; if the Tables use per-sentence averages while Li et al. (2018) and Gupta et al. (2018) report corpus-level BLEU, the comparison is apples-to-oranges and the Twitter gain of +42.67% (Table 2) could be an aggregation artifact. The random-sampling check in Table 4 does not resolve this, because it inherits the same unspecified aggregation and, per Section 4.1, still scores each sampled input against all available references in the full dataset. A separate comparability issue is split provenance: SOTA numbers come from unpublished held-out splits, while parroting is scored on the entire dataset; Table 4 approximates the split distribution but cannot reproduce the exact evaluation protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies a zero-parameter baseline for paraphrase generation: outputting the input sentence unchanged (full parroting), or making simple word-cuts/replacements (partial parroting). The authors evaluate this baseline on Quora, Twitter, and MSCOCO using BLEU, METEOR, and TER, and compare against previously published state-of-the-art numbers. They report that full parroting surpasses the published state of the art on Twitter BLEU and METEOR and on Quora METEOR, and that partial parroting can still beat the state of the art when a substantial fraction of the input is cut. They also analyze how scores vary with the number of reference paraphrases and recommend that future work use multiple references or qualitative analysis before claiming success. The paper's central claim is that current benchmark scores can be matched or exceeded without learning anything, which casts doubt on the validity of these metrics and datasets for paraphrase generation.","tokens_in":7194,"tokens_out":4671,"duration_ms":49252,"significance":"If the comparison is valid, this is an important and timely empirical finding for paraphrase generation. The baseline is genuinely parameter-free and unsupervised, and the paper's recommendation that future work check whether models behave like parroting is practical and actionable. The random-sampling robustness experiment in Table 4 is a good-faith attempt to address the unknown test splits used by prior work. However, the strength of the contribution rests entirely on the comparability of the published state-of-the-art numbers with the authors' own scores, and that comparability is not yet established. The paper does not ship code or an exact metric specification, so the headline margins (e.g., +42.67% BLEU on Twitter in Table 2) cannot currently be verified or reproduced. The empirical observation is valuable even if the final conclusion is partly negative, and the paper does not overstate the interpretation heavily, but the central comparison needs more rigor before the claim is fully supported.","major_comments":[{"comment":"The paper never states whether BLEU, METEOR, and TER are computed as corpus-level scores or as averages of per-sentence scores. This distinction is load-bearing because the published state-of-the-art numbers in Tables 1-3 were likely produced with a specific pipeline (e.g., multi-bleu.perl or a Python implementation with particular smoothing and tokenization), while Figures 1-4 in the manuscript clearly use per-sentence scores bucketed by reference count. If Tables 1-3 use per-sentence macro-averaging while the prior papers report corpus-level BLEU, then the headline differences, such as BLEU 65.26 vs. 45.74 on Twitter in Table 2, could be an artifact of aggregation rather than a real property of the datasets. Please specify the exact metric implementation, the smoothing method, the tokenization, whether scores are averaged over sentences or computed on pooled text, and how multiple references are combined.","section":"Section 4.1 and Tables 1-3"},{"comment":"The comparison between full parroting and the published state of the art is indirect: the state-of-the-art scores in Tables 1-3 come from test splits that are not specified, while full parroting is evaluated on the entire datasets for Quora and Twitter. The random-sampling experiment in Table 4 reduces, but does not eliminate, this mismatch, because it uses the same unspecified scoring protocol and still evaluates on samples drawn by the authors rather than on the exact splits used by the prior papers. In addition, Table 4 labels are inconsistent with the text: the text states that 4K Quora test sets and 5K Twitter test sets were sampled, but the table headers read 'QUORA (5K test set× 1200)' and 'TWITTER (4K test set× 250)'. Please either re-run released models or outputs through the same evaluation harness on the same splits, or clearly state the full pipeline and justify why the full-dataset scores are directly comparable to the previously reported point estimates.","section":"Section 3 and Table 4"},{"comment":"The paper uses the word 'significantly' in the abstract and Section 4.2, but no statistical significance testing is performed on the comparisons against the published state-of-the-art scores. The statement in Section 4.2 that 'it is logical to assume that parroting performance would still exceed or be on par with the state-of-the-art on those test sets' goes beyond what Table 4 can establish, since the sampled test sets are not the exact ones used by prior work and the standard deviations do not capture the uncertainty in the prior papers' preprocessing and scoring choices. Please either obtain the exact test splits or rephrase the conclusion to reflect the conditional nature of the comparison.","section":"Section 4.2 and Table 4"}],"minor_comments":[{"comment":"The test-set sizes in the column headers appear to be swapped relative to the text; the text says 4K for Quora and 5K for Twitter, while the table says the opposite.","section":"Table 4"},{"comment":"Several figures show aggregated metric scores but do not report the number of sentences in each bucket or any measure of spread; adding counts or error bars would help the reader judge the stability of the observed trends.","section":"Figures 1-4 and Appendix B"},{"comment":"The purpose of computing BLEU with non-reference sentences is not fully explained; it may help to state explicitly that this is a sanity check showing that BLEU is higher for sentences similar to the input, and to note that this does not constitute a paraphrase-evaluation metric.","section":"Appendix C"},{"comment":"The partial-parroting thresholds mentioned in the text (e.g., 'modified by 42%' on Twitter BLEU) are not accompanied by the corresponding numerical scores; providing the exact values in a small table or in the text would make the claims easier to verify.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short empirical note whose value depends on the comparability of its scoring to prior work. The authors should be encouraged to provide the exact metric code and, if possible, run a released state-of-the-art system through the same harness. The current indirect comparison is suggestive but not yet conclusive enough for the strength of the headline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead 1908.07831. The punchline: a zero-parameter baseline that outputs the input sentence outperforms the published state of the art on Twitter paraphrase generation under BLEU (+42.7% relative) and METEOR, and beats SOTA on Quora METEOR. That is a real, citable result, and it should embarrass the field into reporting copy baselines.\n\nWhat's new: prior work on Quora/Twitter/MSCOCO didn't include an identity baseline. The paper runs full and partial parroting, shows score-vs-reference-count curves, and does a random test-set sampling check that supports the qualitative conclusion. The suggestion to check whether models behave differently from parroting is sensible, and the discussion of single-reference Quora entries is a genuine practical point.\n\nWhere it's soft: the biggest reporting gap is that the paper never states whether BLEU is corpus-level or the average of per-sentence scores. Standard MT BLEU is corpus-level, but because Figures 1-4 plot per-sentence scores and no metric command is given, a reader can't tell if the headline numbers in Tables 1-3 are directly comparable to prior work. If the prior papers used corpus-level BLEU and the authors used per-sentence averaging on the full dataset, the Twitter gap could be an aggregation artifact. I don't think that's what happened—the effect survives random sampling in Table 4, which is large and stable—but the protocol should have been stated. Also, test-set provenance is an issue: prior SOTA came from unpublished splits, and full-dataset evaluation plus random sampling is an approximation, not a true reproduction. Table 4's header swaps the sample sizes for Quora and Twitter (text says 4K for Quora, 5K for Twitter; table says the reverse), a minor but annoying typo. No code or exact metric script is shipped, which makes verification harder.\n\nOverall, the central qualitative claim—that a trivial copy baseline is competitive with or better than trained models on these datasets—holds up. The more aggressive quantitative claim about exact margins should be treated as indicative rather than definitive. This paper deserves a serious referee; it's short, honest, and useful for the field.","headline":"Short, useful empirical note showing an identity copy baseline beats published SOTA on two paraphrase benchmarks; the qualitative claim holds, but the exact margins rest on unstated evaluation details.","tokens_in":7683,"tokens_out":3028,"would_cite":true,"duration_ms":31056,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An unsupervised baseline that repeats the input sentence beats published paraphrase-generation results on BLEU and METEOR for Quora and Twitter, showing these metrics reward copying rather than paraphrase quality.","keywords":["paraphrase generation","parroting baseline","BLEU","METEOR","TER","evaluation metrics","Quora dataset","Twitter dataset"],"falsifier":"Re-run the published paraphrase-generation systems and the full parroting baseline under one identical evaluation protocol—same tokenization, same reference set per input, same dataset split, and same BLEU, METEOR, and TER implementations. If parroting no longer beats the trained systems on Twitter and Quora, the paper's headline result is an artifact of mismatched evaluation; if it still does, the result is a property of the datasets and metrics. A complementary check is a human preference study: if annotators clearly prefer trained-model outputs to the input itself despite metric ties, the metrics, not the models, are the uninformative component.","tokens_in":6782,"feed_emoji":"🦜","tokens_out":7345,"duration_ms":67056,"temperature":0.7,"pith_summary":"The paper tests what happens when a paraphrase-generation system does nothing but return the input sentence unchanged, with no training data at all. On the Quora dataset this full parroting baseline beats the published best METEOR score (38.60 versus 33.6) and matches the reported BLEU score; on Twitter it beats the published best on both BLEU (65.26 versus 45.74) and METEOR (41.73 versus 20.18). The authors conclude that high BLEU and METEOR scores do not demonstrate paraphrase ability under these evaluation conditions, because a model can clear the bar by making no changes or only trivial changes to the input. They recommend that future work compare against parroting, prefer multi-reference entries, and consider human judgment or downstream-task evaluation alongside overlap metrics.","feed_headline":"Parroting the input beats published paraphrase models","feed_subtitle":"A zero-parameter identity baseline outscores trained systems on BLEU and METEOR for Quora and Twitter.","key_machinery":"The load-bearing mechanism is the parroting function itself: for an input sentence $i$, the output is $o=i$, with partial variants that cut or replace a fraction of words from the head, tail, or random positions. Its strength comes from three interacting facts: BLEU and METEOR reward n-gram overlap with reference sentences; Quora and Twitter contain many entries with a single reference paraphrase, so a copy of the input necessarily overlaps heavily with that reference; and the input sentence is statistically close to the reference distribution in these datasets. The paper additionally samples test sets of the same sizes used by prior work, showing that parroting's scores remain stable across many random splits.","core_discovery":"The paper's central claim is that the identity function, output equals input, is a de facto strong baseline for paraphrase generation on Quora and Twitter when scored by BLEU, METEOR, and TER. Parroting achieves Twitter BLEU 65.26 versus the reported 45.74 and Twitter METEOR 41.73 versus 20.18, and Quora METEOR 38.60 versus 33.6. Partial parroting, which cuts or replaces words from the head, tail, or random positions, remains competitive with the published numbers until a substantial fraction of the sentence is modified. On MSCOCO, parroting performs clearly worse on all three metrics, which the paper attributes to the larger edit distances between captions of the same image.","pith_inferences":["Editorial inference: if the published systems were re-evaluated with the same tokenization, reference counts, and full-dataset evaluation protocol, some reported margins over previous work could shrink substantially, because the parroting comparison currently mixes evaluation setups.","Editorial inference: training a paraphrase model may still produce outputs that people prefer, such as more diverse or more faithful paraphrases, even when BLEU and METEOR tie with parroting; the paper's result indicts the metrics more than it proves the models are trivial.","Editorial inference: the parroting diagnostic transfers naturally to other tasks where input and output are highly overlapping, such as style transfer, grammatical error correction, or sentence simplification, where identity baselines may also outperform published numbers."],"forward_implications":["Any future claim of improved paraphrase generation on Quora or Twitter should first be checked against the parroting baseline, since a zero-parameter model already clears the published bar on BLEU and METEOR.","Evaluation on Quora entries with a single reference paraphrase is particularly unreliable for overlap metrics; restricting to multi-reference entries, or using downstream-task performance or human judgment, gives a more meaningful signal.","On MSCOCO, parroting does not work because captions of the same image differ in ways that require genuine edit distance, so that dataset tests a different and harder kind of paraphrase generation.","The stability of parroting's scores across many random test-set samples of the sizes used by prior work makes it unlikely that the comparison is driven purely by test-set selection on the authors' side."],"supporting_citations":[{"why":"Provides the state-of-the-art Quora METEOR and TER scores that parroting is compared against, and the 4K test-set sampling convention.","marker":"(Gupta et al., 2018)"},{"why":"Provides the state-of-the-art Twitter BLEU and METEOR scores and the Quora BLEU score; its 5K sampled test-set convention is replicated.","marker":"(Li et al., 2018)"},{"why":"Provides the MSCOCO state-of-the-art scores, the comparison that shows parroting fails when edit distance is high.","marker":"(Wang et al., 2019)"},{"why":"Source of the Twitter paraphrase dataset, collected from tweets sharing identical URLs.","marker":"(Lan et al., 2017)"},{"why":"Source of the MSCOCO image-caption dataset used as paraphrase pairs.","marker":"(Lin et al., 2014)"},{"why":"Defines BLEU, the primary overlap metric whose high scores parroting exploits.","marker":"(Papineni et al., 2002)"},{"why":"Defines METEOR, the overlap metric on which parroting sets the reported best on Quora and Twitter.","marker":"(Lavie and Agarwal, 2007)"},{"why":"Defines TER, the edit-distance metric on which parroting performs worse and which reveals high edit distance on MSCOCO.","marker":"(Snover et al., 2006)"}],"fun_headline_variants":["Parroting input beats state-of-the-art on paraphrase datasets","Identity baseline outscores trained paraphrase models on Quora, Twitter","Zero-parameter parroting tops published paraphrase systems","Copy-paste defeats SOTA on paraphrase generation metrics","Paraphrase benchmarks fall to trivial no-edit baseline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the published state-of-the-art scores, which were measured on test subsets and with preprocessing details that the paper could not verify, can be directly compared with parroting scores computed on the full datasets.","fun_headline_variants_meta":{"raw":{"variants":["Parroting input beats state-of-the-art on paraphrase datasets","Identity baseline outscores trained paraphrase models on Quora, Twitter","Zero-parameter parroting tops published paraphrase systems","Copy-paste defeats SOTA on paraphrase generation metrics","Paraphrase benchmarks fall to trivial no-edit baseline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1369,"prompt_tokens":767,"completion_tokens":602,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":383,"completion_tokens_details":{"reasoning_tokens":520}},"tokens_in":383,"tokens_out":602,"duration_ms":6332,"temperature":1.0,"reasoning_tokens":520,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:39:36.225979+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the published paraphrase-generation systems and the full parroting baseline under one identical evaluation protocol—same tokenization, same reference set per input, same dataset split, and same BLEU, METEOR, and TER implementations. If parroting no longer beats the trained systems on Twitter and Quora, the paper's headline result is an artifact of mismatched evaluation; if it still does, the result is a property of the datasets and metrics. A complementary check is a human preference study: if annotators clearly prefer trained-model outputs to the input itself despite metric ties, the metrics, not the models, are the uninformative component.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the state-of-the-art Quora METEOR and TER scores that parroting is compared against, and the 4K test-set sampling convention."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the Twitter paraphrase dataset, collected from tweets sharing identical URLs."},{"cited_title":"Belongie, Lubomir D","cited_arxiv_id":null,"evidence_quote":"Source of the MSCOCO image-caption dataset used as paraphrase pairs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines TER, the edit-distance metric on which parroting performs worse and which reveals high edit distance on MSCOCO."}],"review_version":1}