Pith. sign in

REVIEW 3 major objections 4 minor 21 references

Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An unsupervised baseline that repeats the input sentence beats published paraphrase-generation results on BLEU and METEOR for Quora and Twitter, showing these metrics reward copying rather than paraphrase quality.

desk verdict Short, useful empirical note showing an identity copy baseline beats published SOTA on two paraphrase benchmarks; the qualitative claim holds, but the exact margins rest on unstated evaluation details. read the letter →

arxiv 1908.07831 v1 pith:UHNDY6JY submitted 2019-08-19 cs.CL

Hongren Mao , Hung-yi Lee This is my paper · ORCID
classification cs.CL
keywords paraphrasegenerationparrotingbaselineBLEUMETEORTERevaluationmetricsQuoradatasetTwitter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests what happens when a paraphrase-generation system does nothing but return the input sentence unchanged, with no training data at all. On the Quora dataset this full parroting baseline beats the published best METEOR score (38.60 versus 33.6) and matches the reported BLEU score; on Twitter it beats the published best on both BLEU (65.26 versus 45.74) and METEOR (41.73 versus 20.18). The authors conclude that high BLEU and METEOR scores do not demonstrate paraphrase ability under these evaluation conditions, because a model can clear the bar by making no changes or only trivial changes to the input. They recommend that future work compare against parroting, prefer multi-reference entries, and consider human judgment or downstream-task evaluation alongside overlap metrics.

What carries the argument

The load-bearing mechanism is the parroting function itself: for an input sentence $i$, the output is $o=i$, with partial variants that cut or replace a fraction of words from the head, tail, or random positions. Its strength comes from three interacting facts: BLEU and METEOR reward n-gram overlap with reference sentences; Quora and Twitter contain many entries with a single reference paraphrase, so a copy of the input necessarily overlaps heavily with that reference; and the input sentence is statistically close to the reference distribution in these datasets. The paper additionally samples test sets of the same sizes used by prior work, showing that parroting's scores remain stable across many random splits.

What would settle it

Re-run the published paraphrase-generation systems and the full parroting baseline under one identical evaluation protocol—same tokenization, same reference set per input, same dataset split, and same BLEU, METEOR, and TER implementations. If parroting no longer beats the trained systems on Twitter and Quora, the paper's headline result is an artifact of mismatched evaluation; if it still does, the result is a property of the datasets and metrics. A complementary check is a human preference study: if annotators clearly prefer trained-model outputs to the input itself despite metric ties, the metrics, not the models, are the uninformative component.

Watch

Extended reading notes

Core claim

The paper's central claim is that the identity function, output equals input, is a de facto strong baseline for paraphrase generation on Quora and Twitter when scored by BLEU, METEOR, and TER. Parroting achieves Twitter BLEU 65.26 versus the reported 45.74 and Twitter METEOR 41.73 versus 20.18, and Quora METEOR 38.60 versus 33.6. Partial parroting, which cuts or replaces words from the head, tail, or random positions, remains competitive with the published numbers until a substantial fraction of the sentence is modified. On MSCOCO, parroting performs clearly worse on all three metrics, which the paper attributes to the larger edit distances between captions of the same image.

Load-bearing premise

The load-bearing premise is that the published state-of-the-art scores, which were measured on test subsets and with preprocessing details that the paper could not verify, can be directly compared with parroting scores computed on the full datasets.

Editorial extensions

If this is right

  • Any future claim of improved paraphrase generation on Quora or Twitter should first be checked against the parroting baseline, since a zero-parameter model already clears the published bar on BLEU and METEOR.
  • Evaluation on Quora entries with a single reference paraphrase is particularly unreliable for overlap metrics; restricting to multi-reference entries, or using downstream-task performance or human judgment, gives a more meaningful signal.
  • On MSCOCO, parroting does not work because captions of the same image differ in ways that require genuine edit distance, so that dataset tests a different and harder kind of paraphrase generation.
  • The stability of parroting's scores across many random test-set samples of the sizes used by prior work makes it unlikely that the comparison is driven purely by test-set selection on the authors' side.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the published systems were re-evaluated with the same tokenization, reference counts, and full-dataset evaluation protocol, some reported margins over previous work could shrink substantially, because the parroting comparison currently mixes evaluation setups.
  • Editorial inference: training a paraphrase model may still produce outputs that people prefer, such as more diverse or more faithful paraphrases, even when BLEU and METEOR tie with parroting; the paper's result indicts the metrics more than it proves the models are trivial.
  • Editorial inference: the parroting diagnostic transfers naturally to other tasks where input and output are highly overlapping, such as style transfer, grammatical error correction, or sentence simplification, where identity baselines may also outperform published numbers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper studies a zero-parameter baseline for paraphrase generation: outputting the input sentence unchanged (full parroting), or making simple word-cuts/replacements (partial parroting). The authors evaluate this baseline on Quora, Twitter, and MSCOCO using BLEU, METEOR, and TER, and compare against previously published state-of-the-art numbers. They report that full parroting surpasses the published state of the art on Twitter BLEU and METEOR and on Quora METEOR, and that partial parroting can still beat the state of the art when a substantial fraction of the input is cut. They also analyze how scores vary with the number of reference paraphrases and recommend that future work use multiple references or qualitative analysis before claiming success. The paper's central claim is that current benchmark scores can be matched or exceeded without learning anything, which casts doubt on the validity of these metrics and datasets for paraphrase generation.

Significance. If the comparison is valid, this is an important and timely empirical finding for paraphrase generation. The baseline is genuinely parameter-free and unsupervised, and the paper's recommendation that future work check whether models behave like parroting is practical and actionable. The random-sampling robustness experiment in Table 4 is a good-faith attempt to address the unknown test splits used by prior work. However, the strength of the contribution rests entirely on the comparability of the published state-of-the-art numbers with the authors' own scores, and that comparability is not yet established. The paper does not ship code or an exact metric specification, so the headline margins (e.g., +42.67% BLEU on Twitter in Table 2) cannot currently be verified or reproduced. The empirical observation is valuable even if the final conclusion is partly negative, and the paper does not overstate the interpretation heavily, but the central comparison needs more rigor before the claim is fully supported.

major comments (3)
  1. [Section 4.1 and Tables 1-3] The paper never states whether BLEU, METEOR, and TER are computed as corpus-level scores or as averages of per-sentence scores. This distinction is load-bearing because the published state-of-the-art numbers in Tables 1-3 were likely produced with a specific pipeline (e.g., multi-bleu.perl or a Python implementation with particular smoothing and tokenization), while Figures 1-4 in the manuscript clearly use per-sentence scores bucketed by reference count. If Tables 1-3 use per-sentence macro-averaging while the prior papers report corpus-level BLEU, then the headline differences, such as BLEU 65.26 vs. 45.74 on Twitter in Table 2, could be an artifact of aggregation rather than a real property of the datasets. Please specify the exact metric implementation, the smoothing method, the tokenization, whether scores are averaged over sentences or computed on pooled text, and how multiple references are combined.
  2. [Section 3 and Table 4] The comparison between full parroting and the published state of the art is indirect: the state-of-the-art scores in Tables 1-3 come from test splits that are not specified, while full parroting is evaluated on the entire datasets for Quora and Twitter. The random-sampling experiment in Table 4 reduces, but does not eliminate, this mismatch, because it uses the same unspecified scoring protocol and still evaluates on samples drawn by the authors rather than on the exact splits used by the prior papers. In addition, Table 4 labels are inconsistent with the text: the text states that 4K Quora test sets and 5K Twitter test sets were sampled, but the table headers read 'QUORA (5K test set× 1200)' and 'TWITTER (4K test set× 250)'. Please either re-run released models or outputs through the same evaluation harness on the same splits, or clearly state the full pipeline and justify why the full-dataset scores are directly comparable to the previously reported point estimates.
  3. [Section 4.2 and Table 4] The paper uses the word 'significantly' in the abstract and Section 4.2, but no statistical significance testing is performed on the comparisons against the published state-of-the-art scores. The statement in Section 4.2 that 'it is logical to assume that parroting performance would still exceed or be on par with the state-of-the-art on those test sets' goes beyond what Table 4 can establish, since the sampled test sets are not the exact ones used by prior work and the standard deviations do not capture the uncertainty in the prior papers' preprocessing and scoring choices. Please either obtain the exact test splits or rephrase the conclusion to reflect the conditional nature of the comparison.
minor comments (4)
  1. [Table 4] The test-set sizes in the column headers appear to be swapped relative to the text; the text says 4K for Quora and 5K for Twitter, while the table says the opposite.
  2. [Figures 1-4 and Appendix B] Several figures show aggregated metric scores but do not report the number of sentences in each bucket or any measure of spread; adding counts or error bars would help the reader judge the stability of the observed trends.
  3. [Appendix C] The purpose of computing BLEU with non-reference sentences is not fully explained; it may help to state explicitly that this is a sanity check showing that BLEU is higher for sentences similar to the input, and to note that this does not constitute a paraphrase-evaluation metric.
  4. [Section 4.2] The partial-parroting thresholds mentioned in the text (e.g., 'modified by 42%' on Twitter BLEU) are not accompanied by the corresponding numerical scores; providing the exact values in a small table or in the text would make the claims easier to verify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the parroting baseline is parameter-free, externally scored, and its comparison to published state-of-the-art numbers is an empirical claim rather than a derived equality.

full rationale

The paper's central claim is that an unsupervised identity baseline, o = i, outperforms published state-of-the-art scores on BLEU and METEOR for Twitter and METEOR for Quora. No parameter is fitted, no model is trained, and no target result is embedded in the scoring procedure. The parroting scores are computed directly from the datasets using standard evaluation metrics and then compared against numbers reported by prior external papers. The paper does not derive the state-of-the-art scores from its own assumptions, nor does it define its method in terms of the metric outcomes. The authors explicitly avoid selecting a favorable subset by evaluating on the entire dataset, and Table 4 checks robustness under random sampling. There are no self-citations, no imported uniqueness theorems, and no ansatz smuggled in through citations. The acknowledged limitations, such as the unavailability of prior test splits and the possibility that different metric implementations could affect comparability, are concerns about measurement validity and external comparability, not circularity. Under the stated review rules, an empirical baseline against external benchmarks is self-contained, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim borrows from prior work the evaluation metrics and the quoted SOTA scores. There are no fitted free parameters and no invented entities; the three assumptions above are load-bearing because if any of them fails, the comparison that supports the word 'outperforms' becomes incomparable or meaningless.

assumptions (3)
  • domain assumption Scores reported by prior state-of-the-art papers are directly comparable to the scores computed here on full datasets or random samples.
    The headline comparisons in Tables 1 to 3 assume the prior test sets, preprocessing, and metric computations match the authors' pipeline.
  • domain assumption BLEU, METEOR, and TER, as computed by the authors, are valid measures of paraphrase quality for this comparison.
    The paper uses the same metrics as prior work; the entire finding is expressed through these metrics, and the paper later questions their validity.
  • domain assumption The processed versions of Quora and Twitter used for scoring are equivalent to those used by the cited SOTA papers.
    Section 3 states the authors process the data and include single-reference entries, but does not show that the SOTA papers used identical preprocessing or reference sets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets." pith.science (2026). https://pith.science/paper/UHNDY6JY

@misc{pith2026190807831,
  author       = {Pith},
  title        = {Pith review of: Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UHNDY6JY}},
  note         = {Machine review of arXiv:1908.07831}
}
read the original abstract

Paraphrase generation is an interesting and challenging NLP task which has numerous practical applications. In this paper, we analyze datasets commonly used for paraphrase generation research, and show that simply parroting input sentences surpasses state-of-the-art models in the literature when evaluated on standard metrics. Our findings illustrate that a model could be seemingly adept at generating paraphrases, despite only making trivial changes to the input sentence or even none at all.

Figures

Figures reproduced from arXiv: 1908.07831 by the authors.

Figure 2
Figure 2. Metric scores v.s. number of reference para [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Metric scores v.s. ratio of text that is cut from [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Number of reference paraphrases v.s. per [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figures from the paper (8 more)
Figure 6
Figure 6. Figure 6: Number of reference paraphrases v.s. per [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Number of reference paraphrases v.s. per [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Metric scores v.s. ratio of text that is cut from [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Metric scores v.s. ratio of text that is cut from [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Metric scores v.s. ratio of text that is cut [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 11
Figure 11. Figure 11: Metric scores v.s. ratio of text that is cut [PITH_FULL_IMAGE:figures/full_fig_p007_11.png]
Figure 16
Figure 16. Figure 16: Metric scores v.s. ratio of text that is re [PITH_FULL_IMAGE:figures/full_fig_p007_16.png]
Figure 17
Figure 17. Figure 17: Metric scores v.s. ratio of text that is re [PITH_FULL_IMAGE:figures/full_fig_p008_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 14 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bolshakov and Alexander Gelbukh

    Igor A. Bolshakov and Alexander Gelbukh. 2004. Synonymous paraphrasing using wordnet and internet. In Natural Language Processing and Information Systems, pages 312--323, Berlin, Heidelberg. Springer Berlin Heidelberg

    work page 2004

  4. [4]

    Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. https://doi.org/10.1145/2623330.2623677 Open question answering over curated and extracted knowledge bases . In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, pages 1156--1165, New York, NY, USA. ACM

    arXiv 2014

  5. [5]

    Ankush Gupta, Arvind Agarwal, Prawaan Singh, and Piyush Rai. 2018. A deep generative framework for paraphrase generation. In AAAI

    work page 2018

  6. [6]

    Samer Hassan, Andras Csomai, Carmen Banea, Ravi Sinha, and Rada Mihalcea. 2007. http://aclweb.org/anthology/S07-1091 Unt: Subfinder: Combining knowledge sources for automatic lexical substitution . In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 410--413. Association for Computational Linguistics

    work page 2007

  7. [7]

    Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. http://aclweb.org/anthology/N18-1170 Adversarial example generation with syntactically controlled paraphrase networks . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers...

    work page 2018

  8. [8]

    David Kauchak and Regina Barzilay. 2006. https://doi.org/10.3115/1220835.1220893 Paraphrasing for automatic evaluation . In Proceedings of the Main Conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, HLT-NAACL '06, pages 455--462, Stroudsburg, PA, USA. Association for Computatio...

    work page arXiv 2006

Show all 21 references
  1. [9]

    Wuwei Lan, Siyu Qiu, Hua He, and Wei Xu. 2017. A continuously growing dataset of sentential paraphrases. In EMNLP

    2017

  2. [10]

    Wuwei Lan and Wei Xu. 2018. Neural network models for paraphrase identification, semantic textual similarity, natural language inference, and question answering. In Proceedings of the 27th International Conference on Computational Linguistics (COLING)

    2018

  3. [11]

    Alon Lavie and Abhaya Agarwal. 2007. http://dl.acm.org/citation.cfm?id=1626355.1626389 Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments . In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT '07, pag...

    2007

  4. [12]

    Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li. 2018. Paraphrase generation with deep reinforcement learning. In EMNLP

    2018

  5. [13]

    Belongie, Lubomir D

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV

    2014

  6. [14]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: A method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, pages 31...

    2002

  7. [15]

    Hasan, Kathy Lee, Vivek Datla, Ashequl Qadir, Joey Liu, and Oladimeji Farri

    Aaditya Prakash, Sadid A. Hasan, Kathy Lee, Vivek Datla, Ashequl Qadir, Joey Liu, and Oladimeji Farri. 2016. http://www.aclweb.org/anthology/C16-1275 Neural paraphrase generation with stacked residual lstm networks . In Proceedings of COLING 2016, the 26th International Confer...

    2016

  8. [16]

    Lorenza Romano, Milen Kouylekov, Idan Szpektor, Ido Dagan, and Alberto Lavelli. 2006. http://aclweb.org/anthology/E06-1052 Investigating a generic paraphrase-based approach for relation extraction . In 11th Conference of the European Chapter of the Association for Computationa...

    2006

  9. [17]

    Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas, pages 223--231

    2006

  10. [18]

    Su Wang, Rahul Gupta, Nancy Chang, and Jason Baldridge. 2019. http://arxiv.org/abs/1811.00119 A task in a suit and a tie: paraphrase generation with semantic augmentation . In AAAI

    2019 arXiv

  11. [19]

    John Wieting and Kevin Gimpel. 2018. http://aclweb.org/anthology/P18-1042 Paranmt-50m: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol...

    2018

  12. [20]

    Pengcheng Yin, Nan Duan, Ben Kao, Junwei Bao, and Ming Zhou. 2015. https://doi.org/10.1145/2806416.2806542 Answering questions with complex semantic constraints on open knowledge bases . In Proceedings of the 24th ACM International on Conference on Information and Knowledge Ma...

    2015

  13. [21]

    Liang Zhou, Chin-Yew Lin, and Eduard H. Hovy. 2006. Re-evaluating machine translation results with paraphrase support. In EMNLP

    2006

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.