Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Denoising based Sequence-to-Sequence Pre-training for Text Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that jointly pre-training a sequence-to-sequence model by denoising corrupted text improves text generation quality and speeds up convergence on abstractive summarization and grammatical error correction tasks.

desk verdict The internal pre-training ablation is convincing, but the state-of-the-art comparisons are built on mismatched dataset versions and preprocessing, so the headline claim overreaches. read the letter →

arxiv 1908.08206 v1 pith:SH5LB4K6 submitted 2019-08-22 cs.CL

Liang Wang , Wei Zhao , Ruoyu Jia , Sujian Li , Jingming Liu This is my paper · ORCID
classification cs.CL
keywords pre-trainingsequence-to-sequencedenoisingautoencodertextgenerationabstractivesummarizationgrammaticalerrorcorrectionTransformerpointer-generatornetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that sequence-to-sequence models for text generation benefit from a pre-training step that denoises corrupted text, rather than pre-training only the encoder or only the decoder. The proposed method, PoDA, corrupts text by shuffling, deleting, and replacing words, then trains a Transformer-plus-pointer-generator network to reconstruct the original sequence, jointly pre-training encoder and decoder. On four datasets spanning abstractive summarization and grammatical error correction, PoDA reports higher ROUGE, F0.5, and GLEU scores than strong baselines and reaches comparable quality in far fewer fine-tuning epochs. The practical stake is a reusable initialization that improves both final quality and training speed without changing the architecture or adding task-specific machinery.

What carries the argument

The central object is PoDA, a denoising autoencoder for sequence-to-sequence learning built on a hybrid Transformer encoder-decoder with a pointer-generator output layer that can copy words from the input. The mechanism is the noising algorithm: each sequence is corrupted by randomly shuffling positions with a Gaussian offset, deleting words with a Beta-sampled probability, and replacing words with samples from a unigram distribution; the model then reconstructs the clean sequence. The masked loss keeps only positions where the input was corrupted (plus 3% uncorrupted positions), preventing the model from succeeding by copying and forcing it to learn meaning-bearing representations.

What would settle it

Re-run the strongest baselines (e.g., rnn-ext+RL on CNN/Daily Mail, Re3Sum on Gigaword, NQE on CoNLL-2014) using exactly the same non-anonymized splits and preprocessing as PoDA; if any baseline matches or exceeds PoDA's ROUGE-1, F0.5, or GLEU scores, the claim of state-of-the-art performance is refuted. Alternatively, train PoDA from scratch on the full datasets with the same compute budget and compare final metrics; the convergence claim predicts a large gap at early epochs that narrows but does not vanish at convergence.

Watch

Extended reading notes

Core claim

The central claim is that joint encoder-decoder denoising pre-training produces representations that transfer to text generation tasks. PoDA treats pre-training as a denoising autoencoder: given a noise-corrupted sequence $x'$, the model maximizes the conditional probability of the original sequence $x$, factorized as $p(x \mid x') = \prod_{i} p(x_i \mid x', x_{<i})$. The loss is masked so the model is forced to predict words that were corrupted, with a small fraction of uncorrupted positions retained to learn copying. The authors show that fully pre-trained PoDA outperforms the same architecture trained from scratch on CNN/Daily Mail, Gigaword, CoNLL-2014, and JFLEG, and also outperforms prior state-of-the-art systems that use task-specific enhancements; ablations show that both the pre-trained encoder and decoder contribute, and that pre-training is especially valuable in low-resource settings.

Load-bearing premise

The claimed superiority over prior systems rests on the assumption that published baseline scores are comparable even though this paper evaluates on a non-anonymized version of CNN/Daily Mail and on grammatical-error-correction training sets slightly different from previous work; if those differences shift scores by more than the reported margins, the state-of-the-art claims would not hold, though the internal pre-training-versus-no-pre-training comparison likely would.

Editorial extensions

If this is right

  • A pre-trained PoDA initialization reduces the labeled data needed: with only $10^3$ Gigaword examples it approaches the quality of an attention-based system trained on nearly 4 million examples.
  • Fine-tuning converges much faster: after 5 epochs PoDA reaches validation perplexity that a from-scratch model takes 30 or more epochs to reach on both summarization datasets.
  • The pre-trained decoder contributes independently: ablations show F0.5 drops by 2.42 without it and 3.42 without the encoder, so both sides of the network carry transferable knowledge.
  • Because pre-training is architecture-agnostic, the same denoising objective could be applied to other seq2seq backbones such as convolutional or recurrent networks without modifying the fine-tuning stage.
  • PoDA without fine-tuning performs poorly on GEC (F0.5 of 20.86 on CoNLL-2014), showing that the pre-trained model is a representation initialization, not a ready-made error corrector.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The success of the masked denoising objective suggests that predicting corrupted tokens, rather than reconstructing everything, is what forces the encoder-decoder pair to learn transferable structure; one testable extension would be varying the corruption type distribution to see which noise drives the gain.
  • The same pre-training could plausibly transfer to other generation tasks like dialogue response generation or data-to-text, which the paper lists as future work.
  • Because the gains persist on the full Gigaword dataset, the benefit is not only about compensating for scarce labels; pre-training appears to shape optimization, a claim that could be probed by measuring gradient norms or representation similarity during fine-tuning.
  • If comparable gains hold across backbones, PoDA-style denoising could become a standard initialization step for seq2seq models, paralleling BERT's role for understanding tasks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This manuscript introduces PoDA (Pre-training of Denoising Autoencoders), a sequence-to-sequence pre-training method that jointly trains the encoder and decoder by reconstructing text corrupted with shuffle, delete, and replace noise. The model uses a Transformer encoder-decoder augmented with a pointer-generator layer and a masked denoising loss. Pre-training is performed on English Wikipedia and the Billion Word Benchmark (roughly 2.99B words), followed by fine-tuning on abstractive summarization (CNN/Daily Mail and Gigaword) and grammatical error correction (CoNLL-2014 and JFLEG). The central claim is that PoDA improves final performance and convergence speed relative to training from scratch without task-specific techniques, and in some settings establishes new state-of-the-art results.

Significance. If the claims are sustained, the contribution is a simple, architecture-preserving pre-training scheme for text generation that complements encoder-only and decoder-only pre-training. The paper's internal ablation—PoDA versus PoDA w/o pre-training under identical data and hyperparameters—is consistent across all four datasets and is the strongest evidence for the core benefit of joint denoising pre-training. The authors also provide code and pre-trained models, which increases the reproducibility value. The headline state-of-the-art comparisons, however, rest on datasets and preprocessing pipelines that are not matched to the cited baselines, and therefore require qualification.

major comments (3)
  1. [Section 3.2, Table 3] The claimed new state-of-the-art on CNN/Daily Mail is confounded by a dataset-version mismatch. The paper states that it uses the non-anonymized version of CNN/Daily Mail, whereas the cited baselines PGNet, rnn-ext+RL, and NeuSum are standardly evaluated on the anonymized version; the paper does not report the baselines on its own version. Given the ROUGE-L margin over NeuSum is only 0.56 (38.54 vs 37.98), the version difference alone could plausibly account for the improvement. The authors should either re-run the baselines on the non-anonymized version or evaluate PoDA on the anonymized version before claiming state-of-the-art.
  2. [Section 3.3, Tables 5-7] The GEC comparisons are not controlled. The paper acknowledges that training sentence-pair counts are 'slightly different from previous work' (Section 3.3) and the baselines use different preprocessing, tokenization, reranking, and data-augmentation strategies (e.g., MLConv uses reranking, char-seq2seq uses characters). It also does not report the exact training-set sizes or preprocessing applied to the baselines. Since the single-model F0.5 margin over the strongest ensemble baseline is large (59.40 vs 56.52 for NQE), it is possible the result survives; but as reported, the comparison is insufficiently specified to support 'surpassing previous state-of-the-art single model results' and 'outperforming' ensemble systems.
  3. [Section 6 and Abstract] The abstract and conclusion state without qualification that PoDA 'improves model performance over strong baselines' and 'establishes new state-of-the-art' on CNN/Daily Mail. In light of the comparability issues above, these claims should be restricted to the internal controlled comparison (PoDA vs PoDA w/o pre-training) or substantiated with matched baseline evaluations. The controlled comparisons in Tables 3, 4, 6, and 7 do support the more modest claim that pre-training helps, so the fix is to temper or verify the SOTA wording.
minor comments (6)
  1. [Table 3] The rnn-ext+RL row has a missing citation placeholder '(?)' in the text; it should cite the original source (e.g., Paulus et al., 2018).
  2. [Section 3.2, Evaluation] There is a typo: 'in terms of of ROUGE-1' should read 'in terms of ROUGE-1'.
  3. [Section 3.2, Gigaword] The paper notes that the lowercased Gigaword corpus is inconsistent with the case-sensitive vocabulary, which likely hurts PoDA; this is an unusual setup for a SOTA claim and should be clarified in the discussion.
  4. [Section 4.3] The ablation labels 'w/o pre-trained encoder' and 'w/o pre-trained decoder' should define which parameters are randomly initialized and which are fine-tuned from the pre-trained checkpoint.
  5. [Figure 2] No error bars or multiple seeds are reported; given the observed fluctuations in F0.5, some estimate of variance would strengthen the convergence analysis.
  6. [Algorithm 1] In the REPLACE function, sampling from the unigram distribution can yield a replacement identical to the original word with nonzero probability; the effect is minor but should be stated explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pre-training objective is independent of downstream labels and the central comparison is a controlled ablation.

full rationale

The core claim—that joint encoder-decoder denoising pre-training improves text generation performance and convergence—is supported by an internally controlled comparison and does not reduce to its inputs. The pre-training objective (Eq. 1-2) maximizes p(x|x') on unlabeled Wikipedia and Billion Word corpus; it does not use downstream training or test labels, and the noise hyperparameters (sigma=0.5; Beta mean 0.15) are fixed design choices rather than fits to the reported benchmarks. The ablation 'PoDA w/o pre-training' uses the same architecture, data, and hyperparameters, isolating the pre-trained initialization; Table 9 further isolates encoder vs. decoder initialization. The paper explicitly reports that 'PoDA w/o fine-tuning' performs poorly (F0.5 20.86 on CoNLL-2014), showing the pre-training step is not merely restating the GEC task as data augmentation. The only self-citation, 'A parallel work (Zhao et al., 2019) observes similar gain...', is not load-bearing: it is a side remark, not used to justify PoDA's derivation or results. Concerns about comparability with prior systems, such as the CNN/Daily Mail non-anonymized version and GEC training-set counts being 'slightly different from previous work,' are external-validity/correctness risks, not circularity: they do not make the paper's own equations or fitted parameters equivalent to its predictions. No step in the derivation chain is self-definitional, no fitted parameter is renamed as a prediction, and no load-bearing claim rests on a self-citation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

PoDA introduces no new physical or conceptual entities beyond the method itself, but the central claim relies on hand-set noise hyperparameters, on the assumed transferability of general corpora to summarization and GEC, on the sufficiency of the denoising surrogate, and on the comparability of baseline numbers that are not measured under identical conditions.

free parameters (2)
  • noise hyperparameters (sigma, alpha, beta) = sigma=0.5; Beta with mean 0.15 and std 0.03
    Set by hand in Section 2.2; controls the amount and type of corruption and is not tuned on downstream tasks, but no sensitivity analysis is provided.
  • retention ratio for uncorrupted positions in masked loss = 0.03
    Section 2.2: 3% of uncorrupted positions are kept in the loss to teach copying; a hand-set value that affects the masked loss computation in Eq. (2).
assumptions (4)
  • domain assumption Large-scale unsupervised text corpora (Wikipedia and Billion Word Benchmark) provide representations transferable to summarization and GEC.
    Section 2.3: pre-training uses these corpora; the transferability to downstream tasks is assumed and not separately justified.
  • ad hoc to paper Denoising autoencoder reconstruction with shuffle/delete/replace noise is a sufficient surrogate for the semantics needed in abstractive summarization and grammatical error correction.
    Section 2.2: this is the central methodological hypothesis of PoDA, asserted without a derivation or theoretical justification.
  • domain assumption ROUGE, M2, and GLEU are valid proxies for the quality differences that matter in summarization and GEC.
    Sections 3.2 and 3.3 use these metrics for all claims; no human evaluation is reported.
  • ad hoc to paper Cited baseline results are comparable to the reported PoDA numbers despite dataset version and preprocessing differences.
    Section 3.2 uses the non-anonymized CNN/Daily Mail against anonymized baselines; Section 3.3 notes GEC training set counts differ from previous work. The paper does not justify comparability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Denoising based Sequence-to-Sequence Pre-training for Text Generation." pith.science (2026). https://pith.science/paper/SH5LB4K6

@misc{pith2026190808206,
  author       = {Pith},
  title        = {Pith review of: Denoising based Sequence-to-Sequence Pre-training for Text Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SH5LB4K6}},
  note         = {Machine review of arXiv:1908.08206}
}
read the original abstract

This paper presents a new sequence-to-sequence (seq2seq) pre-training method PoDA (Pre-training of Denoising Autoencoders), which learns representations suitable for text generation tasks. Unlike encoder-only (e.g., BERT) or decoder-only (e.g., OpenAI GPT) pre-training approaches, PoDA jointly pre-trains both the encoder and decoder by denoising the noise-corrupted text, and it also has the advantage of keeping the network architecture unchanged in the subsequent fine-tuning stage. Meanwhile, we design a hybrid model of Transformer and pointer-generator networks as the backbone architecture for PoDA. We conduct experiments on two text generation tasks: abstractive summarization, and grammatical error correction. Results on four datasets show that PoDA can improve model performance over strong baselines without using any task-specific techniques and significantly speed up convergence.

Figures

Figures reproduced from arXiv: 1908.08206 by the authors.

Figure 1
Figure 1. PoDA model architecture. The masked loss is calculated only for the blue underlined words. “ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. F0.5 score on CoNLL-2014 test set with re￾spect to the number of pre-training iterations [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. ROUGE-1 on Gigaword test set with respect [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Validation perplexity with respect to the training epochs on CNN/Daily Mail and Gigaword datasets. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OpenRTAG: A Comprehensive Benchmark for Robust Text-Attributed Graph Learning under Data Quality Degradation

    cs.AI 2026-07 conditional novelty 5.0 of 10

    OpenRTAG is a benchmark that organizes text-attributed-graph data-quality issues into a 3x3 taxonomy (text/structure/label by sparsity/noise/imbalance) and evaluates model robustness across nine datasets and three tasks.

Reference graph

Works this paper leans on

66 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. 2007. Greedy layer-wise training of deep networks. In Advances in neural information processing systems, pages 153--160

    work page 2007

  4. [4]

    Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632--642

    work page 2015

  5. [5]

    Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018 a . Retrieve, rerank and rewrite: Soft template based neural summarization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 152--161

    work page 2018

  6. [6]

    Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018 b . Faithful to the original: Fact aware neural abstractive summarization. In Thirty-Second AAAI Conference on Artificial Intelligence

    work page 2018

  7. [7]

    Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005

    arXiv 2013

  8. [8]

    Shamil Chollampatt and Hwee Tou Ng. 2018 a . A multilayer convolutional encoder-decoder neural network for grammatical error correction. arXiv preprint arXiv:1801.08831

    work page Pith review arXiv 2018

Show all 66 references
  1. [9]

    Shamil Chollampatt and Hwee Tou Ng. 2018 b . Neural quality estimation of grammatical error correction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2528--2539

    2018

  2. [10]

    George E Dahl, Dong Yu, Li Deng, and Alex Acero. 2012. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on audio, speech, and language processing, 20(1):30--42

    2012

  3. [11]

    Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568--572. Association for Comput...

    2012

  4. [12]

    Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building a large annotated corpus of learner english: The nus corpus of learner english. In Proceedings of the eighth workshop on innovative use of NLP for building educational applications, pages 22--31

    2013

  5. [13]

    Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079--3087

    2015

  6. [14]

    Jun Deng, Zixing Zhang, Erik Marchi, and Bjorn Schuller. 2013. Sparse autoencoder-based feature transfer learning for speech emotion recognition. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, pages 511--516. IEEE

    2013

  7. [15]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  8. [16]

    Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197

    2019 arXiv

  9. [17]

    Sergey Edunov, Alexei Baevski, and Michael Auli. 2019. Pre-trained language model representations for language generation. arXiv preprint arXiv:1903.09722

    2019 arXiv

  10. [18]

    Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. 2010. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11(Feb):625--660

    2010

  11. [19]

    Mariano Felice, Zheng Yuan, istein E Andersen, Helen Yannakoudakis, and Ekaterina Kochmar. 2014. Grammatical error correction using hybrid systems and type filtering. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 15--24

    2014

  12. [20]

    Markus Freitag and Scott Roy. 2018. Unsupervised natural language generation with denoising autoencoders. arXiv preprint arXiv:1804.07899

    2018 arXiv

  13. [21]

    Tao Ge, Furu Wei, and Ming Zhou. 2018 a . Fluency boost learning and inference for neural grammatical error correction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1055--1065

    2018

  14. [22]

    Tao Ge, Furu Wei, and Ming Zhou. 2018 b . Reaching human-level performance in automatic grammatical error correction: An empirical study. arXiv preprint arXiv:1807.01270

    2018 arXiv

  15. [23]

    Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122

    2017 arXiv

  16. [24]

    Roman Grundkiewicz and Marcin Junczys-Dowmunt. 2018. Near human-level performance in grammatical error correction with hybrid machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua...

    2018

  17. [25]

    Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. arXiv preprint arXiv:1603.06393

    2016 arXiv

  18. [26]

    Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 140--149

    2016

  19. [27]

    Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In Proceedings of NAACL-HLT, pages 1367--1377

    2016

  20. [28]

    Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 328--339

    2018

  21. [29]

    Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018. Approaching neural grammatical error correction as a low-resource machine translation task. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computati...

    2018

  22. [30]

    Yunsu Kim, Jiahui Geng, and Hermann Ney. 2018. Improving unsupervised word-by-word translation with language model and denoising autoencoder. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 862--868

    2018

  23. [31]

    Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66--71

    2018

  24. [32]

    Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291

    2019 arXiv

  25. [33]

    Jiwei Li, Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (...

    2015

  26. [34]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out

    2004

  27. [35]

    Junyang Lin, Xu Sun, Shuming Ma, and Qi Su. 2018. Global encoding for abstractive summarization. arXiv preprint arXiv:1805.03989

    2018 arXiv

  28. [36]

    Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6294--6305

    2017

  29. [37]

    Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011. Mining revision log of language learning sns for automated japanese error correction of second language learners. In Proceedings of 5th International Joint Conference on Natural Language Processing, pag...

    2011

  30. [38]

    Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lan...

    2015

  31. [39]

    Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. Jfleg: A fluency corpus and benchmark for grammatical error correction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume...

    2017

  32. [40]

    Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The conll-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pa...

    2014

  33. [41]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311--318. Association for Computational Linguistics

    2002

  34. [42]

    Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages ...

    2017

  35. [43]

    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: H...

    2018

  36. [44]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/language-unsupervised/language\_ understanding\_paper. pdf

    2018

  37. [45]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8)

    2019

  38. [46]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383--2392

    2016

  39. [47]

    Prajit Ramachandran, Peter J Liu, and Quoc V Le. 2016. Unsupervised pretraining for sequence to sequence learning. arXiv preprint arXiv:1611.02683

    2016 arXiv

  40. [48]

    Ehud Reiter and Robert Dale. 2000. Building natural language generation systems. Cambridge university press

    2000

  41. [49]

    Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. 2011. Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pages 833--840. Omnipress

    2011

  42. [50]

    Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379--389

    2015

  43. [51]

    Keisuke Sakaguchi, Matt Post, and Benjamin Van Durme. 2017. Grammatical error correction with neural reinforcement learning. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pages 366--372

    2017

  44. [52]

    Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368

    2017 arXiv

  45. [53]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning, pages 5926--5936

    2019

  46. [54]

    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104--3112

    2014

  47. [55]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998--6008

    2017

  48. [56]

    Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096--1103. ACM

    2008

  49. [57]

    Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. 2010. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(Dec):3371--3408

    2010

  50. [58]

    Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869

    2015 arXiv

  51. [59]

    Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461

    2018 arXiv

  52. [60]

    Hideo Watanabe and Koichi Takeda. 1998. A pattern-based machine translation system extended by example-based processing. In Proceedings of the 17th international conference on Computational linguistics-Volume 2, pages 1369--1373. Association for Computational Linguistics

    1998

  53. [61]

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144

    2016 arXiv

  54. [62]

    Ziang Xie, Anand Avati, Naveen Arivazhagan, Dan Jurafsky, and Andrew Y Ng. 2016. Neural language correction with character-based attention. arXiv preprint arXiv:1603.09727

    2016 arXiv

  55. [63]

    Kun Xu, Lingfei Wu, Zhiguo Wang, Yansong Feng, and Vadim Sheinin. 2018. Sql-to-text generation with graph-to-sequence model. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 931--936

    2018

  56. [64]

    Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. 2019. Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computatio...

    2019

  57. [65]

    Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. Neural document summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)...

    2018

  58. [66]

    Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on com...

    2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.