PoDA pre-trains a Transformer plus pointer-generator seq2seq model as a denoising autoencoder and reports gains over non-pre-trained baselines on summarization and grammatical error correction.
Unsupervised Natural Language Generation with Denoising Autoencoders
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Generating text from structured data is important for various tasks such as question answering and dialog systems. We show that in at least one domain, without any supervision and only based on unlabeled text, we are able to build a Natural Language Generation (NLG) system with higher performance than supervised approaches. In our approach, we interpret the structured data as a corrupt representation of the desired output and use a denoising auto-encoder to reconstruct the sentence. We show how to introduce noise into training examples that do not contain structured data, and that the resulting denoising auto-encoder generalizes to generate correct sentences when given structured data.
fields
cs.CL 1years
2019 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Denoising based Sequence-to-Sequence Pre-training for Text Generation
PoDA pre-trains a Transformer plus pointer-generator seq2seq model as a denoising autoencoder and reports gains over non-pre-trained baselines on summarization and grammatical error correction.