A Gaussian mixture model over normalized Unicode block counts can score and filter noisy text corpora without hand-written character rules.
On the Impact of Various Types of Noise on Neural Machine Translation
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We examine how various types of noise in the parallel training data impact the quality of neural machine translation systems. We create five types of artificial noise and analyze how they degrade performance in neural and statistical machine translation. We find that neural models are generally more harmed by noise than statistical models. For one especially egregious type of noise they learn to just copy the input sentence.
fields
cs.CL 1years
2019 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
uniblock: Scoring and Filtering Corpus with Unicode Block Information
A Gaussian mixture model over normalized Unicode block counts can score and filter noisy text corpora without hand-written character rules.