A multi-level optimal transport loss combining sequence-level ranking, top-k truncation, and Sinkhorn sequence distance outperforms earlier cross-tokenizer distillation losses on QA and summarization.
ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We propose to train a non-autoregressive machine translation model to minimize the energy defined by a pretrained autoregressive model. In particular, we view our non-autoregressive translation system as an inference network (Tu and Gimpel, 2018) trained to minimize the autoregressive teacher energy. This contrasts with the popular approach of training a non-autoregressive model on a distilled corpus consisting of the beam-searched outputs of such a teacher model. Our approach, which we call ENGINE (ENerGy-based Inference NEtworks), achieves state-of-the-art non-autoregressive results on the IWSLT 2014 DE-EN and WMT 2016 RO-EN datasets, approaching the performance of autoregressive models.
fields
cs.CL 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models
A multi-level optimal transport loss combining sequence-level ranking, top-k truncation, and Sinkhorn sequence distance outperforms earlier cross-tokenizer distillation losses on QA and summarization.