POCL wraps LLM knowledge distillation in a curriculum that increases data difficulty and temperature over stages, improving Rouge-L on small GPT-2 and OPT students, though ablations show temperature drives the gains.
Autoregressive Knowledge Distillation through Imitation Learning
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
The performance of autoregressive models on natural language generation tasks has dramatically improved due to the adoption of deep, self-attentive architectures. However, these gains have come at the cost of hindering inference speed, making state-of-the-art models cumbersome to deploy in real-world, time-sensitive settings. We develop a compression technique for autoregressive models that is driven by an imitation learning perspective on knowledge distillation. The algorithm is designed to address the exposure bias problem. On prototypical language generation tasks such as translation and summarization, our method consistently outperforms other distillation algorithms, such as sequence-level knowledge distillation. Student models trained with our method attain 1.4 to 4.8 BLEU/ROUGE points higher than those trained from scratch, while increasing inference speed by up to 14 times in comparison to the teacher model.
citation-role summary
background 1
citation-polarity summary
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Framework
POCL wraps LLM knowledge distillation in a curriculum that increases data difficulty and temperature over stages, improving Rouge-L on small GPT-2 and OPT students, though ablations show temperature drives the gains.