REVIEW 6 cited by
f-Divergence Minimization for Sequence-Level Knowledge Distillation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Knowledge distillation (KD) is the process of transferring knowledge from a large model to a small one. It has gained increasing attention in the natural language processing community, driven by the demands of compressing ever-growing language models. In this work, we propose an f-DISTILL framework, which formulates sequence-level knowledge distillation as minimizing a generalized f-divergence function. We propose four distilling variants under our framework and show that existing SeqKD and ENGINE approaches are approximations of our f-DISTILL methods. We further derive step-wise decomposition for our f-DISTILL, reducing intractable sequence-level divergence to word-level losses that can be computed in a tractable manner. Experiments across four datasets show that our methods outperform existing KD approaches, and that our symmetric distilling losses can better force the student to learn from the teacher distribution.
Forward citations
Cited by 6 Pith papers
-
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Byte-Prefix Marginalization maps a teacher's next-token distribution onto the student's vocabulary through shared byte prefixes plus an explicit residual, giving a mass-preserving target for on-policy distillation acr...
-
Weak-to-Strong Generalization via Direct On-Policy Distillation
Transferring the log-ratio of a small model's pre-RL and post-RL checkpoints provides a dense implicit reward that improves stronger student models at a fraction of the cost of direct RL.
-
Stable On-Policy Distillation through Adaptive Target Reformulation
Veto replaces the teacher's target distribution with a product of teacher and student probabilities, stabilizing on-policy knowledge distillation and improving small-model benchmarks.
-
ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence
ABKD replaces forward and reverse KL distillation with a two-parameter alpha-beta divergence that interpolates between them, improving results on 17 language and vision benchmarks.
-
Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Framework
POCL wraps LLM knowledge distillation in a curriculum that increases data difficulty and temperature over stages, improving Rouge-L on small GPT-2 and OPT students, though ablations show temperature drives the gains.
-
Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models
A broad survey of knowledge distillation for LLMs that summarizes published methods but contains no new results and several citation errors.
Discussion (0). Continue with ORCID to comment.