REVIEW 5 cited by
Direct Preference Knowledge Distillation for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In the field of large language models (LLMs), Knowledge Distillation (KD) is a critical technique for transferring capabilities from teacher models to student models. However, existing KD methods face limitations and challenges in distillation of LLMs, including efficiency and insufficient measurement capabilities of traditional KL divergence. It is shown that LLMs can serve as an implicit reward function, which we define as a supplement to KL divergence. In this work, we propose Direct Preference Knowledge Distillation (DPKD) for LLMs. DPKD utilizes distribution divergence to represent the preference loss and implicit reward function. We re-formulate KD of LLMs into two stages: first optimizing and objective consisting of implicit reward and reverse KL divergence and then improving the preference probability of teacher outputs over student outputs. We conducted experiments and analysis on various datasets with LLM parameters ranging from 120M to 13B and demonstrate the broad applicability and effectiveness of our DPKD approach. Meanwhile, we prove the value and effectiveness of the introduced implicit reward and output preference in KD through experiments and theoretical analysis. The DPKD method outperforms the baseline method in both output response precision and exact match percentage. Code and data are available at https://aka.ms/dpkd.
Forward citations
Cited by 5 Pith papers
-
Dynamics of Learning under User Choice: Overspecialization and Peer-Model Probing
In competitive ML markets, standard gradient training can drive learners into overspecialized equilibria with arbitrarily poor global performance; a proposed 'peer probing' algorithm provably escapes this under inform...
-
DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)
DPO's connection to human choice theory generalizes: any monotone loss can pair with any choice model in a new KLST* class, so loss convexity is not normatively required.
-
Online Knowledge Distillation with Reward Guidance
A preference-based knowledge distillation framework uses a confidence-set reward model in a min-max imitation game, with offline, online, and white-box variants, and outperforms prior KD baselines on LLM benchmarks.
-
Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
Selective-DPO applies DPO only to the top 40% of tokens ranked by policy-reference log-probability difference, reporting improved benchmark scores but with tuned hyperparameters on test sets and omitted negative results.
-
daDPO: Distribution-Aware DPO for Distilling Conversational Abilities
daDPO is a DPO-style objective that blends student and teacher output distributions, improving distilled conversational performance of pruned and small LLMs.
Discussion (0). Sign in to comment.