Pith. sign in

REVIEW 3 major objections 5 minor 15 references

ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-parameter α-β divergence, used as the distillation loss, interpolates between forward and reverse KL divergence, balancing hard-error focus against confident-class focus and beating both extremes on language and vision benchmarks.

desk verdict A useful two-parameter distillation loss with broad empirical support; the concentration-story theory is plausible but has a genuine gap around the logit-gradient link, and the tuning protocol is per-dataset. read the letter →

arxiv 2505.04560 v3 pith:QXNFSYF6 submitted 2025-05-07 cs.LG

Guanghui Wang , Zhiyong Yang , Zitai Wang , Shi Wang , Qianqian Xu , Qingming Huang This is my paper · ORCID
classification cs.LG
keywords knowledgedistillationalpha-betadivergencelogmassratiohardness-concentrationconfidence-concentrationforwardandreverseKLinstruction-followingCLIP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Knowledge distillation trains a small student to imitate a large teacher's output distribution, usually by minimizing forward or reverse Kullback-Leibler divergence, and the paper argues that both choices misallocate probability mass: FKLD spreads the student too evenly across classes, while RKLD collapses it onto the target class and discards the teacher's soft-label information. The paper tries to establish that this is best understood as an entanglement of two effects, hardness-concentration (weighting classes with large teacher-student error) and confidence-concentration (weighting classes the student already assigns high probability), and that a two-parameter α-β divergence lets a practitioner tune the two effects independently. If this is right, the payoff is a drop-in loss change: improvements of 0.81 to 3.31 ROUGE-L over FKLD and RKLD on five instruction-following benchmarks, plus accuracy gains on CIFAR-100 and base-to-new CLIP distillation, with no extra trainable parameters. The analysis works by tracking the log mass ratio, the one-step change in each class's log probability, which is proportional to that class's logit gradient.

What carries the argument

The load-bearing object is the log mass ratio $LogR_t(y) = \log(q_{t+1}(y)/q_t(y))$, the one-step change in a class's probability under the distillation loss, which equals a negative multiple of the loss gradient with respect to that class's logit plus a normalization term. Proposition 4.2 bounds $|LogR|$ for the $\alpha$-$\beta$-divergence as a product of two separable factors, $q_t(y)^\beta$ and $|p(y)^\alpha - q_t(y)^\alpha|/\alpha$, so that $\beta$ and $\alpha$ independently set the strength of confidence-concentration and hardness-concentration. Because $(\alpha,\beta) = (1,0)$ recovers FKLD and $(0,1)$ recovers RKLD, the same one-step analysis places every other divergence in the family on an interpolation path between the two extremes.

What would settle it

Sweep β at a fixed α on one instruction-following dataset, train to completion, and measure the student's output entropy and target-class probability: the paper predicts a monotone shift from FKLD-like flatness to RKLD-like sharpness, so a non-monotone response, or final entropies identical across β, would falsify the concentration mechanism it proposes.

Watch

Extended reading notes

Core claim

The central claim is that FKLD and RKLD sit at two opposite corners of a single trade-off and are therefore both suboptimal distillation objectives. Within one gradient step the change in a class's log probability obeys $|LogR_t(y)| \leq \eta\, q_t(y)^\beta\, |p(y)^\alpha - q_t(y)^\alpha|/\alpha + \eta\, q_t(y) \sum_k q_t(k)^\beta |p(k)^\alpha - q_t(k)^\alpha|/\alpha + |N_t(y)|$, where the factor $q_t(y)^\beta$ controls confidence-concentration and the factor $|p^\alpha - q^\alpha|/\alpha$ controls hardness-concentration. FKLD is recovered at $(\alpha,\beta) = (1,0)$, where both effects are weak, and RKLD at $(\alpha,\beta) = (0,1)$, where both are strong; interpolating between them, the paper argues, keeps the teacher's distributional information while still concentrating on the target class. The experimental claim is that this loss replacement alone yields ROUGE-L gains of 0.81 to 3.31 over FKLD and RKLD on five instruction-following datasets and improves accuracy on CIFAR-100 and base-to-new CLIP settings across twelve teacher-student configurations.

Load-bearing premise

The central story presumes that when the student's overall gradient with respect to its weights goes to zero, each individual class's logit gradient goes to zero as well; that inference holds only if the mapping from weights to logits has full column rank, a property deep networks do not guarantee.

Editorial extensions

If this is right

  • Choosing $\alpha$ and $\beta$ between the FKLD and RKLD corners produces a loss that trades hardness-concentration against confidence-concentration, avoiding both the over-smoothing of FKLD and the target-class collapse of RKLD.
  • Because the change is confined to the loss, ABKD can be applied on top of existing logit-based distillation methods such as DKD, LSD, and TTM, improving their accuracy on CIFAR-100 in the reported experiments.
  • On instruction-following, the method achieves superior or comparable results to systems that sample student outputs during training (GKD, DISTILLM) while running at the same speed as vanilla KD, since it only changes the loss.
  • The same $\alpha$-$\beta$ family contains the Hellinger distance, $\beta$-divergence, and squared Euclidean distance as special cases, while the $\alpha$-divergence is confined to the submanifold $\alpha + \beta = 1$, which the paper's analysis predicts is a less flexible interpolant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A diagnostic reading of the paper: the two factors in the Proposition 4.2 bound are observable during training (per-class student confidence and per-class error), so one could estimate a good $(\alpha,\beta)$ region before a full run and convert the reported grid search into a per-task recipe.
  • The log-mass-ratio argument is not specific to teacher-student distillation; it should transfer to other soft-label training such as preference optimization, where a similar imbalance between confident and hard tokens has been observed, making the $\alpha$-$\beta$ family a candidate controllable loss there.
  • The theory is a bound on a single gradient step; a testable extension is to log $|LogR|$ across an entire training run and check whether mid-range $(\alpha,\beta)$ actually sits between FKLD and RKLD in final concentration behavior, or whether the trajectory delocalizes the promised interpolation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ABKD, a knowledge-distillation objective based on the α-β-divergence D_AB(p∥q) with two tunable hyperparameters. The authors argue that FKLD and RKLD represent two extreme forms of two 'mode-concentration effects': hardness-concentration (weighting classes by the teacher-student error) and confidence-concentration (weighting classes by student confidence). They derive one-step bounds on the log mass ratio LogR_t(y) for FKLD, RKLD, and the α-β-divergence, and use these to claim that ABKD smoothly interpolates between the two extremes. The method is evaluated on five instruction-following datasets with GPT-2 teacher-student pairs and on CIFAR-100 and CLIP base-to-new vision settings, reporting consistent ROUGE-L and accuracy gains over several baselines, with code released.

Significance. If the empirical results hold, ABKD is a broadly useful and simple contribution: it changes only the distillation loss, adds no trainable parameters, reports gains up to 3.31 ROUGE-L over FKLD/RKLD, and shows improvements on both standard CIFAR-100 distillation and CLIP base-to-new generalization. The empirical section is extensive, includes standard deviations over five seeds for NLP, compares against re-implemented baselines, and releases code. The gradient algebra in Proposition 4.2 and Appendix G.5 is straightforward and appears correct where it is shown in full. However, the theoretical narrative that the paper relies on for its central claim is not fully supported: the link from parameter-space optimization to logit-space concentration is asserted rather than proved, and several load-bearing proof steps are deferred or omitted.

major comments (3)
  1. [App. C.2 / Sec. 3.1] The central theoretical narrative hinges on the claim that training reduces |LogR_t(y)|. The paper analyzes gradients with respect to logits, ∇_f ℓ, while actual training uses parameter gradients given by ∇_W ℓ = J^T ∇_f ℓ with J = ∂f/∂W (C×M). To conclude from ∇_W ℓ → 0 that ∇_f ℓ → 0, the paper asserts in App. C.2 that J is 'close to being full rank C' because M ≫ C. This is not established for deep-network optimization trajectories: full column rank is a generic condition, and 'close to full rank' does not provide the uniform lower bound on singular values needed for the asymptotic implication. If J^T has a near-nullspace containing ∇_f ℓ, the parameter gradient can be small while the logit gradient, and hence |LogR_t(y)|, remains large. Consequently, the one-step bounds in Proposition 4.2 and Theorem D.1 are not yet connected to the behavior of the trained student distribution, which is the load-bearing step of the paper's theoretical argument.
  2. [App. G.2 and App. G.6] Several load-bearing proofs are omitted or incomplete. The formal version of Theorem 3.2 in App. G.2 explicitly omits Case 3, saying it is 'consistent with Tajwar et al. (2024)', and Theorem D.1 in App. G.6 omits Case 2, saying it is 'similar to Case 1'. These are not peripheral: Case 3 of Theorem 3.2 supports the claim that RKLD preferentially increases mass on underestimated classes with larger q_t(y), and Case 2 of Theorem D.1 supports the monotonic hardness-concentration behavior in α. Additionally, in the proof of Case 4 of Theorem 3.2, the inequality in Eq. (35) is justified by 'choosing a sufficiently large p(y1)', but p(y1) is fixed by the case assumptions; the proof must show that the inequality holds for the stated range rather than treating p(y1) as a free parameter.
  3. [Eq. (2) and Prop. 4.2] The gradient analysis is performed on the KD term D_AB(p∥q_t) alone, but the objective that is actually trained is ℓ_CE + λD_AB as in Eq. (2). The cross-entropy term contributes an additional logit-gradient component q_t(y) − 1_y, which changes the log-mass-ratio dynamics and does not vanish when p = q_t; this term is present in the vision experiments, where λ = 32 is used with CE. The paper should either prove that the CE term does not alter the monotonicity and concentration conclusions, or explicitly state the regime in which it is negligible. The current one-step treatment of the KD term alone is insufficient to support the claim that the full training dynamics achieve the described probability-mass allocation.
minor comments (5)
  1. [Throughout] There are several typos that should be corrected, including 'limitaions' in the Section 3 heading, 'divergvence' in Definition 4.1, 'ont-hot label' in Section 3.2, and 'task-agostic' in the Table 2 caption.
  2. [Sec. 3.1] The quantity LogR^A_t(y) is defined for a generic algorithm A, but later equations use LogR^F_t, LogR^R_t, and LogR^{(α,β)}_t; the superscript notation should be defined or explained before first use.
  3. [Sec. 3.1 / Prop. 3.1 / Prop. 4.2] The normalization residual N_t is characterized inconsistently: Sec. 3.1 states that it vanishes when all class-channel gradients vanish, while Propositions 3.1 and 4.2 state that it vanishes when p = q_t. These are different conditions and should be reconciled.
  4. [App. G.2, after Eq. (31)] The proof of Case 4 of Theorem 3.2 writes D_KL(p∥q) in Eq. (31), but the RKLD gradient naturally contains the reverse KL term D_KL(q∥p); please check which KL direction is intended and correct the notation.
  5. [Tab. 4 and App. J.2.3] Several hyperparameter choices in Tab. 4 fall outside [0,1] (e.g., α=1.2, β=−0.1 in ABLSD), while App. J.2.3 argues that values outside [0,1] degrade performance and recommends searching within [0,1]; the search protocol and this guidance should be reconciled.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical gains are evaluated against external benchmarks, and the α-β interpolation analysis is a direct derivation from the chosen divergence rather than a concealed reuse of inputs.

full rationale

The paper's central empirical claim is independent of any circular input: ABKD is tested against FKLD, RKLD, several published KD baselines, and external benchmarks (Tabs. 2, 3, 5, 14, and appendices), with hyperparameters tuned on validation sets in the usual way. No fitted parameter is renamed as a prediction. The theoretical narrative proceeds by deriving an exact one-step log-mass-ratio expression (App. C.1, Eq. 11) and then bounding it for FKLD, RKLD, and α-β-divergence (Prop. 3.1 and Prop. 4.2). The concepts of hardness-concentration and confidence-concentration are explicitly defined from the terms of these derived bounds, so the statement that α and β modulate those terms is a direct consequence of the derivation rather than a hidden identity with the assumption. The claim that FKLD and RKLD are extremes is also read off the same derived expressions, not imported from a self-citation chain. The only notable technical gap is the full-column-rank assumption on the Jacobian in App. C.2, which is a correctness caveat about the convergence interpretation, not a circularity: even if that assumption were false, the one-step gradient formulas would still stand. Self-citations in the related-work sections are not load-bearing for the paper's main argument, and the omitted proof of Theorem 3.2 Case 3 is delegated to the external work of Tajwar et al. (2024), which constitutes independent support rather than self-referential reasoning. The α-β-divergence itself is taken from Cichocki et al. (2011), an external source. Overall, the derivation chain is self-contained in the relevant sense: the loss is defined, its gradient is computed, bounds are proven, and the resulting method is validated against outside benchmarks. No step reduces, by construction or by self-citation, to its own input.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The two mode-concentration effects are interpretive labels for terms already present in the gradient bounds, so they are treated as assumptions about how to read the algebra rather than as invented objects with independent evidence.

free parameters (3)
  • alpha = 0.2 for NLP; 0.5 to 1.2 across vision settings
    Controls hardness-concentration in the alpha-beta divergence; selected per task, dataset, and architecture in Tabs. 4 and 6, with a single fixed value 0.2 for all NLP instruction-following runs (App. I.1.3).
  • beta = 0.7 for NLP; -0.2 to 0.5 across vision settings
    Controls confidence-concentration; selected per task, dataset, and architecture in Tabs. 4 and 6, with beta = 0.7 for all NLP runs (App. I.1.3).
  • lambda = 32 for CIFAR-100; searched in {100, ..., 3000} for base-to-new CLIP
    Weight of the distillation term in Eq. (7); tuned on validation performance in App. I.2.3.
assumptions (5)
  • domain assumption Student and teacher outputs are softmax probability distributions over C classes.
    Used throughout Sec. 3.1 and 4.2 to define log mass ratio and derive gradient bounds; all theoretical claims are about softmax-parameterized categorical distributions.
  • domain assumption The Jacobian J of logits with respect to model parameters is full column rank, or close to it.
    App. C.2 asserts J is close to full rank for GPT-2 and ResNet so that vanishing total gradient implies vanishing logit gradients; this is load-bearing for interpreting |LogR| reduction as convergence.
  • standard math The alpha-beta divergence extends continuously to all real alpha and beta via L'Hopital's rule.
    Definition 4.1 and App. B rely on the known continuous extension from Cichocki et al. (2011) to recover FKLD, RKLD, Hellinger, and other cases as limiting parameters.
  • ad hoc to paper One gradient step on the logits, ignoring the cross-entropy term and the residual normalization term N_t, is a valid proxy for training dynamics.
    Props. 3.1 and 4.2 analyze a single softmax-gradient step; the normalization residuals vanish only at p = q, and the full loss in Eq. (2) contains an additional CE term that is not included in the concentration analysis.
  • ad hoc to paper The two concentration effects are faithfully represented by the factors |p(y)^a - q_t(y)^a| / a and q_t(y)^b in the Prop. 4.2 bound.
    The conclusion that FKLD has weak effects and RKLD has strong effects is read directly from this decomposition; it is an interpretive assumption rather than an independently established property.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence." pith.science (2026). https://pith.science/paper/QXNFSYF6

@misc{pith2026250504560,
  author       = {Pith},
  title        = {Pith review of: ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QXNFSYF6}},
  note         = {Machine review of arXiv:2505.04560}
}
abstract

Knowledge Distillation (KD) transfers knowledge from a large teacher model to a smaller student model by minimizing the divergence between their output distributions, typically using forward Kullback-Leibler divergence (FKLD) or reverse KLD (RKLD). It has become an effective training paradigm due to the broader supervision information provided by the teacher distribution compared to one-hot labels. We identify that the core challenge in KD lies in balancing two mode-concentration effects: the \textbf{\textit{Hardness-Concentration}} effect, which refers to focusing on modes with large errors, and the \textbf{\textit{Confidence-Concentration}} effect, which refers to focusing on modes with high student confidence. Through an analysis of how probabilities are reassigned during gradient updates, we observe that these two effects are entangled in FKLD and RKLD, but in extreme forms. Specifically, both are too weak in FKLD, causing the student to fail to concentrate on the target class. In contrast, both are too strong in RKLD, causing the student to overly emphasize the target class while ignoring the broader distributional information from the teacher. To address this imbalance, we propose ABKD, a generic framework with $\alpha$-$\beta$-divergence. Our theoretical results show that ABKD offers a smooth interpolation between FKLD and RKLD, achieving an effective trade-off between these effects. Extensive experiments on 17 language/vision datasets with 12 teacher-student settings confirm its efficacy. The code is available at https://github.com/ghwang-s/abkd.

Figures

Figures reproduced from arXiv: 2505.04560 by the authors.

Figure 1
Figure 1. (a) Illustration of the unified search space for our proposed ABKD, where height (color) represents performance (↑). The FKLD and RKLD are special cases of ABKD when selecting (α = 1, β = 0) and (α = 0, β = 1) , respectively. The α-divergence can only search along the submanifold α + β = 1 in the ABKD space. (b)-(c) illustrate how adjusting α and β affects hardness-concentration and confidence-concentration. (d)-(g)… view at source ↗
Figure 2
Figure 2. Performance across different loss functions on the validation set. (a) Training Speed (b) Effects of SGOs [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Comparison of training speeds and the effects of using SGOs. Please see Sec. I.1.2 for details of different SGOs strategies. our approach consistently achieves superior or comparable results, especially on Super-Natural and Unnatural datasets. Efficiency Comparison [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Accuracy on CIFAR-100 for student models trained with different distillation methods. ABDKD, ABTTM, ABLSD, and ABKD are our implementations by rectifying the backbone’s loss function. For details on backbones, please refer to Sec. I.2.2. (a) alpha on CIFAR-100 (b) alph…
Figure 6
Figure 6. Figure 6: Sensitivity analysis of hyperparameters α and β. (a)-(b) For low-dimensional output distribution in CIFAR-100, a smaller α leads to excessive penalization for error with limited gains. However, for higher-dimensional distribution in Dolly (e.g., 50,527 for GPT-2), a we…
Figure 7
Figure 7. Figure 7: The prompt template for training and evaluation of instruction-following task experiments from (Ko et al., 2024; Gu et al., 2024a). I.2. Vision Tasks I.2.1. DATASETS In this section, we provide detailed descriptions of the image datasets used. • CIFAR-100: It is a gene…
Figure 8
Figure 8. Figure 8: Performance on the validation set when distilling GPT-2 XL (1.5B) to GPT-2 (0.1B) [PITH_FULL_IMAGE:figures/full_fig_p042_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 13 canonical work pages

  1. [1]

    Ify1 andy2 are such thatδ1 <q t(y1) =q t(y2)≤p(y 1) (whereδ1 >0 ), andp(y1)≥p(y 2) +ζ , it holds that∆α1 t (y1,y 2)≥∆ α2 t (y1,y 2)

    Theα-divergence transfers the probability mass of overestimated classes to underestimated ones more aggressively asα decreases. Ify1 andy2 are such thatδ1 <q t(y1) =q t(y2)≤p(y 1) (whereδ1 >0 ), andp(y1)≥p(y 2) +ζ , it holds that∆α1 t (y1,y 2)≥∆ α2 t (y1,y 2). 2.α -divergence reduces the probability mass of classes with larger error|p(y)−q t(y)| more aggr...

    work page 2024

  2. [2]

    If y1 andy2 are such that p(y1)< qt(y1) =q t(y2)≤1−δ 2, but p(y1)≥p(y 2) +ζ , then, ∆r t (y1,y 2)> ∆f t (y1,y 2)

    RKLD reduces the probability mass of overestimated classes with higher error|p(y)−q t(y) more aggressively than FKLD. If y1 andy2 are such that p(y1)< qt(y1) =q t(y2)≤1−δ 2, but p(y1)≥p(y 2) +ζ , then, ∆r t (y1,y 2)> ∆f t (y1,y 2)

    work page

  3. [3]

    The proof is in App

    Theα-β-divergence becomes more (less) preferential in focusing the error on classes with higher student confidence as βincreases (decreases) when reducing LogR(α,β) t (y) . The proof is in App. G.6. Case 1 shows that a smaller α (stronger hardness-concentration) leads to aggressive mass reallocation across classes when some classes are overestimated. Case...

    work page 2024

  4. [4]

    Ify1 andy2 are such thatqt(y2) +ζ≤q t(y1)≤1−δ 2, andc0·qt(y2)<p(y 1) =p(y 2)<c 1·qt(y1), wherec0 and c1 are constants withc 0 >1andc 1 <1, then,∆ r t (y1,y 2)>∆ f t (y1,y 2)

    RKLD reduces probability mass on overestimated classes with larger probabilityqt(y) more conservatively than FKLD. Ify1 andy2 are such thatqt(y2) +ζ≤q t(y1)≤1−δ 2, andc0·qt(y2)<p(y 1) =p(y 2)<c 1·qt(y1), wherec0 and c1 are constants withc 0 >1andc 1 <1, then,∆ r t (y1,y 2)>∆ f t (y1,y 2). 21 ABKD: Pursuing a Proper Allocation of the Probability Mass in Kn...

    work page 2024

  5. [5]

    Ify1 andy2 are such that δ1 < qt(y1) =q t(y2)< p(y 1) (whereδ1 >0,δ 2 >0 ), but p(y1)≥p(y 2) +ζ , then, ∆r t (y1,y 2)>∆ f t (y1,y 2)

    RKLD transfers probability mass from overestimated classes to underestimated classes more aggressively than FKLD. Ify1 andy2 are such that δ1 < qt(y1) =q t(y2)< p(y 1) (whereδ1 >0,δ 2 >0 ), but p(y1)≥p(y 2) +ζ , then, ∆r t (y1,y 2)>∆ f t (y1,y 2)

    work page

  6. [7]

    If y1 andy2 are such that qt(y2) +ζ≤q t(y1)≤1−δ 2, and p(y1) =p(y 2)> c 0·qt(y1), wherec0 is a positive constant>1, then,∆ r t (y1,y 2)>∆ f t (y1,y 2)

    RKLD more preferentially increases probability mass on underestimated classes with larger probabilityqt(y) than FKLD. If y1 andy2 are such that qt(y2) +ζ≤q t(y1)≤1−δ 2, and p(y1) =p(y 2)> c 0·qt(y1), wherec0 is a positive constant>1, then,∆ r t (y1,y 2)>∆ f t (y1,y 2)

    work page

  7. [9]

    (36) Simplifying: ∂q(i) ∂f(i) =q(i)(1−q(i)).(37)

    Casei=j:Using the quotient rule, we have: ∂q(i) ∂f(i) = ef(i)·P kef(k)−ef(i)·ef(i) P kef(k) 2 = efi P kefk − efi·efi (P kefk)2. (36) Simplifying: ∂q(i) ∂f(i) =q(i)(1−q(i)).(37)

    work page

  8. [10]

    q(y) 1−α (p(y)α−q(y) α) +q(y) X k q(k) 1−α(q(k)α−p(k)α) !#! .(47) Taking the logarithm on both sides, we get: logqα t+1(y) qt(y) =η

    Casei̸=j:We have ∂q(i) ∂fj =− efiefj (P kefk)2 =−q(i)q(j). (38) In conclusion, the derivative is: ∂q(i) ∂f(j) = ( q(i)(1−q(i))ifi=j, −q(i)q(j)ifi̸=j. (39) Restate of Proposition F.2.The updates induced byα-divergence forq t within one gradient descent step are given by: logqα t+1(x) qt(x) ≤ηq t(y)1−α | {z } (a) p(y)α−qt(y)α α | {z } (b) +qt(y) X k qt(k)1−...

    work page

Show all 15 references
  1. [11]

    The functionf(p(y 1))is monotonically decreasing with respect top(y 1)

  2. [12]

    Moreover, for allp(y1)> c0, it holds that f(p(y 1))<0

    There exists a unique constantc0∈(q(y 2),q(y 1)) such thatf(c 0) = 0. Moreover, for allp(y1)> c0, it holds that f(p(y 1))<0. 29 ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation viaα-β-Divergence Proof.The derivative offwith respect top(y 1)i...

  3. [13]

    X k q(k) 1−α (q(k)α−p(k)α) # . (121) Consideringf(α)≜∆ α(y1,y 2)/η, taking the derivative with respect toαyields: f′(α) = 1 α2

    Theα-divergence transfers the probability mass of overestimated classes to underestimated ones more aggressively asα decreases. Ify1 andy2 are such thatδ1 <q t(y1) =q t(y2)≤p(y 1) (whereδ1 >0 ), andp(y1)≥p(y 2) +ζ , it holds that∆α1 t (y1,y 2)≥∆ α2 t (y1,y 2). 2.α -divergence ...

  4. [14]

    Proof.First, we prove Case 1

    Theα-β-divergence becomes more (less) preferential in focusing the error on classes with higher student confidence as βincreases (decreases) when reducing LogR(α,β) t (y) . Proof.First, we prove Case 1. Note thatq t(y1) =q t(y2) =q(x), Based on Eq. 132, we have ∆(α,β) t =ηq(x)...

    2024

  5. [340]

    emnlp-main.340

    URL https://aclanthology.org/2022. emnlp-main.340. Wang, Y ., Kordi, Y ., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Align- ing language models with self-generated instructions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.),Pro-...

    2022 arXiv

  6. [364]

    findings-emnlp.364/

    URL https://aclanthology.org/2024. findings-emnlp.364/. Kim, Y . and Rush, A. M. Sequence-level knowledge dis- tillation.CoRR, abs/1606.07947, 2016. URL http: //arxiv.org/abs/1606.07947. Ko, J., Kim, S., Chen, T., and Yun, S.-Y . Distillm: Towards streamlined distillation for ...

    2024 arXiv

  7. [2023]

    a photo of a{classname}

    for training and 500 samples each for validation and testing. After distillation, the models are evaluated on five task- agnostic instruction-following benchmarks: Dolly-evaluation, Self-Instruct, Vicuna-evaluation, Super-Natural Instructions, and Unnatural Instruction. The de...

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.