Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Generative Distribution Distillation

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that knowledge distillation can be reduced to a single generative reconstruction objective, with no explicit classification loss and no loss-weight tuning.

desk verdict A genuinely new and strong empirical KD pipeline wrapped around a theory that does not hold as written; the method deserves serious referee time, but Theorem 1 needs to be fixed or honestly reframed. read the letter →

arxiv 2507.14503 v1 pith:C4EBBBZA submitted 2025-07-19 cs.LG cs.CV

Jiequan Cui , Beier Zhu , Qingshan Xu , Xiaogang Xu , Pengguang Chen , Xiaojuan Qi , Bei Yu , Hanwang Zhang
show 1 more author
Richang Hong
This is my paper
classification cs.LGcs.CV
keywords knowledgedistillationgenerativemodelingdiffusionmodelsfeaturereconstructionsplittokenizationdistributioncontractionunsupervisedImageNetclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that knowledge distillation—normally a multi-term loss balancing student predictions against teacher logits and features—can be posed as a conditional generative problem: train a diffusion model to generate the teacher's feature representation from the student's, and nothing else. The authors argue that a naive version fails because the teacher's features are very high-dimensional and because pure reconstruction carries no label information, and they supply two fixes: Split Tokenization, which chops teacher features into small positional tokens to make diffusion training stable, and Distribution Contraction, which pulls teacher features toward their class centroid so labels enter through the reconstruction target. Their central theoretical claim is that the contracted reconstruction loss matches, at the gradient level, the old multi-task objective of reconstruction plus cross-entropy, so the classification loss and its weight tuning can be dropped. If this holds, supervised distillation becomes a parameter-free single-loss generative pipeline, and unsupervised distillation on unrelated data becomes competitive: a MobileNet trained on CC3M reaches 67.89% top-1 ImageNet accuracy versus 51.60% for the KL baseline. With labels, the same pipeline reaches 82.28% top-1 ImageNet for ResNet-50, which the paper reports as a new state of the art.

What carries the argument

Two mechanisms carry the argument. Split Tokenization rewrites the teacher feature $\hat x_0 = F_t(x)$ as a sequence of non-overlapping low-dimensional tokens $(\hat x^i_0, i, F_s(x))$ and trains the diffusion head to denoise each token conditioned on the student feature and a positional index; this keeps optimization in subspaces of dimension 64 and avoids the collapse observed at token dimensions 512 to 2048. Distribution Contraction forms the training target $\tilde x_0 = \lambda \hat x_0 + (1-\lambda) c_y$, where $c_y$ is the centroid of teacher features of class $y$, and the proof of Theorem 1 identifies $c_y$ with the teacher classifier's weight row $W_y$. The diffusion head is a three-layer MLP with a 1000-step noise schedule, and at inference the feature is generated with 64 sampling steps plus classifier-free guidance at scale 2.0.

What would settle it

Train with Distribution Contraction but measure the gradient cosine similarity between $L_{\text{GenDD}}$ and $L_{\text{multi}} = \gamma_0 L_{\text{noise}} + \gamma_1 L_{\text{CE}}$ on a teacher whose confidence $p(y|\hat x'_0)$ is deliberately low, or with a label-smoothed teacher; if the gradients diverge substantially, the claimed surrogate equivalence is refuted even if accuracy stays high.

Watch

Extended reading notes

Core claim

The paper's central claim is that the GenDD objective—a diffusion noise-prediction loss on teacher features contracted toward their class centroid—is a gradient-level surrogate for the standard multi-task distillation objective that combines a reconstruction loss with a cross-entropy classification loss (Theorem 1, Appendix 6.1). Concretely, with $\tilde x_0 = \lambda \hat x_0 + (1-\lambda) c_y$, the gradient of $L_{\text{GenDD}}$ with respect to the reconstructed feature $\hat x'_0$ decomposes into a reconstruction term plus a term proportional to $(\hat x_0 - c_y)$, which is read as the classification-gradient term under the identification $c_y = W_y$, where $W_y$ is the teacher classifier's weight row for class $y$. The authors conclude that no explicit cross-entropy loss and no multi-step sampling are needed during training, and that a single fixed $\lambda = 0.9$ works across teacher–student pairs. Empirically the paper reports 72.38% (ResNet-34 to ResNet-18) and 73.78% (ResNet-50 to MobileNet) top-1 ImageNet accuracy under a regular recipe, and 82.28% top-1 for ResNet-50 under a 600-epoch strong recipe with a BEiT-v2-Large teacher.

Load-bearing premise

The proof's equivalence holds only if the one-step reconstructed teacher feature is already essentially correct and the teacher is near-certain of its class, and only if the class centroid $c_y$ can be replaced by the teacher classifier's weight row $W_y$—assumptions that break under label smoothing, unreliable teachers, or unnormalized features.

Editorial extensions

If this is right

  • Supervised distillation no longer needs a cross-entropy term or tuned loss weights: a single reconstruction objective with a fixed $\lambda=0.9$ suffices across CNN and Transformer teachers.
  • Unlabeled, non-target data becomes usable for distillation: a MobileNet trained on CC3M reaches 67.89% ImageNet top-1, 16.29 points above the KL baseline.
  • The student learns a generative model of the teacher's representation, so at inference it produces features by sampling; this removes hyperparameter freedom at the cost of 64-step sampling latency.
  • On long-tailed ImageNet-LT, unsupervised GenDD slightly outperforms the label-using IKL-KD baseline (49.31 vs 49.29 for ResNet-50), suggesting labels may not be needed for distillation under imbalance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The gradient-surrogate theorem is stated for clean cross-entropy, but the paper's best ImageNet result uses label smoothing 0.1; under label smoothing the centroid/weight identification cannot literally hold, so the empirical gain may come from contraction acting as feature sharpening rather than from the exact equivalence.
  • If the equivalence is robust, GenDD could extend to semi-supervised distillation where class centers are estimated on the fly from unlabeled data, removing the need for a teacher classifier altogether.
  • The sharp accuracy drop at token dimension 2048 suggests a representation-capacity bottleneck rather than pure optimization instability; testing intermediate dimensions with much longer schedules would separate the two explanations.
  • The fixed $\lambda=0.9$ could become the single new hyperparameter that needs re-tuning in other settings, mirroring the sensitivity the paper criticizes; the ImageNet-LT results already show contraction gives little gain there.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper recasts knowledge distillation (KD) as a conditional generative problem: conditioned on the student feature representation, a diffusion model learns to reconstruct the teacher feature representation. To overcome the high-dimensional optimization difficulty, the authors propose Split Tokenization, which decomposes the target feature into low-dimensional tokens with positional indices. To inject label supervision, they propose Distribution Contraction, which contracts the teacher feature toward the class centroid. The central theoretical claim (Theorem 1, Appendix 6.1) is that the contracted reconstruction objective is approximately equivalent, at the gradient level, to a multi-task objective combining a noise-prediction loss and a cross-entropy classification loss, with constant weighting coefficients. Empirically, the paper reports strong results on balanced, imbalanced, and unlabeled data, including a ResNet-50 top-1 accuracy of 82.28% on ImageNet with a BEiT-v2-Large teacher under the A1 recipe, and improvements of up to 16.29% over the KL baseline in the unsupervised setting on non-target data.

Significance. The conceptual framing of KD as a single generative objective is attractive and, if the gradient-surrogate claim were sound, would represent a substantial theoretical contribution: it would show that explicit classification losses and multi-step sampling can be eliminated without loss of gradient information. The empirical work is carefully executed, with three-trial averages, re-runs of baselines under a cosine schedule for fair comparison, and ablation studies. The Split Tokenization idea and the unsupervised KD results on non-target data are valuable in themselves. However, the proof of Theorem 1 contains multiple fundamental flaws, so the theoretical contribution is not currently established; the paper's core claim therefore needs substantial revision before it can be accepted.

major comments (4)
  1. [Appendix 6.1, Eq. (17)] The proof rewrites the GenDD loss as E||epsilon - epsilon_theta(x'_0)||^2, but this equality does not follow from the preceding equations. In the original loss, the network prediction epsilon_theta is evaluated at the noisy input x~_m, not at the single-step estimate x'_0. The substitution from Eq. (16) to Eq. (17) conflates the denoiser output at the noisy input with the denoiser output at the reconstruction target, so all subsequent gradient computations are for a different objective.
  2. [Appendix 6.1, Eqs. (18)-(25)] The proof compares gradients with respect to x'_0, the single-step reconstruction target, rather than with respect to the trainable parameters theta = (theta_s, theta_diff). The GenDD loss is optimized by differentiating the noise-prediction error through the student encoder and the diffusion head; x'_0 is a derived quantity, not a training variable. Equality of gradients in the target space does not transfer to equality of parameter gradients, because the two objectives depend on theta through different computational graphs, especially since the multi-task cross-entropy term is evaluated on a multi-step sampled x'_0.
  3. [Appendix 6.1, Eqs. (21) and (25)] To match the two gradient expressions, the weighting coefficient gamma_1 must equal 2(1-lambda)*bar_alpha_m/(1-bar_alpha_m), which depends on the diffusion timestep m. The theorem states that gamma_0 and gamma_1 are constants. Thus the claimed equivalence cannot hold as stated for a single pair of constants.
  4. [Appendix 6.1, Eq. (23) and Definition 1] The proof identifies the class center c_y, which is the feature centroid in Definition 1, with the classifier weight row W_y. These are generally different objects, and the identification is not guaranteed under label smoothing or unnormalized features. Moreover, the argument requires p(y|x'_0) approaching 1 and near-perfect reconstruction so that x''_0 is approximately x'_0; these assumptions essentially encode the conclusion that the multi-task objective is already saturated, and they are stated informally without justification.
minor comments (4)
  1. [Section 4.2.1] The text states that a consistent lambda = 0.9 is used, but Figure 4b and the surrounding discussion report the hyperparameter as 1 - lambda = 0.9. Please clarify the notation and which value is actually used in the experiments.
  2. [Section 4.1, Table 3] The table's grouping is unclear: the caption says 'w/o Label On Target Data' but the table also contains non-target data rows. Please make the grouping by target/non-target data explicit and consistent.
  3. [Section 3.2.2, Definition 1] The class centers c_y are not defined computationally. Please specify how they are computed or updated in practice, e.g., as running averages of teacher features over the training set.
  4. [Section 4.2.1, Table 2] For the A1 training recipe, only KD and GenDD results are reported. To substantiate the claim of a new state-of-the-art, please include the same baselines (e.g., DKD, ReviewKD, IKL-KD) under the A1 recipe or justify their omission.

Circularity Check

2 steps flagged · score 6.0 of 10

Theorem 1's gradient-level surrogate claim is constructed rather than derived: the proof sets c_y = W_y and assumes near-one-hot teacher confidence, so the contracted-reconstruction gradient and the multi-task gradient match by construction.

  1. self definitional [Definition 1 (Section 3.2.2) and Appendix 6.1, Eqs. (23)-(25)]
    "∇ˆx′0 Lmulti = γ0[2¯αm/(1−¯αm)(ˆx′0 − ˆx0)] + γ1[(ˆx′′0 − cy)], where ˆx′′0 = Σ_i[p(i|ˆx′0)Wi], and cy = Wy. Note that we also use cy = Wy in Definition 1."

    Definition 1 defines the contraction target with c_y as "the centroid of features for class y"; the proof then identifies c_y with W_y, the teacher classifier row. The cross-entropy gradient for the teacher's linear softmax classifier is x''_0 − W_y, so after this identification and the additional x''_0 ≈ x'_0 assumption, the multi-task label gradient becomes x'_0 − c_y, exactly the second term of the contracted reconstruction gradient in Eq. (21). The claimed gradient-level surrogate is thus not an independent consequence of diffusion training; it is built into the choice of contraction center and the identification of that center with the classifier weight vector.

  2. other [Appendix 6.1, paragraph after Eq. (25)]
    "In Eq. 25, we observe that ˆx′′0 ≈ ˆx′0 when the predicted probability p(y|ˆx′0) approaches 1.0. ... the improved quality of ˆx′0 naturally leads to higher classification confidence, making the approximation ˆx′′0 ≈ ˆx′0 practically valid."

    This is the load-bearing step that collapses the cross-entropy gradient Σ_i p(i|x'_0)W_i − W_y to x'_0 − W_y. The required condition, p(y|x'_0) → 1, is the desired end state of classification training, not a neutral regularity condition. The theorem therefore assumes that the generated features are already classified with near-certain confidence in order to prove that the contracted reconstruction objective matches the multi-task objective; this presupposes the very effect the multi-task loss is meant to create. Absent this assumption, the two gradients differ by Σ_i p(i|x'_0)W_i − x'_0, which is not shown to vanish.

full rationale

The paper's unsupervised results are externally benchmarked and self-contained, and I found no load-bearing self-citation or imported uniqueness argument. The circularity is localized to the theoretical claim. In Theorem 1, the class-center c_y of Definition 1 is silently set equal to the teacher classifier row W_y, and the proof further assumes p(y|x'_0) → 1 so that x''_0 ≈ x'_0. Together these two choices make the label-gradient term of L_multi literally equal to the contraction term of L_GenDD, so the gradient-level equivalence is an equality by construction rather than a derived theorem. The proof also compares gradients with respect to the reconstruction target x'_0 rather than the trainable parameters θ, and matching the terms forces γ1 = 2(1−λ)ᾱ_m/(1−ᾱ_m), which depends on the diffusion timestep m and contradicts the theorem's claim that γ1 is a constant; these are correctness gaps rather than circularities, but they compound the definitional issue. The empirical SOTA claims rest on controlled comparisons against KD/DKD/IKL-KD under the same recipe and are not themselves circular. Overall score 6: the central theoretical 'prediction' reduces by construction, while the empirical pipeline retains independent content.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method adds two hand-chosen mechanisms: the contraction coefficient λ and the class-center target cy, plus standard diffusion hyperparameters. The theorem's proof silently adds two ad hoc assumptions (cy = Wy, and near-perfect reconstruction so gradient terms align). No new physical entities are introduced; the contracted target is an algebraic transformation of existing quantities.

free parameters (3)
  • λ (Distribution Contraction coefficient) = 0.9 claimed in Section 4.2.1; ablation (Fig. 4b) indicates optimum at 1−λ = 0.9 (λ = 0.1)
    Controls how strongly the training target is pulled toward the class center; the central mechanism of the supervised method and of Theorem 1.
  • Class centers cy (contraction targets) = per-class vectors, empirical centroids of teacher features (computation unspecified; proof equates them with…
    Used as contraction targets in Eq. 7; the paper does not state how they are computed, updated, or normalized during training.
  • Diffusion and tokenization hyperparameters = M=1000, 64-step sampling, token dim 64, CFG scale 2.0
    Chosen hyperparameters required to reproduce the results; token dim critically affects the "high-dimensional disaster" claim (Fig. 4a).
assumptions (4)
  • domain assumption DDPM noise-prediction objective is a valid training target for teacher features conditioned on student features
    The method assumes the diffusion loss in Eqs. 2 and 8 trains a conditional generator faithful enough that argmax Ct(x′0) equals the teacher's own classification; no generation-quality metric is reported.
  • domain assumption The teacher is "well-trained" so reconstructed features are confidently classified
    Invoked in Appendix 6.1 to justify x′′0 ≈ x′0; not quantified and stated only as an informal assumption.
  • ad hoc to paper cy (feature centroid) equals Wy (linear classifier row)
    Definition 1 defines cy as the class feature centroid; the proof sets cy = Wy without comment, and the two objects differ under label smoothing and non-normalized features.
  • ad hoc to paper Gradients w.r.t. the single-step estimate x′0 can be computed while freezing the network input path
    Eqs. 12-21 differentiate the loss treating x′0 as an independent variable, ignoring that the diffusion head's input is a function of the target; this is a pseudo-gradient, not the actual training gradient.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generative Distribution Distillation." pith.science (2026). https://pith.science/paper/C4EBBBZA

@misc{pith2026250714503,
  author       = {Pith},
  title        = {Pith review of: Generative Distribution Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C4EBBBZA}},
  note         = {Machine review of arXiv:2507.14503}
}
read the original abstract

In this paper, we formulate the knowledge distillation (KD) as a conditional generative problem and propose the \textit{Generative Distribution Distillation (GenDD)} framework. A naive \textit{GenDD} baseline encounters two major challenges: the curse of high-dimensional optimization and the lack of semantic supervision from labels. To address these issues, we introduce a \textit{Split Tokenization} strategy, achieving stable and effective unsupervised KD. Additionally, we develop the \textit{Distribution Contraction} technique to integrate label supervision into the reconstruction objective. Our theoretical proof demonstrates that \textit{GenDD} with \textit{Distribution Contraction} serves as a gradient-level surrogate for multi-task learning, realizing efficient supervised training without explicit classification loss on multi-step sampling image representations. To evaluate the effectiveness of our method, we conduct experiments on balanced, imbalanced, and unlabeled data. Experimental results show that \textit{GenDD} performs competitively in the unsupervised setting, significantly surpassing KL baseline by \textbf{16.29\%} on ImageNet validation set. With label supervision, our ResNet-50 achieves \textbf{82.28\%} top-1 accuracy on ImageNet in 600 epochs training, establishing a new state-of-the-art.

Figures

Figures reproduced from arXiv: 2507.14503 by the authors.

Figure 2
Figure 2. Sensitivity to loss weights of KD (Hinton et al., 2015). The accuracy of student models varies with different loss weights. Optimal loss weight varies with dif￾ferent teacher-student configurations. these methods in diverse application scenarios. This problem could exacerbate as the number of loss weight increases. Generative Distribution Distillation (GenDD). Inspired by the success of gen￾erative learning (Li et a… view at source ↗
Figure 3
Figure 3. Conditional generation for KD. As shown in [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. Ablation studies on Split Tokenization and Distribution Contraction. (a) Top-1 Ac￾curacy with different token dimension for Split Tokenization. The teacher-student configuration of (ResNet-50, MobileNet) is used on ImageNet.; (b) Top-1 Accuracy under different 1 − λ values for Distribution Contraction. The teacher-student configuration of (ResNet-34, ResNet-18) is used on ImageNet [PITH_FULL_IMAGE:figures/full_fig_… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Ablation studies on optimizer and learning rate schedule. (a) Comparison be￾tween AdamW and SGD optimizer for GenDD with teacher-student configuration of (ResNet-32x4, ResNet-8x4) on CIFAR; (b) Comparison between Step and Cosine learning rate schedule for GenDD with te…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Visual Token Compression Enhances Robustness of MLLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Pruning visual tokens farthest from the text feature space at selected 'robust' layers improves MLLM jailbreak defense (average +13.29% RAR) and slightly reduces hallucination.

  2. Class-frequency Guided Noise Schedule for Diffusion Models

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Proposes CFRG noise schedule for diffusion models that assigns larger noises to low-frequency classes to improve generation on imbalanced datasets.

Reference graph

Works this paper leans on

59 extracted references · 23 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Block diffusion: Interpolating between autoregressive and diffusion language models

    Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09573, 2025

    arXiv 2025

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  5. [5]

    Learning imbalanced datasets with label-distribution-aware margin loss

    Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems, 32, 2019

    2019

  6. [6]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 3558--3568, 2021

    work page 2021

  7. [7]

    Distilling knowledge via knowledge review

    Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, 2021

    work page 2021

  8. [8]

    On the efficacy of knowledge distillation

    Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, pp.\ 4794--4802, 2019

    work page 2019

Show all 59 references
  1. [9]

    Parametric contrastive learning

    Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 715--724, 2021

    2021

  2. [10]

    Reslt: Residual learning for long-tailed recognition

    Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, and Jiaya Jia. Reslt: Residual learning for long-tailed recognition. IEEE transactions on pattern analysis and machine intelligence, 45 0 (3): 0 3695--3706, 2022

    2022

  3. [11]

    Generalized parametric contrastive learning

    Jiequan Cui, Zhisheng Zhong, Zhuotao Tian, Shu Liu, Bei Yu, and Jiaya Jia. Generalized parametric contrastive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

    2023

  4. [12]

    Decoupled kullback-leibler divergence loss

    Jiequan Cui, Zhuotao Tian, Zhisheng Zhong, Xiaojuan Qi, Bei Yu, and Hanwang Zhang. Decoupled kullback-leibler divergence loss. Advances in Neural Information Processing Systems, 37: 0 74461--74486, 2024 a

    2024

  5. [13]

    Classes are not equal: An empirical study on image recognition fairness

    Jiequan Cui, Beier Zhu, Xin Wen, Xiaojuan Qi, Bei Yu, and Hanwang Zhang. Classes are not equal: An empirical study on image recognition fairness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 23283--23292, 2024 b

    2024

  6. [14]

    Generalized kullback-leibler divergence loss

    Jiequan Cui, Beier Zhu, Qingshan Xu, Zhuotao Tian, Xiaojuan Qi, Bei Yu, Hanwang Zhang, and Richang Hong. Generalized kullback-leibler divergence loss. arXiv preprint arXiv:2503.08038, 2025

    2025 arXiv

  7. [15]

    Class-balanced loss based on effective number of samples

    Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 9268--9277, 2019

    2019

  8. [16]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

  9. [17]

    Unified autoregressive visual generation and understanding with continuous tokens

    Lijie Fan, Luming Tang, Siyang Qin, Tianhong Li, Xuan Yang, Siyuan Qiao, Andreas Steiner, Chen Sun, Yuanzhen Li, Tao Zhu, et al. Unified autoregressive visual generation and understanding with continuous tokens. arXiv preprint arXiv:2503.13436, 2025

    2025 arXiv

  10. [18]

    Born again neural networks

    Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In ICML, pp.\ 1607--1616. PMLR, 2018

    2018

  11. [19]

    Discrete flow matching

    Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. Advances in Neural Information Processing Systems, 37: 0 133345--133385, 2024

    2024

  12. [20]

    Mean flows for one-step generative modeling

    Zhengyang Geng, Mingyang Deng, Xingjian Bai, J Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. arXiv preprint arXiv:2505.13447, 2025

    2025 arXiv

  13. [21]

    Vanillakd: Revisit the power of vanilla knowledge distillation from small scale to large scale

    Zhiwei Hao, Jianyuan Guo, Kai Han, Han Hu, Chang Xu, and Yunhe Wang. Vanillakd: Revisit the power of vanilla knowledge distillation from small scale to large scale. arXiv preprint arXiv:2305.15781, 2023

    2023 arXiv

  14. [22]

    A comprehensive overhaul of feature distillation

    Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In ICCV, 2019

    2019

  15. [23]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. 2015

    2015

  16. [24]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 0 6840--6851, 2020

    2020

  17. [25]

    Knowledge distillation from a stronger teacher

    Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from a stronger teacher. In NeurIPS, 2022

    2022

  18. [26]

    Knowledge diffusion for distillation

    Tao Huang, Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge diffusion for distillation. Advances in Neural Information Processing Systems, 36: 0 65299--65316, 2023

    2023

  19. [27]

    Decoupling representation and classifier for long-tailed recognition

    Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.09217, 2019

    1910 arXiv

  20. [28]

    Auto-encoding variational bayes, 2013

    Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013

    2013

  21. [29]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009

    2009

  22. [30]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  23. [31]

    Autoregressive image generation without vector quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37: 0 56424--56445, 2024

    2024

  24. [32]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  25. [33]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36: 0 34892--34916, 2023

    2023

  26. [34]

    Large-scale long-tailed recognition in an open world

    Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 2537--2546, 2019

    2019

  27. [35]

    Wasserstein distance rivals kullback-leibler divergence for knowledge distillation

    Jiaming Lv, Haoyuan Yang, and Peihua Li. Wasserstein distance rivals kullback-leibler divergence for knowledge distillation. Advances in Neural Information Processing Systems, 37: 0 65445--65475, 2024

    2024

  28. [36]

    Long-tail learning via logit adjustment

    Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314, 2020

    2007 arXiv

  29. [37]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp.\ 8162--8171. PMLR, 2021

    2021

  30. [38]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  31. [39]

    Relational knowledge distillation

    Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019

    2019

  32. [40]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  33. [41]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  34. [42]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10684--10695, 2022

    2022

  35. [43]

    Fitnets: Hints for thin deep nets

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015

    2015

  36. [44]

    Image N et large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, and Michael Bernstein. Image N et large scale visual recognition challenge. IJCV, 2015

    2015

  37. [45]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp.\ 2256--2265. pmlr, 2015

    2015

  38. [46]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020 a

    2010 arXiv

  39. [47]

    Generative modeling by estimating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurIPS, volume 32, 2019

    2019

  40. [48]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020 b

    2011 arXiv

  41. [49]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37: 0 84839--84865, 2024

    2024

  42. [50]

    Contrastive representation distillation

    Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020

    2020

  43. [51]

    Vila-u: a unified foundation model integrating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04429, 2024

    2024 arXiv

  44. [52]

    Mmada: Multimodal large diffusion language models

    Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505.15809, 2025

    2025 arXiv

  45. [53]

    Deep mutual learning

    Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, pp.\ 4320--4328, 2018

    2018

  46. [54]

    Decoupled knowledge distillation

    Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In CVPR, 2022

    2022

  47. [55]

    Transfusion: Predict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024

    2024 arXiv

  48. [56]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  49. [57]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  50. [58]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  51. [59]

    A naive GenDD baseline encounters two major challenges: the curse of high-dimensional optimization and the lack of semantic supervision from labels

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.