Pith. sign in

REVIEW 5 major objections 5 minor 51 references

Hybrid Data-Free Knowledge Distillation

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A hybrid of a small real sample and teacher-generated synthetic images trains a student network with 1/120 of the data previous methods need.

desk verdict Useful hybrid distillation method, but the 120x data-reduction claim needs a duplicate check between the collected set and the teacher's training data before I'd trust it. read the letter →

arxiv 2412.13525 v1 pith:PGOMQVTL submitted 2024-12-18 cs.CV

Jialiang Tang , Shuo Chen , Chen Gong This is my paper · ORCID
classification cs.CV
keywords hybriddata-freeknowledgedistillationteacher-guidedGANfeaturealignmentcategoryfrequencysmoothingdatainflationclass-imbalancedgenerationmodelcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a way to compress a large pre-trained image classifier into a small student network when the teacher's original training data is unavailable. Instead of collecting hundreds of thousands of real images or trusting purely synthetic ones, it combines a small set of collected real examples (as few as 5,000 for CIFAR-10) with synthetic images produced by a GAN that the teacher network supervises. The central claim is that this hybrid recipe trains students that match or beat both prior collection-based and generation-based distillation methods, using about 1/120 of the real data earlier collection-based methods needed. If true, data-free distillation becomes practical in domains such as medical imaging where large labeled collections are scarce or private.

What carries the argument

The load-bearing object is the hybrid data distribution $U=\alpha P+(1-\alpha)Q$, where $P$ is the distribution of collected real examples and $Q$ is the distribution of synthetic ones. The paper derives the total-variation-distance bound $\mathrm{TVD}(U,Q)\le(2-\alpha)\,\mathrm{TVD}(P,Q)$, which identifies synthetic-data quality and the real-data proportion $\alpha$ as the two levers controlling the distribution gap between hybrid and synthetic data. The mechanism that actuates these levers is an ADCGAN-style GAN, an auxiliary-classifier GAN whose discriminator is regularized by teacher features via $L_{\mathrm{blend}}$ and $L_{\mathrm{trans}}$, whose generator is balanced by the teacher-derived class-frequency target $L_{\mathrm{reg}}$, and whose outputs are combined with inflated real examples for student training through the feature alignment $L_{\mathrm{align}}$.

What would settle it

Compare HiDFD's student accuracy against a version in which the teacher's confidence on synthetic images is corrected using the small real set before the feature and frequency losses are computed. If the teacher is overconfident on synthetic images, the correction should improve the student; a reader can check this in a single run by logging teacher confidence and correctness on the generated set. The central claim would fail if student accuracy follows the teacher's reliability on synthetic images rather than the quality of the hybrid data.

Watch

Extended reading notes

Core claim

The paper's discovery is that a small, imbalanced set of real collected images is enough to train a strong student, provided the teacher's features and soft predictions are used to regularize a GAN rather than to train the student directly. In the teacher-guided generation module, two feature-integration terms, a blend term $L_{\mathrm{blend}}$ and a transfer term $L_{\mathrm{trans}}$, stop the discriminator from memorizing the few real examples, and a category-frequency smoothing term $L_{\mathrm{reg}}$ rebalances generator training across classes using the teacher's average softmax vector as the target frequency. In the student distillation module, the collected examples are repeated (inflated) so that the hybrid distribution $U=\alpha P+(1-\alpha)Q$ has a larger share $\alpha$ of real data, and the student is trained without labels by aligning its features to the teacher's features, $L_{\mathrm{align}}=\mathbb{E}_{x\sim D}\lVert\Phi_S(x)-\Phi_T(x)\rVert^2$, while sharing the teacher's classifier. Experiments across CIFAR, CINIC, TinyImageNet, ImageNet and the medical HAM dataset show the student reaching accuracy close to, and sometimes better than, a student trained on the full original data.

Load-bearing premise

The method assumes that the teacher network gives trustworthy answers when it looks at synthetic images it never saw during training; if its confidence or feature maps are unreliable on those images, the distillation signals built from them are unreliable too.

Editorial extensions

If this is right

  • A distillation pipeline needs only a small, representative real image set rather than the teacher's private training set, making deployment feasible for medical and privacy-sensitive domains.
  • Students trained by HiDFD reach accuracy comparable to, and in several configurations higher than, students trained on the full original data, despite never seeing original labels.
  • The framework works across teacher-student pairs such as ResNet, VGG, ShuffleNet, and WideResNet and on datasets up to ImageNet scale, indicating it is not tied to one architecture.
  • Because the student ignores example labels during training, label noise in web-collected data does not corrupt the student's classifier, addressing a known failure mode of collection-based distillation.
  • The inflation factor $N$ gives a practical dial for trading real-data influence against synthetic diversity, with moderate inflation ($N\approx 10$) performing best.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same teacher-guided GAN recipe could be applied to other data-scarce vision tasks; if it transfers, a pretrained teacher could synthesize training views for detection or segmentation from a few real examples.
  • Beyond the paper, the TVD bound suggests the inflation factor should be set by the measured distance between collected and synthetic distributions rather than by the fixed $N=\lfloor|D_s|/|D_c|\rfloor$, a testable tuning rule.
  • Beyond the paper, a calibration step on the small real set could correct the teacher's confidence on synthetic inputs before those predictions drive generation and alignment, directly testing whether the paper's main assumption is the limiting factor.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes HiDFD, a hybrid data-free knowledge distillation method that trains a GAN on a small collected dataset (e.g., 5,000 examples, 1/120 of the 600,000 used by collection-based DFKD) under teacher supervision, generates synthetic examples, and then trains the student on the hybrid real/synthetic data via a classifier-sharing feature-alignment loss. The main empirical claim is state-of-the-art performance on CIFAR, CIFAR100, CINIC, TinyImageNet, HAM, and ImageNet, with ablations supporting the contributions of the proposed loss terms and the inflation strategy.

Significance. If the claims hold, the paper addresses a practically important limitation of DFKD: collection-based methods need massive external data and generation-based methods produce low-quality synthetic data. The reported results are strong and the code availability is a positive feature. The ablations in Table 2 and the sensitivity plots give useful evidence that the proposed components help. However, the manuscript contains several technical inaccuracies in the derivations and a potentially serious data-contamination risk for the HAM experiment, so the central 1/120 data-free claim is not yet fully supported.

major comments (5)
  1. [Datasets and Implementation Details (Collected Datasets); Table A-1] The data-free claim requires that the collected set be disjoint from the teacher's original training data, but the paper does not report any duplicate check. Since HAM's original 8,000 training images come from HAM10000 and the collected set is taken from ISIC, which contains HAM10000 images, the ρ=0.1 HAM result in Table 1 may include teacher training examples; without an explicit deduplication step, the HAM row cannot support the 1/120 claim. Please report an overlap analysis and remove any overlapping images.
  2. [Motivation of the Hybrid Learning, Eqs. (2)-(5); Student Distillation] Direct algebra gives TVD(U,Q)=α·TVD(P,Q), not the looser bound in Eq. (5); Eq. (5) is an upper bound that decreases with α, while the exact TVD(U,Q) increases with α. The text after Eq. (5) refers to TVD(U,P), and the Student Distillation section says that a small α produces a large TVD(U,Q), which is the opposite of the exact expression. The motivation for the inflation factor should be reformulated in terms of TVD(U,P)=(1−α)TVD(P,Q) and the intended target distribution.
  3. [Eq. (12), Teacher-Guided Generation (Lreg)] The frequency target p_T is defined as the teacher's average softmax over the current synthetic distribution Q. If the generator is imbalanced, p_T is imbalanced as well, so minimizing KL(p_T ∥ \hat n_t) does not necessarily push the generator toward balanced class frequencies. Please either define a fixed target independent of the current synthetic batch or provide a derivation showing why the self-referential target still enforces balance.
  4. [Eq. (8), Lblend] With p sampled uniformly from [0,1] and q=0.7, the indicator I(p>q) is 1 with probability 0.3, not 0.7 as stated in the text. Please clarify whether the intended behavior is a Bernoulli draw with probability q or a threshold q; if the former, the sampling should be adjusted accordingly, and the sensitivity analysis should be updated.
  5. [Eqs. (8), (9), (12), (14)] All teacher-guided terms assume that teacher features and softmax probabilities are reliable on synthetic images that lie outside the teacher's training distribution. The paper does not measure teacher calibration or feature quality on generated samples. Please add an analysis (e.g., teacher accuracy/confidence on synthetic data, or comparison with a teacher retrained on the collected set) to support this load-bearing assumption.
minor comments (5)
  1. [Abstract and Section 4.1] The phrase '120 times less collected data' is ambiguous; it should be '1/120 of the collected data' or '120× smaller' to match the 5,000/600,000 comparison.
  2. [Appendix, 'Additional Parametric Sensitivities'] The text says q and γ appear in Eq. (9), but q is defined in Eq. (8) and γ in Eq. (10); please correct the equation references.
  3. [Figure 2] The captions for panels (a) and (b) are incomplete ('Analysis of ') and should explicitly state that they show λ_d and λ_g.
  4. [Algorithm A-1 and references] There are typos: 'limted' should be 'limited', and the Krizhevsky reference lists 'University of Tront' instead of 'University of Toronto'.
  5. [Eq. (6)] The derivation of the vanishing gradient is overly informal because the expectation over Q depends on the generator parameters; a clearer treatment would improve readability, though this does not affect the final loss definitions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline data-efficiency claim is an empirical benchmark result, not a consequence of the paper's equations; self-referential design elements do not constitute circular derivation.

full rationale

The strongest claim—1/120 collected-data reduction with SOTA accuracy—is established by Table 1 experiments, not by a formal derivation; the paper's mathematical content (TVD bound in Eqs. 2-5 and the KL equivalence in Eq. 7) consists of standard identities that are not used to generate the reported accuracies. The Lreg target p_T = E[SoftMax(N_T(x))] over the generator's own synthetic distribution is self-referential, and the student reuses the teacher's classifier, so part of the reported accuracy is inherited from the teacher; however, these are properties of the training design, not predictions claimed to follow from first principles, and neither forces the benchmark outcome. Citations to the authors' prior work (Tang et al. 2023) motivate the classifier-sharing heuristic and are independently ablated in Table 2; no uniqueness theorem or external claim is imported to forbid alternatives. The HAM/ISIC overlap concern is a data-contamination/validity risk, not a circularity of derivation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central empirical method relies on teacher guidance, but the paper's theoretical justification has algebraic and logical gaps. The free parameters are standard tuned hyperparameters; no invented entities are introduced.

free parameters (4)
  • lambda_d and lambda_g = 0.1
    Trade-off weights in Eq. (13), selected after sweeping {0.001, 0.01, 0.1, 1, 10} and picking the best on CIFAR/ImageNet.
  • q (blend indicator threshold) = 0.7
    Controls application of Lblend in Eq. (8); tuned over {0.1, 0.3, 0.5, 0.7, 0.9}.
  • gamma (EMA smoothing weight) = 0.5
    Smoothing weight in Eq. (10), tuned over {0.1, 0.3, 0.5, 0.7, 0.9}.
  • inflation factor N = floor(|Ds|/|Dc|), best at N=10
    Chosen empirically from Fig. 2(c); moderate inflation improves accuracy, while excessive inflation reduces diversity.
assumptions (5)
  • domain assumption Teacher network is well-trained and its softmax over synthetic examples approximates true class proportions p_T
    Used in Eq. (12) to balance generator training; not verified on synthetic out-of-distribution data.
  • domain assumption Teacher features are meaningful supervision for discriminator regularization and student alignment
    Eqs. (8), (9), and (14) assume teacher features on synthetic and collected data are reliable guides.
  • standard math Collected and synthetic data share the same support and TVD triangle inequality applies
    Used in Eqs. (2)-(5), though the algebra in Eqs. (4)-(5) is inconsistent with direct computation.
  • ad hoc to paper Inflation factor N=floor(|Ds|/|Dc|) is a moderate choice
    Justified empirically by Fig. 2(c), not derived from the TVD bound.
  • ad hoc to paper Lblend is applied with probability q=0.7
    The stated p sampled uniformly from [0,1] and q=0.7 give an application probability of 0.3, contradicting the claim of probability q.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hybrid Data-Free Knowledge Distillation." pith.science (2026). https://pith.science/paper/PGOMQVTL

@misc{pith2026241213525,
  author       = {Pith},
  title        = {Pith review of: Hybrid Data-Free Knowledge Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PGOMQVTL}},
  note         = {Machine review of arXiv:2412.13525}
}
read the original abstract

Data-free knowledge distillation aims to learn a compact student network from a pre-trained large teacher network without using the original training data of the teacher network. Existing collection-based and generation-based methods train student networks by collecting massive real examples and generating synthetic examples, respectively. However, they inevitably become weak in practical scenarios due to the difficulties in gathering or emulating sufficient real-world data. To solve this problem, we propose a novel method called \textbf{H}ybr\textbf{i}d \textbf{D}ata-\textbf{F}ree \textbf{D}istillation (HiDFD), which leverages only a small amount of collected data as well as generates sufficient examples for training student networks. Our HiDFD comprises two primary modules, \textit{i.e.}, the teacher-guided generation and student distillation. The teacher-guided generation module guides a Generative Adversarial Network (GAN) by the teacher network to produce high-quality synthetic examples from very few real-world collected examples. Specifically, we design a feature integration mechanism to prevent the GAN from overfitting and facilitate the reliable representation learning from the teacher network. Meanwhile, we drive a category frequency smoothing technique via the teacher network to balance the generative training of each category. In the student distillation module, we explore a data inflation strategy to properly utilize a blend of real and synthetic data to train the student network via a classifier-sharing-based feature alignment technique. Intensive experiments across multiple benchmarks demonstrate that our HiDFD can achieve state-of-the-art performance using 120 times less collected data than existing methods. Code is available at https://github.com/tangjialiang97/HiDFD.

Figures

Figures reproduced from arXiv: 2412.13525 by the authors.

Figure 1
Figure 1. The diagram of (a) generation-based methods (Fang et al. 2021; Yin et al. 2020; Chen et al. 2019; Micaelli and Storkey [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Parametric sensitivities of (a) λd and (b) λg in Eq. (13). Accuracies (in %) of the student networks trained with collected data with (c) varying inflation factors and (d) various quantities. ods generally achieve suboptimal performance due to their inability to effectively handle the potentially noisy examples among the hybrid data. These results highlight the suitability of our training strategy for reliable stude… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 39 canonical work pages

  1. [1]

    K.; Chakraborty, A.; and Radhakrishnan, V

    Addepalli, S.; Nayak, G. K.; Chakraborty, A.; and Radhakrishnan, V. B. 2020. Degan: Data-enriching gan for retrieving representative samples from a trained classifier. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 3130--3137

    work page 2020

  2. [2]

    Arjovsky, M.; and Bottou, L. 2022. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations (ICLR)

    work page 2022

  3. [3]

    T.; Leman, K.; and Mitra, T

    Binici, K.; Aggarwal, S.; Pham, N. T.; Leman, K.; and Mitra, T. 2022. Robust and resource-efficient data-free knowledge distillation by generative pseudo replay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 6089--6096

    work page 2022

  4. [4]

    Chen, D.; Mei, J.-P.; Zhang, H.; Wang, C.; Feng, Y.; and Chen, C. 2022. Knowledge distillation with the reused teacher classifier. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11933--11942

    work page 2022

  5. [5]

    Chen, D.; Mei, J.-P.; Zhang, Y.; Wang, C.; Wang, Z.; Feng, Y.; and Chen, C. 2021 a . Cross-layer distillation with semantic calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 7028--7036

    work page 2021

  6. [6]

    Chen, H.; Guo, T.; Xu, C.; Li, W.; Xu, C.; Xu, C.; and Wang, Y. 2021 b . Learning student networks in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 6428--6437

    work page 2021

  7. [7]

    Chen, H.; Wang, Y.; Xu, C.; Yang, Z.; Liu, C.; Shi, B.; Xu, C.; Xu, C.; and Tian, Q. 2019. Data-free learning of student networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 3514--3522

    work page 2019

  8. [8]

    C.; Gutman, D.; Celebi, M

    Codella, N. C.; Gutman, D.; Celebi, M. E.; Helba, B.; Marchetti, M. A.; Dusza, S. W.; Kalloo, A.; Liopyris, K.; Mishra, N.; Kittler, H.; et al. 2018. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging, hosted by the international skin imaging collaboration. In 15th IEEE International Sympo...

    work page 2018

Show all 51 references
  1. [9]

    Creswell, A.; White, T.; Dumoulin, V.; Arulkumaran, K.; Sengupta, B.; and Bharath, A. A. 2018. Generative adversarial networks: An overview. IEEE Signal Processing Magazine (SPM), 35(1): 53--65

    2018

  2. [10]

    Cui, K.; Yu, Y.; Zhan, F.; Liao, S.; Lu, S.; and Xing, E. P. 2023. Kd-dlgan: Data limited image generation via knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3872--3882

    2023

  3. [11]

    N.; Crowley, E

    Darlow, L. N.; Crowley, E. J.; Antoniou, A.; and Storkey, A. J. 2018. Cinic-10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.03505

    2018 arXiv

  4. [12]

    Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 248--255

    2009

  5. [13]

    Fang, G.; Song, J.; Wang, X.; Shen, C.; Wang, X.; and Song, M. 2021. Contrastive model inversion for data-free knowledge distillation. arXiv preprint arXiv:2105.08584

    2021 arXiv

  6. [14]

    Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2020. Generative adversarial networks. Communications of the ACM, 63(11): 139--144

    2020

  7. [15]

    Guo, Z.; Yan, H.; Li, H.; and Lin, X. 2023. Class attention transfer based knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11868--11877

    2023

  8. [16]

    Hao, Z.; Guo, J.; Han, K.; Tang, Y.; Hu, H.; Wang, Y.; and Xu, C. 2023. One-for-all: bridge the gap btween heterogeneous architectures in knowledge distillation. Advances in Neural Information Processing Systems (NeurIPS), 36: 79570--79582

    2023

  9. [17]

    Hao, Z.; Guo, J.; Wang, C.; Tang, Y.; Wu, H.; Hu, H.; Han, K.; and Xu, C. 2024. Data-efficient large vision models through sequential autoregression. In Forty-first International Conference on Machine Learning (ICML)

    2024

  10. [18]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 770--778

    2016

  11. [19]

    Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    2015 arXiv

  12. [20]

    Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 33: 6840--6851

    2020

  13. [21]

    Hou, L.; Cao, Q.; Shen, H.; Pan, S.; Li, X.; and Cheng, X. 2022. Conditional gans with auxiliary discriminative classifier. In International Conference on Machine Learning (ICML), 8888--8902. PMLR

    2022

  14. [22]

    Hou, L.; Yuan, Z.; Huang, L.; Shen, H.; Cheng, X.; and Wang, C. 2021. Slimmable generative adversarial networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 7746--7753

    2021

  15. [23]

    Huang, J.; Cui, K.; Guan, D.; Xiao, A.; Zhan, F.; Lu, S.; Liao, S.; and Xing, E. 2022. Masked generative adversarial networks are data-efficient generation learners. Advances in Neural Information Processing Systems (NeurIPS), 35: 2154--2167

    2022

  16. [24]

    Jiang, L.; Dai, B.; Wu, W.; and Loy, C. C. 2021. Deceive d: Adaptive pseudo augmentation for gan training with limited data. Advances in Neural Information Processing Systems (NeurIPS), 34: 21655--21667

    2021

  17. [25]

    P.; and Welling, M

    Kingma, D. P.; and Welling, M. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114

    2013 arXiv

  18. [26]

    Krizhevsky, A. 2009. Learning multiple layers of features from tiny images. Master's Thesis, University of Tront

    2009

  19. [27]

    Le, Y.; and Yang, X. 2015. Tiny imagenet visual recognition challenge. CS 231N, 7(7): 3

    2015

  20. [28]

    Li, W.; Wang, L.; Li, W.; Agustsson, E.; and Van Gool, L. 2017. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862

    2017 arXiv

  21. [29]

    Li, Z.; Li, X.; Yang, L.; Zhao, B.; Song, R.; Luo, L.; Li, J.; and Yang, J. 2023. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 1504--1512

    2023

  22. [30]

    Mei, K.; and Patel, V. 2023. Vidm: Video implicit diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 9117--9125

    2023

  23. [31]

    Micaelli, P.; and Storkey, A. J. 2019. Zero-shot knowledge transfer via adversarial belief matching. Advances in Neural Information Processing Systems (NeurIPS), 32

    2019

  24. [32]

    Miles, R.; and Mikolajczyk, K. 2024. Understanding the role of the projector in knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 4233--4241

    2024

  25. [33]

    Mirza, M.; and Osindero, S. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784

    2014 arXiv

  26. [34]

    Park, W.; Kim, D.; Lu, Y.; and Cho, M. 2019. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3967--3976

    2019

  27. [35]

    Peng, B.; Jin, X.; Liu, J.; Li, D.; Wu, Y.; Liu, Y.; Zhou, S.; and Zhang, Z. 2019. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 5007--5016

    2019

  28. [36]

    R.; and Babu, R

    Rangwani, H.; Mopuri, K. R.; and Babu, R. V. 2021. Class balancing gan with a classifier in the loop. In Uncertainty in Artificial Intelligence (UAI), 1618--1627. PMLR

    2021

  29. [37]

    Steerneman, T. 1983. On the total variation and Hellinger distance between signed measures; an application to product measures. Proceedings of the American Mathematical Society (AMS), 88(4): 684--688

    1983

  30. [38]

    Tang, J.; Chen, S.; Niu, G.; Sugiyama, M.; and Gong, C. 2023. Distribution shift matters for knowledge distillation with webly collected images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    2023

  31. [39]

    H.; and Phung, D

    Tran, M.-T.; Le, T.; Le, X.-M.; Harandi, M.; Tran, Q. H.; and Phung, D. 2024. Nayer: Noisy layer data generation for efficient and effective data-free knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 23860--23869

    2024

  32. [40]

    Tschandl, P.; Rosendahl, C.; and Kittler, H. 2018. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5(1): 1--9

    2018

  33. [41]

    Wang, Y.; Qian, B.; Liu, H.; Rui, Y.; and Wang, M. 2024 a . Unpacking the gap box against data-free knowledge distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

    2024

  34. [42]

    Wang, Y.; Yang, D.; Chen, Z.; Liu, Y.; Liu, S.; Zhang, W.; Zhang, L.; and Qi, L. 2024 b . De-confounded data-free knowledge distillation for handling distribution shifts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 12615--12625

    2024

  35. [43]

    Wang, Y.; Zhang, J.; and Wang, Y. 2024. Do generated data always help contrastive learning? In International Conference on Learning Representations (ICLR)

    2024

  36. [44]

    Yang, Z.; Zeng, A.; Li, Z.; Zhang, T.; Yuan, C.; and Li, Y. 2023. From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels. arXiv preprint arXiv:2303.13005

    2023 arXiv

  37. [45]

    M.; Li, Z.; Mallya, A.; Hoiem, D.; Jha, N

    Yin, H.; Molchanov, P.; Alvarez, J. M.; Li, Z.; Mallya, A.; Hoiem, D.; Jha, N. K.; and Kautz, J. 2020. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8715--8724

    2020

  38. [46]

    Yu, S.; Chen, J.; Han, H.; and Jiang, S. 2023. Data-free knowledge distillation via feature exchange and activation region constraint. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 24266--24275

    2023

  39. [47]

    Zhao, B.; Cui, Q.; Song, R.; Qiu, Y.; and Liang, J. 2022 a . Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11953--11962

    2022

  40. [48]

    Zhao, H.; Sun, X.; Dong, J.; Manic, M.; Zhou, H.; and Yu, H. 2022 b . Dual discriminator adversarial distillation for data-free model compression. International Journal of Machine Learning and Cybernetics (IJMLC), 13(5): 1213--1230

    2022

  41. [49]

    Zhao, S.; Song, J.; and Ermon, S. 2019. Infovae: Balancing learning and inference in variational autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 5885--5892

    2019

  42. [50]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  43. [51]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.