Pith. sign in

REVIEW 3 major objections 3 minor 1 cited by

Large-Scale Data-Free Knowledge Distillation for ImageNet via Multi-Resolution Data Generation

T0 review · 3 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that MUSE, a data-free distillation method, achieves two-digit accuracy improvements over prior baselines on ImageNet and subsets by generating low-resolution, CAM-guided synthetic images.

desk verdict Good core idea, but the ImageNet gains are measured under a data-ratio rule that gives MUSE up to 4x more samples than the baselines, so the two-digit SOTA claims rest on an unconfounded comparison that the paper never supplies. read the letter →

arxiv 2411.17046 v1 pith:4I7JJBHH submitted 2024-11-26 cs.CV

Minh-Tuan Tran , Trung Le , Xuan-May Le , Jianfei Cai , Mehrtash Harandi , Dinh Phung This is my paper · ORCID
classification cs.CV
keywords data-freeknowledgedistillationImageNetmulti-resolutiondatagenerationclassactivationmapsembeddingdiversitysyntheticCIFAR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MUSE, a data-free knowledge distillation method that trains a student network on synthetic images generated by a teacher, with no access to the original training data. Its central claim is that generating synthetic images at lower resolutions, guided by class activation maps, preserves the class-specific features that high-resolution generation loses in noise, and that this makes ImageNet-scale data-free distillation both faster and far more accurate. Against the Fast100 and NAYER baselines, MUSE reports accuracy gains of up to two digits across ImageNet and its subsets at the same compute-equivalent data ratios, and also improves results on CIFAR-10 and CIFAR-100. The paper argues that the bottleneck for large-scale data-free distillation is not image resolution itself but whether the generated pixels carry discriminative information.

What carries the argument

The load-bearing object is the Class Activation Map (CAM) margin loss $L_{\mathrm{cam}} = \max\{0, M_{\mathrm{target}} - M(\hat{x}, \hat{y})\}$, where $M(\hat{x}, \hat{y})$ is the teacher's unnormalized CAM for a synthetic image and $M_{\mathrm{target}}$ is a center-high mask; maximizing this term biases the generator to concentrate class-discriminative pixels near the image center instead of scattering noise across the frame. Around that core, MUSE couples a lower-resolution generator $G_{e \times e}$ with a multi-resolution set $E$, and an embedding in-out game: the student pulls embeddings of pooled synthetic data within an inner radius $r_i$ of the class text embedding $f_y$, while the generator pushes new batches beyond an outer radius $r_o$. This adversarial loop is what the paper credits for both image quality and latent diversity.

What would settle it

Compare MUSE-S against NAYER at identical sample counts rather than identical compute budgets, for example by giving both methods the same number of generated training images per class at the same data-ratio label on ImageNet1k. If the accuracy gap shrinks from two digits to a few points, the difference is mostly sample count, not the CAM or embedding losses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a data-free generator can replace high-resolution 224x224 synthetic images with lower-resolution images (112x112 or 96x96) without losing the information a student needs, provided generation is steered by a CAM-enhanced quality loss that forces the class activation map of each synthetic image to dominate a center-weighted target mask. Two additional mechanisms, multi-resolution generation and embedding-diversity losses that keep old synthetic embeddings inside an inner radius and push new ones beyond an outer radius around the class text embedding, preserve feature diversity at low resolution. The paper reports that MUSE and its single-resolution variant MUSE-S outperform Fast100 and NAYER by large margins on Imagenette, Imagewoof, and ImageNet1k under equal training-time budgets, with gains often exceeding ten percentage points, while also setting the highest numbers in most CIFAR-10 and CIFAR-100 configurations.

Load-bearing premise

The results rest on counting a 224x224 image as equal in cost to four 112x112 images (or five 96x96), so the data ratio expresses compute budget rather than number of samples; if that equivalence is wrong or unfair, the reported gains may largely reflect that MUSE simply trains on more images.

Editorial extensions

If this is right

  • ImageNet-scale data-free distillation becomes practical: two-digit accuracy improvements over Fast100 and NAYER at 1-20% compute-equivalent data ratios, with similar or lower training time.
  • Lower-resolution synthetic data is not a limitation when paired with CAM guidance; the paper's resolution sweep peaks at 96x96, not 224x224.
  • Multi-resolution generation and embedding diversity transfer to small-scale benchmarks, giving MUSE the best CIFAR-10/CIFAR-100 numbers in most tested architecture pairs.
  • The approach extends to vision-transformer students through a center-biased patch-reduction scheme, improving top-1 accuracy over NAYER by more than two points.
  • Because one 224x224 image is treated as the same compute budget as four 112x112 images, MUSE trains on several times more samples than its baselines at the same reported data ratio.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The compute-equivalent data ratio is the main confound: if the gains were reproduced with equal sample counts rather than equal compute, the headline two-digit advantage would be harder to attribute to CAM and embedding diversity.
  • The same recipe, lower resolution plus localization priors plus more samples under a compute budget, could be dropped into other synthetic-data pipelines such as dataset condensation or federated distillation, where generation cost dominates.
  • Because the paper's ViT extension previews center-biased patch selection, a natural stress test is whether the method still wins when the teacher itself is a ViT and the student must match full-resolution inputs.
  • A direct experiment that would separate the method from the budget is running NAYER with four 112x112 images per 224x224 slot; if NAYER's accuracy rises to MUSE's level, the novelty is the sample-count accounting rather than the losses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes MUSE (MUlti-reSolution Data-freE), a data-free knowledge distillation method that generates synthetic images at lower resolutions (e.g., 112x112 or 96x96) than the teacher's native 224x224, uses a Class Activation Map-based loss to preserve class-specific features, and adds multi-resolution generation and embedding-diversity losses. Experiments are reported on CIFAR-10, CIFAR-100, ImageNet1k, and the ImageNet subsets Imagenette and Imagewoof, with two backbone pairs. The central claim is that MUSE achieves state-of-the-art DFKD performance, with two-digit accuracy gains over prior methods at matched data-ratio budgets.

Significance. If the headline comparison is valid, this would be a practically important result: MUSE reports large accuracy gains over NAYER and Fast while substantially reducing the cost of synthetic-data generation on ImageNet-scale problems. The paper includes useful strengths: an extensive experimental matrix (two large-scale benchmarks, two backbone pairs, multiple data ratios), ablations over resolution, CAM loss, and embedding-diversity components, parameter-sensitivity tables, a ViT extension, and a public code link. However, the comparison protocol in Section 4.2 introduces a sample-count confound that directly affects the headline two-digit gains, and there are internal inconsistencies in the small-scale results. Because the central quantitative claim is not yet established by a clean comparison, major revision is needed.

major comments (3)
  1. [Section 4.2, Table 1, Eqs. (2)/(8)] This is the load-bearing issue. Without an equal-sample-count control, the two-digit gains claimed in the abstract and conclusion are not supported as a clean comparison.
  2. [Section 4.3, Table 2]
  3. [Section 3.2, Section D, Table 13]
minor comments (3)
  1. [Throughout] There are repeated typos and inconsistent naming: "Imagenetee" and "ImageWoof" in Table 1, "Comparasion" in Section C, "Marging Loss" in the caption of Table 10, "indidate" in Table 13, and "SRe 2L" versus "SRe2L" versus "SReL" in Section C and Table 11.
  2. [Table 2] The row label "MUSE-Mul" is used in Table 2 but is not defined in the main text; the main text usually refers to single-resolution MUSE-S and multi-resolution MUSE. Please clarify the relationship between MUSE-Mul and MUSE.
  3. [Section 5] The subsection heading "Lower-resolution for Vision Transformer" is missing a hyphen and an article; more importantly, the statement that the patch-reduction strategy "outperforms the original NAYER training, showing improvements of over two percentage points" is inconsistent with Table 15, which shows improvements far larger than two percentage points (e.g., 15.24% vs. 4.52% at 1% top-1).

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; the two-digit ImageNet gains are confounded by the compute-equivalent data-ratio normalization, but no equation forces the results.

full rationale

The paper's derivation chain is not circular. The generator and student objectives (Eqs. 6, 8, 11, and 12) are proposed heuristic losses, and the reported accuracies are empirical results rather than quantities derived by construction from the loss definitions. The self-citations to NAYER [28] and to the authors' prior Bounding Loss [27] serve as baseline architecture and loss inspiration, respectively, but they are not invoked as proof of the central claim; the paper provides ablations (Table 3 and Figure 6) and comparisons with external methods. The main concern is Section 4.2, where a compute-equivalent data-ratio definition lets MUSE-S use 40,000 112x112 images where NAYER and Fast use 10,000 224x224 images at the same reported 10% ratio. This gives MUSE a fourfold larger training-sample budget at the same nominal ratio and could by itself explain part of the accuracy jump, so the headline 'two-digit gains' claim is confounded. However, this is a fairness and soundness issue, not a circular reduction: no equation in the paper defines the accuracy result in terms of the data-ratio normalization, and the accuracy is not a fitted restatement of the input. The self-citations are real prior work with external publication venues, and the central method is independently evaluated, so the circularity score is low.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central empirical claim depends on several manually chosen hyperparameters and on the assumption that the resource-equivalent data ratio is a fair comparison. No fundamentally new physical or mathematical entity is introduced.

free parameters (7)
  • alpha_cam = 0.1
    Scaling factor for CAM-enhanced quality loss; chosen by sensitivity analysis on ImageNette and ImageWoof (Table 7).
  • alpha_ed = 10
    Scaling factor for embedding diversity student loss; selected via Table 8.
  • alpha_aed = 5
    Scaling factor for generator-side embedding diversity loss; selected via Table 9.
  • ri = 0.015
    Inner radius for bounding loss; chosen based on minimum distance between label text embeddings (Table 10).
  • ro = 0.03
    Outer radius for margin loss; chosen as the minimum distance between label text embeddings.
  • Mtarget = Full(1) matrix
    Target mask for CAM loss; selected via ablation (Table 13).
  • resolution set = e.g., [96,112] for Imagenette, [112,128] for ImageNet1k, [28,32] for CIFAR
    Chosen per dataset to maximize accuracy in Table 3 and Table 6.
assumptions (5)
  • domain assumption Only a small, crucial region of a real image is essential for classifier training
    Motivates generating low-resolution images and using CAM; stated in Introduction and Section 3.2.
  • ad hoc to paper The teacher's class activation map computed on the synthetic image is a reliable guide for class-specific feature generation
    The CAM-enhanced loss (Eq. 6) assumes that maximizing CAM toward a target mask produces synthetic images with the discriminative features of the target class.
  • domain assumption A pre-trained language model's text embedding of the class label is a valid class representative for embedding diversity
    Used to define fy in Eq. 8-12; the paper also offers class center as an alternative.
  • domain assumption A student trained on low-resolution synthetic images will generalize to full-resolution real images at test time
    The student is trained and evaluated at different resolutions (e.g., trained at 112x112, tested at 224x224 for ImageNet).
  • standard math Standard KL divergence, MSE and cross-entropy losses are valid for knowledge transfer
    Used in the distillation and generator objectives.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large-Scale Data-Free Knowledge Distillation for ImageNet via Multi-Resolution Data Generation." pith.science (2026). https://pith.science/paper/4I7JJBHH

@misc{pith2026241117046,
  author       = {Pith},
  title        = {Pith review of: Large-Scale Data-Free Knowledge Distillation for ImageNet via Multi-Resolution Data Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4I7JJBHH}},
  note         = {Machine review of arXiv:2411.17046}
}
abstract

Data-Free Knowledge Distillation (DFKD) is an advanced technique that enables knowledge transfer from a teacher model to a student model without relying on original training data. While DFKD methods have achieved success on smaller datasets like CIFAR10 and CIFAR100, they encounter challenges on larger, high-resolution datasets such as ImageNet. A primary issue with previous approaches is their generation of synthetic images at high resolutions (e.g., $224 \times 224$) without leveraging information from real images, often resulting in noisy images that lack essential class-specific features in large datasets. Additionally, the computational cost of generating the extensive data needed for effective knowledge transfer can be prohibitive. In this paper, we introduce MUlti-reSolution data-freE (MUSE) to address these limitations. MUSE generates images at lower resolutions while using Class Activation Maps (CAMs) to ensure that the generated images retain critical, class-specific features. To further enhance model diversity, we propose multi-resolution generation and embedding diversity techniques that strengthen latent space representations, leading to significant performance improvements. Experimental results demonstrate that MUSE achieves state-of-the-art performance across both small- and large-scale datasets, with notable performance gains of up to two digits in nearly all ImageNet and subset experiments. Code is available at https://github.com/tmtuan1307/muse.

Figures

Figures reproduced from arXiv: 2411.17046 by the authors.

Figure 1
Figure 1. Accuracies of our MUSE method and current SOTA [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) The previous model fails to capture class-specific [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Accuracies and training time of using lower-resolution [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: (a) Overview of the MUSE architecture, illustrating the two-phase training process: generator training and student training. The [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The accuracy at data ratios from 10% to 100% is shown [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: (a) Accuracy of our MUSE method with and without [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs

    cs.CV 2024-12 conditional novelty 6.0 of 10

    LoRA Recycle distills a meta-LoRA from diverse pre-tuned LoRAs using data generated by inverting those LoRAs, enabling tuning-free few-shot classification at test time.

Reference graph

Works this paper leans on

42 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Robust and resource- efficient data-free knowledge distillation by generative pseudo replay

    Kuluhan Binici, Shivam Aggarwal, Nam Trung Pham, Karianto Leman, and Tulika Mitra. Robust and resource- efficient data-free knowledge distillation by generative pseudo replay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6089–6096, 2022. 1

    work page 2022

  2. [2]

    Data-free network quantization with adversarial knowl- edge distillation

    Yoojin Choi, Jihwan Choi, Mostafa El-Khamy, and Jungwon Lee. Data-free network quantization with adversarial knowl- edge distillation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition Work- shops, pages 710–711, 2020. 7

    work page 2020

  3. [3]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2009. 5

    work page 2009

  4. [4]

    Momentum adversarial distillation: Handling large distribution shifts in data-free knowledge distilla- tion

    Kien Do, Thai Hung Le, Dung Nguyen, Dang Nguyen, Haripriya Harikumar, Truyen Tran, Santu Rana, and Svetha Venkatesh. Momentum adversarial distillation: Handling large distribution shifts in data-free knowledge distilla- tion. Advances in Neural Information Processing Systems , 35:10055–10067, 2022. 1, 3

    work page 2022

  5. [5]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 4, 8

    arXiv 2010

  6. [6]

    Up to 100x faster data- free knowledge distillation

    Gongfan Fang, Kanya Mo, Xinchao Wang, Jie Song, Shitao Bei, Haofei Zhang, and Mingli Song. Up to 100x faster data- free knowledge distillation. InProceedings of the AAAI Con- ference on Artificial Intelligence , volume 36, pages 6597– 6604, 2022. 1, 2, 3, 5, 6, 7, 11

    work page 2022

  7. [7]

    Contrastive model inver- sion for data-free knowledge distillation

    Gongfan Fang, Jie Song, Xinchao Wang, Chengchao Shen, Xingen Wang, and Mingli Song. Contrastive model inver- sion for data-free knowledge distillation. arXiv preprint arXiv:2105.08584, 2021. 1, 3

    arXiv 2021

  8. [8]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 5, 6, 7

    work page 2016

Show all 42 references
  1. [9]

    Distill- ing the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- ing the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 1

    2015 arXiv

  2. [10]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6

    2009

  3. [11]

    Shapeformer: Shapelet transformer for multivariate time series classification

    Xuan-May Le, Ling Luo, Uwe Aickelin, and Minh-Tuan Tran. Shapeformer: Shapelet transformer for multivariate time series classification. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1484–1494, 2024. 4

    2024

  4. [12]

    Learning perceptual position-aware shapelets for time series classification

    Xuan-May Le, Minh-Tuan Tran, and Van-Nam Huynh. Learning perceptual position-aware shapelets for time series classification. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 53–

  5. [13]

    Tiny imagenet visual recognition challenge

    Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015. 5

    2015

  6. [14]

    Attention guided cam: Visual explanations of vision transformer guided by self- attention

    Saebom Leem and Hyunseok Seo. Attention guided cam: Visual explanations of vision transformer guided by self- attention. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 38, pages 2956–2964, 2024. 8

    2024

  7. [15]

    Memory efficient data-free distillation for continual learning

    Xiaorong Li, Shipeng Wang, Jian Sun, and Zongben Xu. Memory efficient data-free distillation for continual learning. Pattern Recognition, 144:109875, 2023. 6

    2023

  8. [16]

    Variational data-free knowledge distillation for continual learning

    Xiaorong Li, Shipeng Wang, Jian Sun, and Zongben Xu. Variational data-free knowledge distillation for continual learning. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 45(10):12618–12634, 2023. 6

    2023

  9. [17]

    Small scale data-free knowledge distillation

    He Liu, Yikai Wang, Huaping Liu, Fuchun Sun, and Anbang Yao. Small scale data-free knowledge distillation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6008–6016, 2024. 6, 7

    2024

  10. [18]

    Zero-shot knowledge transfer via adversarial belief matching

    Paul Micaelli and Amos J Storkey. Zero-shot knowledge transfer via adversarial belief matching. Advances in Neu- ral Information Processing Systems, 32, 2019. 1, 3, 7

    2019

  11. [19]

    Zero-shot knowledge distillation in deep networks

    Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. Zero-shot knowledge distillation in deep networks. In In- ternational Conference on Machine Learning , pages 4743–

  12. [20]

    Learn- ing to retain while acquiring: Combating distribution-shift in adversarial data-free knowledge distillation

    Gaurav Patel, Konda Reddy Mopuri, and Qiang Qiu. Learn- ing to retain while acquiring: Combating distribution-shift in adversarial data-free knowledge distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7786–7794, 2023. 1, 3

    2023

  13. [21]

    Better teacher better stu- dent: Dynamic prior knowledge for knowledge distillation

    Zengyu Qiu, Xinzhu Ma, Kunlin Yang, Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Better teacher better stu- dent: Dynamic prior knowledge for knowledge distillation. arXiv preprint arXiv:2206.06067, 2022. 1

    2022 arXiv

  14. [22]

    Mobilenetv2: Inverted residuals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4510–4520, 2018. 5

    2018

  15. [23]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,

    1910 arXiv

  16. [24]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE in- ternational conference on computer vision , pages 618–626,

  17. [25]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 6, 7

    2014 arXiv

  18. [26]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. In International conference on machine learning , pages 10347–10357. PMLR, 2021. 8

    2021

  19. [27]

    Text-enhanced data-free approach for federated class-incremental learning

    Minh-Tuan Tran, Trung Le, Xuan-May Le, Mehrtash Ha- randi, and Dinh Phung. Text-enhanced data-free approach for federated class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23870–23880, 2024. 1, 4, 6, 11, 12

    2024

  20. [28]

    Nayer: Noisy layer data generation for efficient and effective data-free knowl- edge distillation

    Minh-Tuan Tran, Trung Le, Xuan-May Le, Mehrtash Ha- randi, Quan Hung Tran, and Dinh Phung. Nayer: Noisy layer data generation for efficient and effective data-free knowl- edge distillation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition ...

    2024

  21. [29]

    Pisd: A linear complexity distance beats dynamic time warping on time series classification and clus- tering

    Minh-Tuan Tran, Xuan-May Le, Van-Nam Huynh, and Sung-Eui Yoon. Pisd: A linear complexity distance beats dynamic time warping on time series classification and clus- tering. Engineering Applications of Artificial Intelligence , 138:109222, 2024. 4

    2024

  22. [30]

    Focal and global knowledge distillation for detectors

    Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Ze- huan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4643–4652, 2022. 1

    2022

  23. [31]

    Dreaming to distill: Data-free knowledge transfer via deep- inversion

    Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deep- inversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8...

    2020

  24. [32]

    Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective

    Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. Advances in Neural Information Process- ing Systems, 36, 2024. 12, 13

    2024

  25. [33]

    Tutornet: Towards flexible knowledge distillation for end-to-end speech recognition

    Ji Won Yoon, Hyeonseung Lee, Hyung Yong Kim, Won Ik Cho, and Nam Soo Kim. Tutornet: Towards flexible knowledge distillation for end-to-end speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1626–1638, 2021. 1

    2021

  26. [34]

    Data-free knowledge distillation via feature exchange and activation region constraint

    Shikang Yu, Jiachen Chen, Hu Han, and Shuqiang Jiang. Data-free knowledge distillation via feature exchange and activation region constraint. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 24266–24275, 2023. 1, 3

    2023

  27. [35]

    Wide residual net- works

    Sergey Zagoruyko and Nikos Komodakis. Wide residual net- works. arXiv preprint arXiv:1605.07146, 2016. 6, 7

    2016 arXiv

  28. [36]

    Visualizing and understanding convolutional net- works

    MD Zeiler. Visualizing and understanding convolutional net- works. In European conference on computer vision/arXiv , volume 1311, 2014. 2

    2014

  29. [37]

    Identifiable con- trastive learning with automatic feature importance discov- ery

    Qi Zhang, Yifei Wang, and Yisen Wang. Identifiable con- trastive learning with automatic feature importance discov- ery. Advances in Neural Information Processing Systems , 36, 2024. 4

    2024

  30. [38]

    Decoupled knowledge distillation

    Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953–11962, 2022. 1

    2022

  31. [39]

    Learning deep features for discrimina- tive localization

    Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discrimina- tive localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921–2929,

  32. [40]

    Deformable detr: Deformable trans- formers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable trans- formers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020. 2

    2010 arXiv

  33. [41]

    Data-free knowledge distillation for heterogeneous federated learn- ing

    Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learn- ing. In International conference on machine learning, pages 12878–12889. PMLR, 2021. 6 A. Training Details In this section, we provide the details of model train- in...

    2021

  34. [42]

    To further optimize training, a lambda scheduler is em- ployed to adjust the learning rate dynamically throughout the training process. A.4. Other Settings We trained the model for E epochs, incorporating a warm-up phase during the first 10% of E, as outlined in the settings d...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.