REVIEW 5 major objections 4 minor 39 references
Generative Dataset Distillation Based on Self-knowledge Distillation
T0 review · 5 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A generative dataset distillation method that standardizes prediction logits and matches their probability distributions through self-knowledge distillation reports higher accuracy than existing distillation baselines on MNIST…
desk verdict A modest DiM variant with real but small gains; the claimed attribution to the KL loss needs a controlled ablation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the self-knowledge distillation loss $L_{\text{SKD}}$, which matches the class-probability distributions of original and synthetic logits through a KL-style divergence instead of minimizing the mean squared error between raw logits. The standardization step $Z(x;\tau)$ removes scale and shift differences between the two logit vectors before the softmax, which the paper argues prevents the matching from being misled by similar ranges of inaccurate predictions. At each matching step, a model is randomly selected from a pool of ConvNet3, ResNet10, and ResNet18, so the generator is not optimized against a single fixed architecture. The generator itself is trained with the conditional GAN loss $L_{\text{CGAN}}$, and the distillation loss is added with a weighted temperature-scaled term to form the total objective.
What would settle it
Run the method with a model pool containing only ConvNet3 and evaluate the distilled generator on VGG11; if cross-architecture accuracy falls to roughly the DiM level, the random-pool assumption, rather than standardization, is responsible for the generalization gain. Equivalently, sweep the temperature $\tau$ over a wide range; if the standardization benefit disappears at some values of $\tau$, the claim that standardization itself is the cause of the improvement would need to be revised.
Extended reading notes
Core claim
The central discovery is that replacing plain logits matching with a self-knowledge-distillation loss, applied to standardized logits, improves the accuracy of the synthetic data produced by the generator. Formally, the paper computes $Z(x;\tau) = \frac{x - \text{mean}(x)}{\text{std}(x)} \times \tau$, feeds the standardized logits through softmax to obtain $d(x_O)$ and $d(x_S)$, and then uses $L_{\text{SKD}} = \sum_{k=1}^{K} d(x_O)^{(k)} \log \frac{d(x_O)^{(k)}}{d(x_S)^{(k)}}$. The total training loss is $L_{\text{total}} = L_{\text{CGAN}} + \lambda_{\text{SKD}} \tau^2 L_{\text{SKD}}$, where $L_{\text{CGAN}}$ is the conditional GAN loss. The paper's empirical claim is that this design outperforms DiM and other state-of-the-art condensation methods on MNIST, Fashion-MNIST, and CIFAR-10 at images-per-class values of 1, 10, and 50, and that the standardization step alone produces consistent accuracy gains.
Load-bearing premise
The load-bearing premise is that a matching model randomly drawn from a pool of just three architectures, ConvNet3, ResNet10, and ResNet18, provides a distribution-matching signal that transfers to other architectures; the paper does not show why three models suffice or how sensitive the results are to the pool's composition.
Editorial extensions
If this is right
- Standardizing logits before distribution matching produces consistent accuracy gains over the same method without standardization, across all three datasets and all images-per-class settings tested.
- Randomly drawing the matching model from a small pool of architectures gives synthetic data that transfers to unseen architectures such as AlexNet and VGG11, with the standardized version reporting the best cross-architecture results.
- Distribution matching of standardized probability vectors captures global structure and inter-class relationships better than direct logits matching, which is why the generator retains more representative information from the original dataset.
- The proposed method remains accurate even at very low compression budgets such as one image per class, and it improves further as the budget grows to 10 and 50 images per class.
Reading between the lines
- The paper does not isolate how much of the gain comes from the random model pool versus the standardization step; a natural next experiment is to hold the pool fixed while toggling standardization, and vice versa, on a held-out architecture.
- Because standardization removes logits scale and shift, the same normalization could likely be inserted into other logits-comparing distillation losses, such as trajectory matching or gradient matching, with similar stabilizing effects.
- The fixed temperature $\tau = 2$ and per-dataset weight $\lambda_{\text{SKD}}$ suggest a tuning interaction worth probing; per-dataset temperature schedules might widen the reported margins beyond the fixed settings used here.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a generative dataset distillation method. A conditional GAN is trained to produce synthetic images, and the generator is further optimized by matching the softmax distributions of standardized logits from a randomly selected model in a pool, using a KL-divergence-based 'self-knowledge distillation' loss. Experiments on MNIST, FashionMNIST, and CIFAR-10 report accuracy improvements over DiM and other baselines, plus cross-architecture results on CIFAR-10.
Significance. If the reported gains are robust, the combination of logits standardization and KL-based distribution matching would be a simple, useful modification to existing generative dataset distillation (e.g., DiM). The paper ships a clear loss formulation and reproducible-looking experimental protocol. However, the empirical evidence currently does not isolate the contribution of the proposed self-knowledge distillation loss, because the main comparison with DiM differs in both the matching objective and the underlying generator/pool setup. The paper is nonetheless a plausible incremental contribution that could become acceptable after a controlled ablation study and additional baselines.
major comments (5)
- [Section 2.2, Table 1] The comparison with DiM is confounded. The proposed method changes the matching objective (KL divergence on standardized softmax outputs instead of DiM's MSE on logits) and also uses a different generator, discriminator, and model-pool training protocol. The 'Ours (No Stand.)' ablation removes only the standardization step; there is no condition that replaces LSKD with DiM-style logits MSE while keeping the generator, discriminator, pool, and schedule identical. As a result, the accuracy advantage over DiM in Tables 1 and 2 cannot be attributed to the self-knowledge distillation loss, which is the central novelty claimed in the abstract and introduction.
- [Section 3.2] The model pool used for random model selection is underspecified. The paper states that a model is 'randomly selected from a pool of models' consisting of ConvNet3, ResNet10, and ResNet18, but it does not state how these networks are trained (on which dataset, with what optimizer, for how many epochs, with which augmentation). Without this information the cross-architecture generalization claimed in Section 3.3 is not reproducible, and the reader cannot assess whether the pool composition or the loss is responsible for the cross-architecture performance.
- [Table 1] The abstract claims the method 'outperforms existing state-of-the-art methods', but the reported results do not support this uniformly. On MNIST IPC=50, Ours (99.1±0.1) is below DiM (99.2±0.2), and on FashionMNIST IPC=50 Ours ties DiM at 89.8. The claim should be qualified to reflect the settings where the improvement is consistent, or the discrepancy should be discussed.
- [Section 3.1 (missing reference [17])] The authors' own related generative dataset distillation method, reference [17] (CVPR Workshop 2024), is cited in the bibliography but never compared in the experiments. Since [17] addresses the same task and is likely closely related, a direct empirical comparison is necessary to establish the novelty and incremental benefit of the proposed self-knowledge distillation and standardization components.
- [Equations (1) and (7)] The use of LCGAN in the total loss is ambiguous. Equation (1) defines a min-max GAN objective, but Equation (7) treats LCGAN as an additive term in Ltotal, which is then said to be minimized with respect to the generator parameters W. The paper should specify whether the discriminator is fixed when W is updated, how the adversarial term is computed (e.g., using the standard generator loss rather than the full min-max objective), and how the gradient flows through the GAN loss during the distribution-matching stage. This is essential for reproducibility.
minor comments (4)
- [Tables 1 and 2] The word 'comparation' in the table captions should be 'comparison'.
- [Section 3.2] The sentence 'IPC values were set to 1, 10, and 50, while batch sizes were 32, 64, and 128' should specify which batch size corresponds to which IPC value, since the pairing is needed to interpret the experiments.
- [Equation (3)] The standardization in Equation (3) computes mean and std over the logits vector for each input sample; this should be stated explicitly, and the paper should discuss how zero standard deviation (e.g., for very confident or degenerate outputs) is handled.
- [Remark on terminology] The term 'self-knowledge distillation' is used for a KL divergence between the softmax outputs of the same model on original and synthetic data; this is essentially distribution matching (as in DM) applied to logits. The paper should relate the term to prior usage of self-knowledge distillation and clarify the difference from simple logits-matching in DiM.
Circularity Check
No significant circularity; reported gains are measured on held-out test data and the proposed loss is not derived from the reported accuracies.
full rationale
The central claim is an empirical accuracy comparison, not a quantity defined by the proposed loss. LSKD (Eq. 6) is a training objective that matches softmax distributions of standardized logits between original and synthetic data; the reported accuracies in Tables 1 and 2 are obtained by training fresh models on generated synthetic data and evaluating on held-out test sets, so the result is not forced by construction. The self-citations [15, 17, 18] appear only in related-work context and are not invoked as evidence for the proposed loss or the empirical gains, so they are not load-bearing. The standardization ablation ('Ours (No Stand.)' vs. 'Ours') is a within-method comparison, and the hyperparameters lambda_SKD and tau are dataset-level training choices rather than fitted predictors of test accuracy. The skeptical concern about confounded baselines (different GAN architecture, unspecified model-pool training) is an experimental-rigor and validity threat, not a circular derivation. No passage in the manuscript asserts a limitation or missing proof that would alter this assessment; the underspecification of GAN and model-pool training is an experimental reporting gap, not a circular step.
Assumptions & free parameters
free parameters (2)
- lambda_SKD =
0.001 (MNIST), 0.01 (FashionMNIST, CIFAR-10)
- tau =
2
assumptions (4)
- domain assumption Conditional GAN training with loss in Eq. (1) yields a generator whose synthetic distribution is a useful starting point for distillation.
- domain assumption KL divergence between softmax outputs (Eq. 6) is an appropriate objective for aligning original and synthetic data distributions.
- domain assumption A pool of three architectures (ConvNet3, ResNet10, ResNet18) is representative enough for cross-architecture generalization.
- domain assumption Per-sample z-score standardization of logits (Eq. 3) removes scale and shift variability without losing class-discriminative information.
Cite this review
Pith. "Pith review of Generative Dataset Distillation Based on Self-knowledge Distillation." pith.science (2026). https://pith.science/paper/HSINMGLM
@misc{pith2026250104202,
author = {Pith},
title = {Pith review of: Generative Dataset Distillation Based on Self-knowledge Distillation},
year = {2026},
howpublished = {\url{https://pith.science/paper/HSINMGLM}},
note = {Machine review of arXiv:2501.04202}
}
read the original abstract
Dataset distillation is an effective technique for reducing the cost and complexity of model training while maintaining performance by compressing large datasets into smaller, more efficient versions. In this paper, we present a novel generative dataset distillation method that can improve the accuracy of aligning prediction logits. Our approach integrates self-knowledge distillation to achieve more precise distribution matching between the synthetic and original data, thereby capturing the overall structure and relationships within the data. To further improve the accuracy of alignment, we introduce a standardization step on the logits before performing distribution matching, ensuring consistency in the range of logits. Through extensive experiments, we demonstrate that our method outperforms existing state-of-the-art methods, resulting in superior distillation performance.
Reference graph
Works this paper leans on
-
[17]
Dataset condensation with gradi- ent matching,
Bo Zhao and Hakan Bilen, “Dataset condensation with gradi- ent matching,” in Proc. ICLR, 2021, pp. 1–20
work page 2021
-
[1]
Generative Dataset Distillation Based on Self-knowledge Distillation
INTRODUCTION The rapid advancement of deep learning has driven the creation of increasingly large models that require vast amounts of data to achieve optimal performance [1]. However, this growth in data vol- ume brings with it several significant challenges. First, storing and maintaining large datasets incurs high costs, both in terms of storage and the...
work page Pith review arXiv 2025
-
[2]
First, we train a generative adversarial network (GAN) to generate a synthetic dataset S
GENERA TIVE DA TASET DISTILLA TION BASED ON SELF-KNOWLEDGE DISTILLA TION The proposed method consists of two main steps. First, we train a generative adversarial network (GAN) to generate a synthetic dataset S. Next, a model is randomly selected from a model pool to align the synthetic dataset S with the original dataset O. To improve this alignment, we i...
-
[3]
EXPERIMENTS 3.1. Datasets and Comparative Methods We conducted extensive experiments to verify the effectiveness of the proposed method. First, we designed benchmark experi- ments using three datasets: MNIST [29], FashionMNIST [30], and CIFAR-10 [31]. Each dataset was categorized into 10 classes. To validate the performance of our approach, we compared it...
-
[4]
CONCLUSION In this paper, we have proposed a novel generative dataset distillation method that incorporates self-knowledge distillation to improve the overall distillation process. A key innovation of our approach is the redesign of the logits matching process, where we employ distribu- tion matching to better align the prediction logits between the origi...
-
[5]
A survey of deep learning and its applica- tions: a new paradigm to machine learning,
Shaveta Dargan, Munish Kumar, Maruthi Rohit Ayyagari, and Gulshan Kumar, “A survey of deep learning and its applica- tions: a new paradigm to machine learning,” Archives of Com- putational Methods in Engineering , vol. 27, pp. 1071–1092, 2020
work page 2020
-
[6]
Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions,
Laith Alzubaidi, Jinglan Zhang, and et al., “Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions,” Journal of Big Data, vol. 8, pp. 1–74, 2021
work page 2021
-
[7]
Applica- tions for deep learning in ecology,
Sylvain Christin, ´Eric Hervet, and Nicolas Lecomte, “Applica- tions for deep learning in ecology,” Methods in Ecology and Evolution, vol. 10, no. 10, pp. 1632–1644, 2019
work page 2019
Show all 39 references
-
[8]
Big healthcare data: preserving security and pri- vacy,
Karim Abouelmehdi, Abderrahim Beni-Hessane, and Hayat Khaloufi, “Big healthcare data: preserving security and pri- vacy,” Journal of Big Data, vol. 5, no. 1, pp. 1–18, 2018
2018
-
[9]
Dataset distillation,
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros, “Dataset distillation,” arXiv preprint arXiv:1811.10959, 2018
2018 arXiv
-
[10]
Awesome dataset distillation,
Guang Li, Bo Zhao, and Tongzhou Wang, “Awesome dataset distillation,” https://github.com/Guang000/ Awesome-Dataset-Distillation, 2022
2022
-
[11]
Soft-label anonymous gastric x-ray image distillation,
Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama, “Soft-label anonymous gastric x-ray image distillation,” in Proc. ICIP, 2020, pp. 305–309
2020
-
[12]
Compressed gastric image generation based on soft-label dataset distillation for medical data sharing,
Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama, “Compressed gastric image generation based on soft-label dataset distillation for medical data sharing,” Computer Meth- ods and Programs in Biomedicine , vol. 227, pp. 1–10, 2022
2022
-
[13]
Dataset distillation for medical dataset sharing,
Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama, “Dataset distillation for medical dataset sharing,” in Proc. AAAI Workshop, 2023, pp. 1–6
2023
-
[14]
An efficient dataset condensation plugin and its application to continual learning,
Enneng Yang, Li Shen, Zhenyi Wang, Tongliang Liu, and Guibing Guo, “An efficient dataset condensation plugin and its application to continual learning,” in Proc. NeurIPS, 2023, pp. 1–18
2023
-
[15]
Feder- ated learning via decentralized dataset distillation in resource- constrained edge environments,
Rui Song, Dai Liu, Dave Zhenyu Chen, Andreas Festag, Carsten Trinitis, Martin Schulz, and Alois Knoll, “Feder- ated learning via decentralized dataset distillation in resource- constrained edge environments,” in Proc. IJCNN, 2023, pp. 1–10
2023
-
[16]
Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data,
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth Stanley, and Jeffrey Clune, “Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data,” in Proc. ICML, 2020, pp. 9206–9216
2020
-
[18]
Dataset distillation by matching training trajectories,
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu, “Dataset distillation by matching training trajectories,” in Proc. CVPR , 2022, pp. 4750–4759
2022
-
[19]
Importance-aware adaptive dataset distillation,
Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama, “Importance-aware adaptive dataset distillation,” Neural Net- works, 2024
2024
-
[20]
DiM: Distilling dataset into generative model,
Kai Wang, Jianyang Gu, Daquan Zhou, Zheng Zhu, Wei Jiang, and Yang You, “DiM: Distilling dataset into generative model,” arXiv preprint arXiv:2303.04707, 2023
2023 arXiv
-
[21]
Generative dataset distillation: Balancing global structure and local details,
Longzhen Li, Guang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa, and Miki Haseyama, “Generative dataset distillation: Balancing global structure and local details,” in Proc. CVPR Workshop, 2024, pp. 7664–7671
2024
-
[22]
Generative dataset distillation based on diffusion model,
Duo Su, Junjie Hou, Guang Li, Ren Togo, Rui Song, Takahiro Ogawa, and Miki Haseyama, “Generative dataset distillation based on diffusion model,” in Proc. ECCV Workshop, 2024
2024
-
[23]
Generative adversarial nets,
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio, “Generative adversarial nets,” in Proc. NeurIPS, 2014, pp. 1–9
2014
-
[24]
Super-samples from kernel herding,
Yutian Chen, Max Welling, and Alex Smola, “Super-samples from kernel herding,” in Proc. UAI, 2010, pp. 109–116
2010
-
[25]
Fair clustering through fairlets,
Flavio Chierichetti, Ravi Kumar, Silvio Lattanzi, and Sergei Vassilvitskii, “Fair clustering through fairlets,” in Proc. NeurIPS, 2017, pp. 1–9
2017
-
[26]
An empirical study of example forgetting during deep neural net- work learning,
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon, “An empirical study of example forgetting during deep neural net- work learning,” in Proc. ICLR, 2019, pp. 1–18
2019
-
[27]
Dataset condensation with dif- ferentiable siamese augmentation,
Bo Zhao and Hakan Bilen, “Dataset condensation with dif- ferentiable siamese augmentation,” in Proc. ICML, 2021, pp. 12674–12685
2021
-
[28]
Dataset condensation with distri- bution matching,
Bo Zhao and Hakan Bilen, “Dataset condensation with distri- bution matching,” in Proc. WACV, 2023, pp. 6514–6523
2023
-
[29]
CAFE: Learning to condense dataset by aligning features,
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You, “CAFE: Learning to condense dataset by aligning features,” in Proc. CVPR, 2022, pp. 12196–12205
2022
-
[30]
Dataset distillation with infinitely wide convolutional networks,
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee, “Dataset distillation with infinitely wide convolutional networks,” in Proc. NeurIPS, 2021, pp. 5186–5198
2021
-
[31]
Dataset distillation using neural feature regression,
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba, “Dataset distillation using neural feature regression,” in Proc. NeurIPS, 2022, pp. 1–41
2022
-
[32]
Conditional generative adversarial nets,
Mehdi Mirza and Simon Osindero, “Conditional generative adversarial nets,” arXiv preprint arXiv:1411.1784, 2014
2014 arXiv
-
[33]
Gradient-based learning applied to document recog- nition,
Yann LeCun, L ´eon Bottou, Yoshua Bengio, and Patrick Haffner, “Gradient-based learning applied to document recog- nition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278– 2324, 1998
1998
-
[34]
Fashion-mnist: a novel image dataset for benchmarking machine learning al- gorithms,
Han Xiao, Kashif Rasul, and Roland V ollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning al- gorithms,” arXiv preprint arXiv:1708.07747, 2017
2017 arXiv
-
[35]
Learning multiple layers of features from tiny images,
Alex Krizhevsky, Geoffrey Hinton, et al., “Learning multiple layers of features from tiny images,” 2009
2009
-
[36]
Dynamic few-shot vi- sual learning without forgetting,
Spyros Gidaris and Nikos Komodakis, “Dynamic few-shot vi- sual learning without forgetting,” in Proc. CVPR, 2018, pp. 4367–4375
2018
-
[37]
Deep residual learning for image recognition,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition,” in Proc. CVPR, 2016, pp. 770–778
2016
-
[38]
Im- agenet classification with deep convolutional neural networks,
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, “Im- agenet classification with deep convolutional neural networks,” in Proc. NeurIPS, 2012, pp. 1097–1105
2012
-
[39]
Very deep convo- lutional networks for large-scale image recognition,
Karen Simonyan and Andrew Zisserman, “Very deep convo- lutional networks for large-scale image recognition,” in Proc. ICLR, 2015, pp. 1–14
2015
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.