Pith. sign in

REVIEW 5 major objections 4 minor 39 references

Generative Dataset Distillation Based on Self-knowledge Distillation

T0 review · 5 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read A generative dataset distillation method that standardizes prediction logits and matches their probability distributions through self-knowledge distillation reports higher accuracy than existing distillation baselines on MNIST…

desk verdict A modest DiM variant with real but small gains; the claimed attribution to the KL loss needs a controlled ablation. read the letter →

arxiv 2501.04202 v1 pith:HSINMGLM submitted 2025-01-08 cs.CV cs.AIcs.LG

classification cs.CVcs.AIcs.LG
keywords generativedatasetdistillationself-knowledgelogitsstandardizationdistributionmatchingcondensationcross-architecturegeneralizationconditionalGAN
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dataset distillation compresses a large training set into a small synthetic one, and generative variants train a GAN to produce that synthetic set on demand. This paper proposes a generative distillation method whose core claim is that matching the probability distributions of prediction logits, rather than matching raw logits, makes the synthetic data capture the original data's structure more faithfully. To make the matching stable, the paper standardizes the logits before the softmax, so that original and synthetic predictions are placed on the same scale. The authors report that this combination beats existing dataset distillation methods on three benchmark datasets and that the gains persist when the distilled generator is evaluated on architectures not used during training.

What carries the argument

The machinery is the self-knowledge distillation loss $L_{\text{SKD}}$, which matches the class-probability distributions of original and synthetic logits through a KL-style divergence instead of minimizing the mean squared error between raw logits. The standardization step $Z(x;\tau)$ removes scale and shift differences between the two logit vectors before the softmax, which the paper argues prevents the matching from being misled by similar ranges of inaccurate predictions. At each matching step, a model is randomly selected from a pool of ConvNet3, ResNet10, and ResNet18, so the generator is not optimized against a single fixed architecture. The generator itself is trained with the conditional GAN loss $L_{\text{CGAN}}$, and the distillation loss is added with a weighted temperature-scaled term to form the total objective.

What would settle it

Run the method with a model pool containing only ConvNet3 and evaluate the distilled generator on VGG11; if cross-architecture accuracy falls to roughly the DiM level, the random-pool assumption, rather than standardization, is responsible for the generalization gain. Equivalently, sweep the temperature $\tau$ over a wide range; if the standardization benefit disappears at some values of $\tau$, the claim that standardization itself is the cause of the improvement would need to be revised.

Watch

Extended reading notes

Core claim

The central discovery is that replacing plain logits matching with a self-knowledge-distillation loss, applied to standardized logits, improves the accuracy of the synthetic data produced by the generator. Formally, the paper computes $Z(x;\tau) = \frac{x - \text{mean}(x)}{\text{std}(x)} \times \tau$, feeds the standardized logits through softmax to obtain $d(x_O)$ and $d(x_S)$, and then uses $L_{\text{SKD}} = \sum_{k=1}^{K} d(x_O)^{(k)} \log \frac{d(x_O)^{(k)}}{d(x_S)^{(k)}}$. The total training loss is $L_{\text{total}} = L_{\text{CGAN}} + \lambda_{\text{SKD}} \tau^2 L_{\text{SKD}}$, where $L_{\text{CGAN}}$ is the conditional GAN loss. The paper's empirical claim is that this design outperforms DiM and other state-of-the-art condensation methods on MNIST, Fashion-MNIST, and CIFAR-10 at images-per-class values of 1, 10, and 50, and that the standardization step alone produces consistent accuracy gains.

Load-bearing premise

The load-bearing premise is that a matching model randomly drawn from a pool of just three architectures, ConvNet3, ResNet10, and ResNet18, provides a distribution-matching signal that transfers to other architectures; the paper does not show why three models suffice or how sensitive the results are to the pool's composition.

Editorial extensions

If this is right

  • Standardizing logits before distribution matching produces consistent accuracy gains over the same method without standardization, across all three datasets and all images-per-class settings tested.
  • Randomly drawing the matching model from a small pool of architectures gives synthetic data that transfers to unseen architectures such as AlexNet and VGG11, with the standardized version reporting the best cross-architecture results.
  • Distribution matching of standardized probability vectors captures global structure and inter-class relationships better than direct logits matching, which is why the generator retains more representative information from the original dataset.
  • The proposed method remains accurate even at very low compression budgets such as one image per class, and it improves further as the budget grows to 10 and 50 images per class.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not isolate how much of the gain comes from the random model pool versus the standardization step; a natural next experiment is to hold the pool fixed while toggling standardization, and vice versa, on a held-out architecture.
  • Because standardization removes logits scale and shift, the same normalization could likely be inserted into other logits-comparing distillation losses, such as trajectory matching or gradient matching, with similar stabilizing effects.
  • The fixed temperature $\tau = 2$ and per-dataset weight $\lambda_{\text{SKD}}$ suggest a tuning interaction worth probing; per-dataset temperature schedules might widen the reported margins beyond the fixed settings used here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes a generative dataset distillation method. A conditional GAN is trained to produce synthetic images, and the generator is further optimized by matching the softmax distributions of standardized logits from a randomly selected model in a pool, using a KL-divergence-based 'self-knowledge distillation' loss. Experiments on MNIST, FashionMNIST, and CIFAR-10 report accuracy improvements over DiM and other baselines, plus cross-architecture results on CIFAR-10.

Significance. If the reported gains are robust, the combination of logits standardization and KL-based distribution matching would be a simple, useful modification to existing generative dataset distillation (e.g., DiM). The paper ships a clear loss formulation and reproducible-looking experimental protocol. However, the empirical evidence currently does not isolate the contribution of the proposed self-knowledge distillation loss, because the main comparison with DiM differs in both the matching objective and the underlying generator/pool setup. The paper is nonetheless a plausible incremental contribution that could become acceptable after a controlled ablation study and additional baselines.

major comments (5)
  1. [Section 2.2, Table 1] The comparison with DiM is confounded. The proposed method changes the matching objective (KL divergence on standardized softmax outputs instead of DiM's MSE on logits) and also uses a different generator, discriminator, and model-pool training protocol. The 'Ours (No Stand.)' ablation removes only the standardization step; there is no condition that replaces LSKD with DiM-style logits MSE while keeping the generator, discriminator, pool, and schedule identical. As a result, the accuracy advantage over DiM in Tables 1 and 2 cannot be attributed to the self-knowledge distillation loss, which is the central novelty claimed in the abstract and introduction.
  2. [Section 3.2] The model pool used for random model selection is underspecified. The paper states that a model is 'randomly selected from a pool of models' consisting of ConvNet3, ResNet10, and ResNet18, but it does not state how these networks are trained (on which dataset, with what optimizer, for how many epochs, with which augmentation). Without this information the cross-architecture generalization claimed in Section 3.3 is not reproducible, and the reader cannot assess whether the pool composition or the loss is responsible for the cross-architecture performance.
  3. [Table 1] The abstract claims the method 'outperforms existing state-of-the-art methods', but the reported results do not support this uniformly. On MNIST IPC=50, Ours (99.1±0.1) is below DiM (99.2±0.2), and on FashionMNIST IPC=50 Ours ties DiM at 89.8. The claim should be qualified to reflect the settings where the improvement is consistent, or the discrepancy should be discussed.
  4. [Section 3.1 (missing reference [17])] The authors' own related generative dataset distillation method, reference [17] (CVPR Workshop 2024), is cited in the bibliography but never compared in the experiments. Since [17] addresses the same task and is likely closely related, a direct empirical comparison is necessary to establish the novelty and incremental benefit of the proposed self-knowledge distillation and standardization components.
  5. [Equations (1) and (7)] The use of LCGAN in the total loss is ambiguous. Equation (1) defines a min-max GAN objective, but Equation (7) treats LCGAN as an additive term in Ltotal, which is then said to be minimized with respect to the generator parameters W. The paper should specify whether the discriminator is fixed when W is updated, how the adversarial term is computed (e.g., using the standard generator loss rather than the full min-max objective), and how the gradient flows through the GAN loss during the distribution-matching stage. This is essential for reproducibility.
minor comments (4)
  1. [Tables 1 and 2] The word 'comparation' in the table captions should be 'comparison'.
  2. [Section 3.2] The sentence 'IPC values were set to 1, 10, and 50, while batch sizes were 32, 64, and 128' should specify which batch size corresponds to which IPC value, since the pairing is needed to interpret the experiments.
  3. [Equation (3)] The standardization in Equation (3) computes mean and std over the logits vector for each input sample; this should be stated explicitly, and the paper should discuss how zero standard deviation (e.g., for very confident or degenerate outputs) is handled.
  4. [Remark on terminology] The term 'self-knowledge distillation' is used for a KL divergence between the softmax outputs of the same model on original and synthetic data; this is essentially distribution matching (as in DM) applied to logits. The paper should relate the term to prior usage of self-knowledge distillation and clarify the difference from simple logits-matching in DiM.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; reported gains are measured on held-out test data and the proposed loss is not derived from the reported accuracies.

full rationale

The central claim is an empirical accuracy comparison, not a quantity defined by the proposed loss. LSKD (Eq. 6) is a training objective that matches softmax distributions of standardized logits between original and synthetic data; the reported accuracies in Tables 1 and 2 are obtained by training fresh models on generated synthetic data and evaluating on held-out test sets, so the result is not forced by construction. The self-citations [15, 17, 18] appear only in related-work context and are not invoked as evidence for the proposed loss or the empirical gains, so they are not load-bearing. The standardization ablation ('Ours (No Stand.)' vs. 'Ours') is a within-method comparison, and the hyperparameters lambda_SKD and tau are dataset-level training choices rather than fitted predictors of test accuracy. The skeptical concern about confounded baselines (different GAN architecture, unspecified model-pool training) is an experimental-rigor and validity threat, not a circular derivation. No passage in the manuscript asserts a limitation or missing proof that would alter this assessment; the underspecification of GAN and model-pool training is an experimental reporting gap, not a circular step.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method relies on standard GAN training, the validity of KL divergence as a distribution-matching objective, the sufficiency of a three-model pool for cross-architecture generalization, and the assumption that z-score standardizing logits preserves class information. It introduces two fitted hyperparameters (lambda_SKD and tau) and no invented entities.

free parameters (2)
  • lambda_SKD = 0.001 (MNIST), 0.01 (FashionMNIST, CIFAR-10)
    Weight of the self-knowledge distillation loss in Eq. (7); chosen per dataset without sensitivity analysis.
  • tau = 2
    Temperature in Eq. (3); fixed across all experiments, no ablation reported.
assumptions (4)
  • domain assumption Conditional GAN training with loss in Eq. (1) yields a generator whose synthetic distribution is a useful starting point for distillation.
    The method relies on the standard GAN objective to provide synthetic data before logits matching.
  • domain assumption KL divergence between softmax outputs (Eq. 6) is an appropriate objective for aligning original and synthetic data distributions.
    The paper replaces MSE logits matching with KL divergence without theoretical justification or comparison to other divergences.
  • domain assumption A pool of three architectures (ConvNet3, ResNet10, ResNet18) is representative enough for cross-architecture generalization.
    Section 2.2 randomly selects a model from this pool; Section 3.3 claims generalization to AlexNet and VGG11 based on this.
  • domain assumption Per-sample z-score standardization of logits (Eq. 3) removes scale and shift variability without losing class-discriminative information.
    The central innovation; no analysis is given of what information is discarded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generative Dataset Distillation Based on Self-knowledge Distillation." pith.science (2026). https://pith.science/paper/HSINMGLM

@misc{pith2026250104202,
  author       = {Pith},
  title        = {Pith review of: Generative Dataset Distillation Based on Self-knowledge Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HSINMGLM}},
  note         = {Machine review of arXiv:2501.04202}
}
read the original abstract

Dataset distillation is an effective technique for reducing the cost and complexity of model training while maintaining performance by compressing large datasets into smaller, more efficient versions. In this paper, we present a novel generative dataset distillation method that can improve the accuracy of aligning prediction logits. Our approach integrates self-knowledge distillation to achieve more precise distribution matching between the synthetic and original data, thereby capturing the overall structure and relationships within the data. To further improve the accuracy of alignment, we introduce a standardization step on the logits before performing distribution matching, ensuring consistency in the range of logits. Through extensive experiments, we demonstrate that our method outperforms existing state-of-the-art methods, resulting in superior distillation performance.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 33 canonical work pages

  1. [17]

    Dataset condensation with gradi- ent matching,

    Bo Zhao and Hakan Bilen, “Dataset condensation with gradi- ent matching,” in Proc. ICLR, 2021, pp. 1–20

  2. [1]

    Generative Dataset Distillation Based on Self-knowledge Distillation

    INTRODUCTION The rapid advancement of deep learning has driven the creation of increasingly large models that require vast amounts of data to achieve optimal performance [1]. However, this growth in data vol- ume brings with it several significant challenges. First, storing and maintaining large datasets incurs high costs, both in terms of storage and the...

  3. [2]

    First, we train a generative adversarial network (GAN) to generate a synthetic dataset S

    GENERA TIVE DA TASET DISTILLA TION BASED ON SELF-KNOWLEDGE DISTILLA TION The proposed method consists of two main steps. First, we train a generative adversarial network (GAN) to generate a synthetic dataset S. Next, a model is randomly selected from a model pool to align the synthetic dataset S with the original dataset O. To improve this alignment, we i...

  4. [3]

    Datasets and Comparative Methods We conducted extensive experiments to verify the effectiveness of the proposed method

    EXPERIMENTS 3.1. Datasets and Comparative Methods We conducted extensive experiments to verify the effectiveness of the proposed method. First, we designed benchmark experi- ments using three datasets: MNIST [29], FashionMNIST [30], and CIFAR-10 [31]. Each dataset was categorized into 10 classes. To validate the performance of our approach, we compared it...

  5. [4]

    CONCLUSION In this paper, we have proposed a novel generative dataset distillation method that incorporates self-knowledge distillation to improve the overall distillation process. A key innovation of our approach is the redesign of the logits matching process, where we employ distribu- tion matching to better align the prediction logits between the origi...

  6. [5]

    A survey of deep learning and its applica- tions: a new paradigm to machine learning,

    Shaveta Dargan, Munish Kumar, Maruthi Rohit Ayyagari, and Gulshan Kumar, “A survey of deep learning and its applica- tions: a new paradigm to machine learning,” Archives of Com- putational Methods in Engineering , vol. 27, pp. 1071–1092, 2020

  7. [6]

    Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions,

    Laith Alzubaidi, Jinglan Zhang, and et al., “Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions,” Journal of Big Data, vol. 8, pp. 1–74, 2021

  8. [7]

    Applica- tions for deep learning in ecology,

    Sylvain Christin, ´Eric Hervet, and Nicolas Lecomte, “Applica- tions for deep learning in ecology,” Methods in Ecology and Evolution, vol. 10, no. 10, pp. 1632–1644, 2019

Show all 39 references
  1. [8]

    Big healthcare data: preserving security and pri- vacy,

    Karim Abouelmehdi, Abderrahim Beni-Hessane, and Hayat Khaloufi, “Big healthcare data: preserving security and pri- vacy,” Journal of Big Data, vol. 5, no. 1, pp. 1–18, 2018

  2. [9]

    Dataset distillation,

    Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros, “Dataset distillation,” arXiv preprint arXiv:1811.10959, 2018

  3. [10]

    Awesome dataset distillation,

    Guang Li, Bo Zhao, and Tongzhou Wang, “Awesome dataset distillation,” https://github.com/Guang000/ Awesome-Dataset-Distillation, 2022

  4. [11]

    Soft-label anonymous gastric x-ray image distillation,

    Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama, “Soft-label anonymous gastric x-ray image distillation,” in Proc. ICIP, 2020, pp. 305–309

  5. [12]

    Compressed gastric image generation based on soft-label dataset distillation for medical data sharing,

    Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama, “Compressed gastric image generation based on soft-label dataset distillation for medical data sharing,” Computer Meth- ods and Programs in Biomedicine , vol. 227, pp. 1–10, 2022

  6. [13]

    Dataset distillation for medical dataset sharing,

    Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama, “Dataset distillation for medical dataset sharing,” in Proc. AAAI Workshop, 2023, pp. 1–6

  7. [14]

    An efficient dataset condensation plugin and its application to continual learning,

    Enneng Yang, Li Shen, Zhenyi Wang, Tongliang Liu, and Guibing Guo, “An efficient dataset condensation plugin and its application to continual learning,” in Proc. NeurIPS, 2023, pp. 1–18

  8. [15]

    Feder- ated learning via decentralized dataset distillation in resource- constrained edge environments,

    Rui Song, Dai Liu, Dave Zhenyu Chen, Andreas Festag, Carsten Trinitis, Martin Schulz, and Alois Knoll, “Feder- ated learning via decentralized dataset distillation in resource- constrained edge environments,” in Proc. IJCNN, 2023, pp. 1–10

  9. [16]

    Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data,

    Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth Stanley, and Jeffrey Clune, “Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data,” in Proc. ICML, 2020, pp. 9206–9216

  10. [18]

    Dataset distillation by matching training trajectories,

    George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu, “Dataset distillation by matching training trajectories,” in Proc. CVPR , 2022, pp. 4750–4759

  11. [19]

    Importance-aware adaptive dataset distillation,

    Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama, “Importance-aware adaptive dataset distillation,” Neural Net- works, 2024

  12. [20]

    DiM: Distilling dataset into generative model,

    Kai Wang, Jianyang Gu, Daquan Zhou, Zheng Zhu, Wei Jiang, and Yang You, “DiM: Distilling dataset into generative model,” arXiv preprint arXiv:2303.04707, 2023

  13. [21]

    Generative dataset distillation: Balancing global structure and local details,

    Longzhen Li, Guang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa, and Miki Haseyama, “Generative dataset distillation: Balancing global structure and local details,” in Proc. CVPR Workshop, 2024, pp. 7664–7671

  14. [22]

    Generative dataset distillation based on diffusion model,

    Duo Su, Junjie Hou, Guang Li, Ren Togo, Rui Song, Takahiro Ogawa, and Miki Haseyama, “Generative dataset distillation based on diffusion model,” in Proc. ECCV Workshop, 2024

  15. [23]

    Generative adversarial nets,

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio, “Generative adversarial nets,” in Proc. NeurIPS, 2014, pp. 1–9

  16. [24]

    Super-samples from kernel herding,

    Yutian Chen, Max Welling, and Alex Smola, “Super-samples from kernel herding,” in Proc. UAI, 2010, pp. 109–116

  17. [25]

    Fair clustering through fairlets,

    Flavio Chierichetti, Ravi Kumar, Silvio Lattanzi, and Sergei Vassilvitskii, “Fair clustering through fairlets,” in Proc. NeurIPS, 2017, pp. 1–9

  18. [26]

    An empirical study of example forgetting during deep neural net- work learning,

    Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon, “An empirical study of example forgetting during deep neural net- work learning,” in Proc. ICLR, 2019, pp. 1–18

  19. [27]

    Dataset condensation with dif- ferentiable siamese augmentation,

    Bo Zhao and Hakan Bilen, “Dataset condensation with dif- ferentiable siamese augmentation,” in Proc. ICML, 2021, pp. 12674–12685

  20. [28]

    Dataset condensation with distri- bution matching,

    Bo Zhao and Hakan Bilen, “Dataset condensation with distri- bution matching,” in Proc. WACV, 2023, pp. 6514–6523

  21. [29]

    CAFE: Learning to condense dataset by aligning features,

    Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You, “CAFE: Learning to condense dataset by aligning features,” in Proc. CVPR, 2022, pp. 12196–12205

  22. [30]

    Dataset distillation with infinitely wide convolutional networks,

    Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee, “Dataset distillation with infinitely wide convolutional networks,” in Proc. NeurIPS, 2021, pp. 5186–5198

  23. [31]

    Dataset distillation using neural feature regression,

    Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba, “Dataset distillation using neural feature regression,” in Proc. NeurIPS, 2022, pp. 1–41

  24. [32]

    Conditional generative adversarial nets,

    Mehdi Mirza and Simon Osindero, “Conditional generative adversarial nets,” arXiv preprint arXiv:1411.1784, 2014

  25. [33]

    Gradient-based learning applied to document recog- nition,

    Yann LeCun, L ´eon Bottou, Yoshua Bengio, and Patrick Haffner, “Gradient-based learning applied to document recog- nition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278– 2324, 1998

  26. [34]

    Fashion-mnist: a novel image dataset for benchmarking machine learning al- gorithms,

    Han Xiao, Kashif Rasul, and Roland V ollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning al- gorithms,” arXiv preprint arXiv:1708.07747, 2017

  27. [35]

    Learning multiple layers of features from tiny images,

    Alex Krizhevsky, Geoffrey Hinton, et al., “Learning multiple layers of features from tiny images,” 2009

  28. [36]

    Dynamic few-shot vi- sual learning without forgetting,

    Spyros Gidaris and Nikos Komodakis, “Dynamic few-shot vi- sual learning without forgetting,” in Proc. CVPR, 2018, pp. 4367–4375

  29. [37]

    Deep residual learning for image recognition,

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition,” in Proc. CVPR, 2016, pp. 770–778

  30. [38]

    Im- agenet classification with deep convolutional neural networks,

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, “Im- agenet classification with deep convolutional neural networks,” in Proc. NeurIPS, 2012, pp. 1097–1105

  31. [39]

    Very deep convo- lutional networks for large-scale image recognition,

    Karen Simonyan and Andrew Zisserman, “Very deep convo- lutional networks for large-scale image recognition,” in Proc. ICLR, 2015, pp. 1–14

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.