Pith. sign in

REVIEW 4 major objections 5 minor 69 references

Information-Guided Diffusion Sampling for Dataset Distillation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that maximizing prototype and contextual information during diffusion sampling produces distilled datasets that beat existing methods across all IPC settings.

desk verdict A useful empirical recipe for low-IPC diffusion distillation, but the information-theoretic derivation has a load-bearing sign error; the method is worth reviewing, the theory needs a major rewrite. read the letter →

arxiv 2507.04619 v1 pith:WYC3NXYR submitted 2025-07-07 cs.LG cs.AIcs.CVcs.ITmath.IT

Linfeng Ye , Shayan Mohajer Hamidi , Guang Li , Takahiro Ogawa , Miki Haseyama , Konstantinos N. Plataniotis This is my paper · ORCID
classification cs.LGcs.AIcs.CVcs.ITmath.IT
keywords datasetdistillationdiffusionmodelsinformationtheorymutualconditionalentropyvariationalestimationlow-IPCimagegenerationsamplingguidance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dataset distillation compresses a training set into a small number of images per class, and diffusion-based distillation has struggled when that number is very small. This paper claims the fix is to guide the diffusion sampling process with an information-theoretic objective: maximize prototype information $\mathrm{I}(X;Y)$ plus an IPC-dependent weight $\beta$ times contextual information $\mathrm{H}(X|Y)$. Because these quantities are intractable, the paper constructs a variational estimator from an encoder-classifier pair and derives tight lower bounds that can be evaluated during sampling. The resulting method, IGDS, outperforms prior diffusion-based distillation on Tiny ImageNet and ImageNet subsets, with the largest gains at one image per class.

What carries the argument

The load-bearing component is the variational estimator (VE), a pretrained encoder $f_\theta$ and linear classifier $g_\psi$ trained so that $I(X;\hat{X})$ is maximized via instance discrimination plus a conditional-KL term $\lambda\,\mathbb{E}\,KL(\sigma(\hat{X})\,\|\,Q_Y)$. The paper proves that this training shrinks the residual terms that separate the estimates from the true information, yielding the computable proxy $\mathrm{I}(X;Y)\approx \mathrm{H}(\hat{Y})+\mathbb{E}\log P_{\hat{Y}|Y}$ and $\mathrm{H}(X|Y)\approx \mathbb{E}_{X,Y}KL(\sigma(\hat{X})\,\|\,Q_Y)$. During sampling, the frozen VE supplies the gradient of $L_{\mathrm{IGDS}}=\mathbb{E}\log P_{\hat{Y}|Y}+\beta\,\mathbb{E}_{X,Y}KL(\sigma(\hat{X})\,\|\,Q_Y)$, steering each denoised image to be both label-typical and intra-class diverse.

What would settle it

Compute the true values of $\mathrm{I}(X;Y)$ and $\mathrm{H}(X|Y)$ for a small synthetic dataset where they are tractable (e.g., a Gaussian mixture with known class-conditional means and covariances), then run the trained VE on the same data and compare: if the estimates are not at or below the true values, or if their ordering across IPC-like subsets differs, the information-theoretic grounding fails. Alternatively, retrain the guidance with a randomly initialized encoder and check whether the IPC-1 accuracy gain over RDED persists.

Watch

Extended reading notes

Core claim

The central claim is that the Shannon decomposition $\mathrm{H}(X)=\mathrm{I}(X;Y)+\mathrm{H}(X|Y)$ identifies the two pieces of information a distilled dataset must keep: label-discriminative prototype information and intra-class contextual information. The paper argues the balance between them should depend on the images-per-class setting, and encodes that balance in the sampling objective $\mathrm{I}(X;Y)+\beta\,\mathrm{H}(X|Y)$. To make the objective computable, it develops variational lower bounds using an encoder-classifier network and uses them as a guidance gradient during DDPM sampling. On ImageWoof at IPC-1 this raises ConvNet-6 accuracy from 18.5% for RDED to 23.1%, with consistent gains at higher IPC and on ImageNette and Tiny ImageNet.

Load-bearing premise

The load-bearing premise is that the variational estimates are tight lower bounds on $\mathrm{I}(X;Y)$ and $\mathrm{H}(X|Y)$; the derivation uses an inequality that runs in the wrong direction and equates a model cross-entropy with a conditional entropy, which is only valid when the classifier is Bayes-optimal, so if those conditions fail the sampling objective is not actually maximizing the stated information.

Editorial extensions

If this is right

  • At low IPC, where diffusion distillation has historically collapsed to random-subset accuracy, IGDS reports a clear separation from prior diffusion methods (23.1% vs 18.5% at IPC-1 on ImageWoof).
  • The same frozen VE works with any diffusion prior: combining with MiniMax-DiT pushes ImageWoof IPC-1 accuracy to 37.6%.
  • The accuracy-versus-$\beta$ curves give a practical tuning rule: use small $\beta$ at low IPC to emphasize prototype information and larger $\beta$ as IPC grows.
  • Distilled sets transfer across architectures (ConvNet-6, ResNetAP-10, ResNet-18, and cross-architecture settings), so the method is not tied to the evaluation network.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The $\beta$-versus-IPC trend looks like a rate-distortion trade-off with prototype information as the rate term and contextual information as the distortion; if that analogy holds, $\beta$ could be set from an information budget rather than by grid search.
  • If the variational bounds are looser than claimed, the gains may stem from the classifier gradient acting as a class-conditional prior during diffusion sampling, a mechanism testable by replacing the trained VE with a randomly initialized encoder.
  • The same decomposition of $\mathrm{H}(X)$ could be applied to other generative data-sharing problems, such as continual-learning buffers or privacy-preserving image release, where one wants controllable label-typicality versus intra-class variety.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes IGDS, a diffusion-sampling method for dataset distillation that maximizes an information-theoretic objective I(X;Y) + beta H(X|Y), with beta chosen according to the images-per-class (IPC) setting. The authors argue that prototype information I(X;Y) captures label-relevant features and contextual information H(X|Y) captures intra-class variability, and that low-IPC regimes require more emphasis on prototype information while high-IPC regimes benefit from contextual diversity. Because the two information quantities are intractable, the paper trains a variational estimator (a contrastive encoder plus a classifier) and claims tight lower bounds on both quantities, then uses those estimates to guide diffusion sampling. Experiments on ImageWoof, ImageNette, Tiny ImageNet, and CIFAR-10 report improvements over prior generative distillation methods, especially at low IPC.

Significance. The problem is well motivated: diffusion-based dataset distillation indeed suffers in low-IPC regimes, and the proposed remedy is conceptually appealing. The empirical results are strong and include several architectures and datasets, with particularly notable gains at IPC-1 (e.g., 23.1% vs. 18.5% for RDED on ImageWoof with ConvNet-6, Table 1). The paper also provides appendix proofs, runtime analysis, cross-architecture results, and sample visualizations. However, the central theoretical claim—that the variational estimators tightly lower-bound I(X;Y) and H(X|Y), and hence that Eq. (25) maximizes the stated information measure—is not established. The derivation contains a sign error, an invalid equality in Proposition 4, an unjustified identity in Eqs. (11)-(13), and an inconsistency between Eq. (25) and Algorithm 2. Because these issues are load-bearing for the paper's information-theoretic framing, the contribution as written is not yet sound, even though the empirical method may still be effective.

major comments (4)
  1. [Section 4.1.1, Eq. (6)] The inequality in Eq. (6) has the wrong direction. Conditioning reduces entropy, so H(Y|Ŷ,Y) ≤ H(Y|Ŷ), and therefore H(Y)−H(Y|Ŷ) ≤ H(Y)−H(Y|Ŷ,Y). Since H(Y|Ŷ,Y)=0, the claimed inequality would assert I(Ŷ;Y) ≥ H(Y), which is false except in the trivial case where Ŷ is a sufficient statistic for Y. The accompanying statement that equality holds when PY|Ŷ = PY is also incorrect, since that condition gives I(Ŷ;Y)=0. Consequently, the derivation of Eq. (9) via Eqs. (5)-(7) does not establish the claimed tight lower bound on I(X;Y), although I(Ŷ;Y) is itself a lower bound on I(X;Y) by Eq. (3).
  2. [Section 4.1.2, Proposition 4] The proof of Proposition 4 in Appendix B.4 (Eq. (49)) asserts H(X|Ŷ,Y)=H(X|Ŷ) 'due to Y→X→Ŷ forms a Markov chain.' This equality is not a consequence of that Markov chain; it requires I(X;Y|Ŷ)=0. The Markov chain only gives Y⊥Ŷ|X. Hence the equivalence min_θ H(X|Ŷ,Y) ≡ max_θ I(X;Ŷ) is not established, and Eq. (15) cannot be claimed as a tight lower bound on H(X|Y).
  3. [Section 4.1.2, Eqs. (11)-(13)] The identity I(X;σ(Ŷ)|y)=E_{X|y} KL(σ(Ŷ)||Q_y) is not generally true. Since σ(Ŷ) is a deterministic function of X, I(X;σ(Ŷ)|y)=H(σ(Ŷ)|y). The right-hand side is an average KL divergence between the softmax feature of a sample and the class-average Q_y; this is not the conditional entropy of σ(Ŷ) given y unless a very specific relationship between the per-sample softmax distribution and Q_y holds. No such relationship is stated or proved, so the variational estimator for H(X|Y) in Eq. (15) is not supported.
  4. [Section 4.2, Eq. (25) vs. Algorithm 2] Eq. (25) and Algorithm 2 line 12 are inconsistent. Eq. (25) gives LIGDS = E log P_{Y|Ŷ} + β I(Ŷ;σ(Ŷ)|Y), while Algorithm 2 uses E log P_{ŷ|y} + H(ŷ) + β KL(H_{x_{t-1}}||Q_{t-1}). The prototype-information part differs by the H(ŷ) term and by whether the conditional model is P(Y|Ŷ) or P(Ŷ|Y). The paper should state the exact maximization objective used in the experiments and prove that it corresponds to Eq. (2).
minor comments (5)
  1. [Section 5.3] The text writes 'IDGS' in the comparison paragraph; this should be 'IGDS'.
  2. [Section 4.1.1, Eq. (7)] The notation E_Y log P_{Y|Ŷ} is ambiguous; the expectation should be defined with respect to the joint distribution of (X,Y) and the classifier's soft output, and the sentence 'we use one-hot probabilities to estimate the Bayes probabilities' needs a precise justification or reference.
  3. [Appendix B.1, Proposition 1] Proposition 1 assumes the linear classifier has full column rank, but the rank of the trained classifier is never verified in the experiments; please either provide the verification or qualify the claim.
  4. [Appendix B.3, Proposition 3] The proof assumes the feature representation has zero mean; this condition is not checked empirically, and the paper does not explain how the encoder is normalized to satisfy it.
  5. [Table 5] The CIFAR-10 results in Table 5 report accuracies without standard deviations, unlike Tables 1-3; please add them for consistency and to assess significance.

Circularity Check

2 steps flagged · score 6.0 of 10

The information-theoretic objective is definitionally tied to the KL heuristic because the claimed lower-bound proofs are invalid, and the beta-IPC scaling is a post-hoc summary of tuned hyperparameters.

  1. self definitional [Section 4.1.2 Eq. (15); Section 4.2 Eq. (25); Algorithm 2 line 12]
    "H(X|Y) = EX,Y KL(σ( ˆX)||QY ). (15) ... Using these estimates, the objective function in eq. (2) is reformulated as: LIGDS = E logPY| ˆY +βI( ˆX;σ( ˆX)|Y). (25) ... LIGDS = E logPˆy|y + H(ˆy) +βKL(Hˆxt−1||Qt−1)."

    The contextual-information estimator in Eq. (15) is the KL divergence between softmax features and a class-average Q, and the sampling objective in Eq. (25) and Algorithm 2's beta KL(H||Q) term is the same KL objective over the generated batch. The proof that this KL expression lower-bounds H(X|Y) is invalid: Eq. (6) has the inequality reversed, and Proposition 4's proof uses H(X|Xhat)=H(X|Xhat,Y), which requires I(X;Y|Xhat)=0 and is not implied by the stated Markov chain. Thus the claim to maximize Shannon contextual information reduces by construction to maximizing the chosen heuristic KL loss; the information-theoretic vocabulary is a renaming of the objective rather than an independent derivation.

  2. fitted input called prediction [Section 5.2 and Section 7.1 / Fig. 5]
    "The optimal value of β should be empirically determined for different IPC settings. To illustrate this, fig. 5 shows the test accuracy of the model as a function of β under varying IPC values. As observed, higher IPC settings benefit from a larger β, aligning with the findings discussed in section 5.2."

    Beta is a free hyperparameter selected per IPC by sweeping to maximize model accuracy, so the observed trend that optimal beta increases with IPC is a summary of the fitted values rather than an independent prediction of an information-theoretic scaling law. The supporting experiment in Section 5.2 also controls H(X|Y) using the paper's own KL estimator (Eqs. 15 and 26), so the 'required contextual information scales with IPC' claim is a property of the adopted estimator, not an external validation of Shannon conditional entropy.

full rationale

The paper's central empirical comparisons on ImageWoof, ImageNette, and Tiny ImageNet are external and not circular: the distilled datasets are evaluated on held-out validation sets against published baselines. However, the theoretical derivation that motivates the method is not self-contained in the way claimed. The variational lower bounds in Eqs. (9) and (15) are not established: Eq. (6) has the inequality reversed, and Proposition 4's Markov-chain step is invalid, so the estimator in Eq. (15) is not shown to lower-bound H(X|Y). As a result, the sampling objective in Eq. (25) and Algorithm 2 is definitionally the same KL heuristic it calls contextual information, which is a self-definitional reduction of the information-theoretic claim. Additionally, the beta-IPC relationship is presented as an observed scaling law but is actually a post-hoc description of hyperparameters tuned to maximize accuracy. These issues are partial circularity: the theoretical framing reduces to the heuristic loss, but the reported accuracy improvements remain independent empirical findings.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method depends on several unproven or loosely justified assumptions about how well contrastive features and averaged softmax outputs represent the information-theoretic quantities in Eq. (1). The estimator is inherited from the authors' prior work, and its validity is not checked against ground-truth entropies for small synthetic cases. Beta is a fitted hyperparameter, and the claimed IPC-dependence of contextual information is an empirical observation using the same estimator.

free parameters (4)
  • beta = IPC-dependent; selected empirically (Fig. 5, e.g., 0, 0.1, 0.5)
    Controls the trade-off between prototype and contextual information; chosen per dataset and IPC to maximize validation accuracy, not derived.
  • lambda = Not specified in the paper
    Balances the contrastive loss and the KL term in the VE training objective (Eq. 24); no sensitivity analysis is provided.
  • temperature tau = 0.07 (fixed in all experiments)
    Borrowed from MoCo and used in both VE training and IGDS sampling; treated as a constant.
  • sampling steps = 250
    Number of diffusion steps for IGDS; fixed across experiments.
assumptions (5)
  • domain assumption Y -> X -> \hat{X} -> \hat{Y} forms a Markov chain
    Assumed in Section 4.1 and Fig. 1; standard for supervised learning.
  • domain assumption The encoder features \hat{X} are zero mean so that softmax is injective
    Required for Proposition 3; not verified after training the MoCo encoder.
  • standard math The contrastive loss (MoCo) maximizes I(X;\hat{X}) via InfoNCE
    InfoNCE is a lower bound on MI; the paper assumes it is tight enough for the downstream objective.
  • ad hoc to paper Q_y = average softmax features over class y estimates the class prototype distribution
    Used in Eq. (13) and Algorithm 2; no proof that this average captures the needed distribution.
  • domain assumption The relationship between required contextual information and IPC is a fixed property of the data
    Empirically observed in Fig. 2 and used to justify beta selection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Information-Guided Diffusion Sampling for Dataset Distillation." pith.science (2026). https://pith.science/paper/WYC3NXYR

@misc{pith2026250704619,
  author       = {Pith},
  title        = {Pith review of: Information-Guided Diffusion Sampling for Dataset Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WYC3NXYR}},
  note         = {Machine review of arXiv:2507.04619}
}
abstract

Dataset distillation aims to create a compact dataset that retains essential information while maintaining model performance. Diffusion models (DMs) have shown promise for this task but struggle in low images-per-class (IPC) settings, where generated samples lack diversity. In this paper, we address this issue from an information-theoretic perspective by identifying two key types of information that a distilled dataset must preserve: ($i$) prototype information $\mathrm{I}(X;Y)$, which captures label-relevant features; and ($ii$) contextual information $\mathrm{H}(X | Y)$, which preserves intra-class variability. Here, $(X,Y)$ represents the pair of random variables corresponding to the input data and its ground truth label, respectively. Observing that the required contextual information scales with IPC, we propose maximizing $\mathrm{I}(X;Y) + \beta \mathrm{H}(X | Y)$ during the DM sampling process, where $\beta$ is IPC-dependent. Since directly computing $\mathrm{I}(X;Y)$ and $\mathrm{H}(X | Y)$ is intractable, we develop variational estimations to tightly lower-bound these quantities via a data-driven approach. Our approach, information-guided diffusion sampling (IGDS), seamlessly integrates with diffusion models and improves dataset distillation across all IPC settings. Experiments on Tiny ImageNet and ImageNet subsets show that IGDS significantly outperforms existing methods, particularly in low-IPC regimes. The code will be released upon acceptance.

Figures

Figures reproduced from arXiv: 2507.04619 by the authors.

Figure 1
Figure 1. Multi-class classification can be modeled as a Markov chain. Sample [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. (a) H(X|Y ) of selected subsets Vs. model validation accuracy for different IPC settings; the subset associated with the highest model accuracy is marked with a red dot. (b) H(X|Y ) of the best subset compared across different IPC settings. As per theorem 1, to maximize I(X; Xˆ) one can instead maximize the three terms: H(Y˜ ), −H(Y˜ |Xˆ), and I(X; Xˆ|Y˜ ). In the following, we discuss how to max￾imize these three t… view at source ↗
Figure 3
Figure 3. Illustration of prototype and contextual information. ( [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Generated images of two classes: English Springer (first row) and Tench (second row), with varying [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: The model’s accuracy on the distilled dataset Vs. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Mutual information between Xˆ and X for models trained with objectives of (a) information bottleneck and (b) VE (ours). We depict the Venn diagram of which show the relationships between the H(X), prototype information I(X; Y ), contextual information H(X|Y ) and I(Xˆ;…
Figure 7
Figure 7. Figure 7: Distilled Image Visualization: ImageNette with IPC 10. [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Distilled Image Visualization: ImageWoof with IPC 10. [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 56 canonical work pages

  1. [1]

    Emmanuel Brempong, Simon Kornblith, Ting Chen, Niki Parmar, Matthias Minderer, and Mohammad. Norouzi. Denoising pretraining for semantic segmentation. In Proc. CVPR, pp.\ 4175–4186, 2022

    work page 2022

  2. [2]

    Efros, and Jun-Yan Zhu

    George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proc. CVPR, pp.\ 10718--10727, 2022 a

    work page 2022

  3. [3]

    Dataset distillation by matching training trajectories

    George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proc. CVPR, pp.\ 4750--4759, 2022 b

    work page 2022

  4. [4]

    Generalizing dataset distillation via deep generative prior

    George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Generalizing dataset distillation via deep generative prior. In Proc. CVPR, pp.\ 3739--3748, 2023

    work page 2023

  5. [5]

    Influence-guided diffusion for dataset distillation

    Mingyang Chen, Jiawei Du, Bo Huang, Yi Wang, Xiaobo Zhang, and Wei Wang. Influence-guided diffusion for dataset distillation. In Proc. ICLR, 2025. URL https://openreview.net/forum?id=0whx8MhysK

    work page 2025

  6. [6]

    Cover and Joy A

    Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006. ISBN 0471241954

    2006

  7. [7]

    Scaling up dataset distillation to imagenet-1k with constant memory

    Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet-1k with constant memory. In Proc. ICML, pp.\ 6565--6590, 2023

    work page 2023

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. CVPR, pp.\ 248--255, 2009

    work page 2009

Show all 69 references
  1. [9]

    Exploiting inter-sample and inter-feature relations in dataset distillation

    Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang, Hongguang Zhang, Kuihua Huang, Jing Huo, and Yang Gao. Exploiting inter-sample and inter-feature relations in dataset distillation. In Proc. CVPR, pp.\ 17057--17066, 2024

    2024

  2. [10]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Proc. NeurIPS, pp.\ 8780--8794, 2021

    2021

  3. [11]

    Calibrated dataset condensation for faster hyperparameter search

    Mucong Ding, Yuancheng Xu, Tahseen Rabbani, Xiaoyu Liu, Brian Gravelle, Teresa Ranadive, Tai-Ching Tuan, and Furong Huang. Calibrated dataset condensation for faster hyperparameter search. arXiv preprint arXiv:2405.17535, pp.\ 1--37, 2024

    2024 arXiv

  4. [12]

    Incorporating second-order functional knowledge for better option pricing

    Charles Dugas, Yoshua Bengio, Fran c ois B \'e lisle, Claude Nadeau, and Ren \'e Garcia. Incorporating second-order functional knowledge for better option pricing. In Proc. NeurIPS, 2000

    2000

  5. [13]

    The role of entropy and reconstruction in multi-view self-supervised learning

    Borja Rodr guez G \'a lvez, Arno Blaas, Pau Rodr \' guez, Adam Golinski, Xavier Suau, Jason Ramapuram, Dan Busbridge, and Luca Zappella. The role of entropy and reconstruction in multi-view self-supervised learning. In Proc. ICML, pp.\ 29143--29160, 2023

    2023

  6. [14]

    Efficient dataset distillation via minimax diffusion

    Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In Proc. CVPR, pp.\ 15793--15803, 2024 a

    2024

  7. [15]

    Efficient dataset distillation via minimax diffusion

    Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In Proc. CVPR, pp.\ 15793--15803, 2024 b

    2024

  8. [16]

    Towards lossless dataset distillation via difficulty-aligned trajectory matching

    Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. In Proc. ICLR, 2024

    2024

  9. [17]

    R. V. L. Hartley. Transmission of information. The Bell System Technical Journal, 7 0 (3): 0 535--563, 1928. doi:10.1002/j.1538-7305.1928.tb01236.x

    1928

  10. [18]

    William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank. Wood. Flexible diffusion modeling of long videos. arXiv preprint arXiv:2205.11495, pp.\ 1--23, 2022

    2022 arXiv

  11. [19]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proc. ICCV, pp.\ 1026--1034, 2015

    2015

  12. [20]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proc. CVPR, pp.\ 9729--9738, 2020

    2020

  13. [21]

    Hinton, Oriol Vinyals, and Jeffrey Dean

    Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015. URL https://api.semanticscholar.org/CorpusID:7200347

    2015 arXiv

  14. [22]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proc. NeurIPS, pp.\ 6840--6851, 2020

    2020

  15. [23]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  16. [24]

    Dataset condensation via efficient synthetic-data parameterization

    Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In Proc. ICML, pp.\ 11102--11118, 2022 a

    2022

  17. [25]

    Dataset condensation via efficient synthetic-data parameterization

    Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In Proc. ICML, pp.\ 11102--11118, 2022 b

    2022

  18. [26]

    Tiny imagenet visual recognition challenge

    Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7 0 (7): 0 3, 2015

    2015

  19. [27]

    Gradient-based learning applied to document recognition

    Yann LeCun, L \'e on Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 0 (11): 0 2278--2324, 1998

    1998

  20. [28]

    Deep learning

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 0 (7553): 0 436--444, 2015

    2015

  21. [29]

    Soft-label anonymous gastric x-ray image distillation

    Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Soft-label anonymous gastric x-ray image distillation. In Proc. ICIP, pp.\ 305--309, 2020

    2020

  22. [30]

    Compressed gastric image generation based on soft-label dataset distillation for medical data sharing

    Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Compressed gastric image generation based on soft-label dataset distillation for medical data sharing. Computer Methods and Programs in Biomedicine, 227: 0 107189, 2022 a

    2022

  23. [31]

    Awesome dataset distillation

    Guang Li, Bo Zhao, and Tongzhou Wang. Awesome dataset distillation. https://github.com/Guang000/Awesome-Dataset-Distillation, 2022 b

    2022

  24. [32]

    Dataset distillation for medical dataset sharing

    Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Dataset distillation for medical dataset sharing. In Proc. AAAI Workshop, pp.\ 1--6, 2023

    2023

  25. [33]

    Importance-aware adaptive dataset distillation

    Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Importance-aware adaptive dataset distillation. Neural Networks, 2024 a

    2024

  26. [34]

    Generative dataset distillation: Balancing global structure and local details

    Longzhen Li, Guang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa, and Miki Haseyama. Generative dataset distillation: Balancing global structure and local details. In Proc. CVPR Workshop, pp.\ 7664--7671, 2024 b

    2024

  27. [35]

    Generative dataset distillation based on self-knowledge distillation

    Longzhen Li, Guang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa, and Miki Haseyama. Generative dataset distillation based on self-knowledge distillation. In Proc. ICASSP, 2025 a

    2025

  28. [36]

    Diversity-driven generative dataset distillation based on diffusion model with self-adaptive memory

    Mingzhuo Li, Guang Li, Jiafeng Mao, Takahiro Ogawa, and Miki Haseyama. Diversity-driven generative dataset distillation based on diffusion model with self-adaptive memory. In Proc. ICIP, 2025 b

    2025

  29. [37]

    Hyperbolic dataset distillation

    Wenyuan Li, Guang Li, Keisuke Maeda, Takahiro Ogawa, and Miki Haseyama. Hyperbolic dataset distillation. arXiv preprint arXiv:2505.24623, 2025 c

    2025

  30. [38]

    Dataset distillation by automatic training trajectories

    Dai Liu, Jindong Gu, Hu Cao, Carsten Trinitis, and Martin Schulz. Dataset distillation by automatic training trajectories. In Proc. ECCV, pp.\ 334--351, 2024

    2024

  31. [39]

    Elucidating the design space of classifier-guided diffusion generation

    Jiajun Ma, Tianyang Hu, Wenjia Wang, and Jiacheng Sun. Elucidating the design space of classifier-guided diffusion generation. In Proc. ICLR, 2024. URL https://openreview.net/forum?id=9DXXMXnIGm

    2024

  32. [40]

    Dataset meta-learning from kernel ridge-regression

    Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. In Proc. ICLR, 2021 a

    2021

  33. [41]

    Dataset distillation with infinitely wide convolutional networks

    Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In Proc. NeurIPS, pp.\ 5186--5198, 2021 b

    2021

  34. [42]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  35. [43]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. High-resolution image synthesis with latent diffusion models. In Proc. CVPR, pp.\ 10684--10695, 2022

    2022

  36. [44]

    Data distillation: A survey

    Noveen Sachdeva and Julian McAuley. Data distillation: A survey. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=lmXMXP74TO. Survey Certification

    2023

  37. [45]

    Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad. Norouzi. Palette: Image-to-image diffusion models. In Proc. ACM SIGGRAPH, pp.\ 1–10, 2022

    2022

  38. [46]

    Fleet, and Mohammad Norouzi

    Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 0 (4): 0 4713--4726, 2023

    2023

  39. [47]

    Datadam: Efficient dataset distillation with attention matching

    Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z Liu, Yuri A Lawryshyn, and Konstantinos N Plataniotis. Datadam: Efficient dataset distillation with attention matching. In Proc. ICCV, pp.\ 17097--17107, 2023

    2023

  40. [48]

    Salamah, Kaixiang Zheng, Linfeng Ye, and En-Hui Yang

    Ahmed H. Salamah, Kaixiang Zheng, Linfeng Ye, and En-Hui Yang. Jpeg compliant compression for dnn vision. IEEE Journal on Selected Areas in Information Theory, 5: 0 520--533, 2024

    2024

  41. [49]

    On the information bottleneck theory of deep learning

    Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. In Proc. ICLR, 2018. URL https://openreview.net/forum?id=ry_WPG-A-

    2018

  42. [50]

    C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27 0 (3): 0 379--423, 1948. doi:10.1002/j.1538-7305.1948.tb01338.x

    1948

  43. [51]

    D4M : Dataset distillation via disentangled diffusion model

    Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. D4M : Dataset distillation via disentangled diffusion model. In Proc. CVPR, pp.\ 5809--5818, 2024 a

    2024

  44. [52]

    Generative dataset distillation based on diffusion model

    Duo Su, Junjie Hou, Guang Li, Ren Togo, Rui Song, Takahiro Ogawa, and Miki Haseyama. Generative dataset distillation based on diffusion model. In Proc. ECCV Workshop, pp.\ 1--12, 2024 b

    2024

  45. [53]

    On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm

    Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proc. CVPR, pp.\ 9390--9399, 2024

    2024

  46. [54]

    The information bottleneck method

    Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000

    2000 arXiv

  47. [55]

    Cafe: Learning to condense dataset by aligning features

    Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proc. CVPR, pp.\ 12196--12205, 2022

    2022

  48. [56]

    Dataset distillation

    Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018

    2018 arXiv

  49. [57]

    Conditional mutual information constrained deep learning: Framework and preliminary results

    En-Hui Yang, Shayan Mohajer Hamidi, Linfeng Ye, Renhao Tan, and Beverly Yang. Conditional mutual information constrained deep learning: Framework and preliminary results. In Proc. ISIT, pp.\ 569--574, 2024

    2024

  50. [58]

    Conditional mutual information constrained deep learning for classification

    En-Hui Yang, Shayan Mohajer Hamidi, Linfeng Ye, Renhao Tan, and Beverly Yang. Conditional mutual information constrained deep learning for classification. IEEE Transactions on Neural Networks and Learning Systems, pp.\ 1--13, 2025

    2025

  51. [59]

    An efficient dataset condensation plugin and its application to continual learning

    Enneng Yang, Li Shen, Zhenyi Wang, Tongliang Liu, and Guibing Guo. An efficient dataset condensation plugin and its application to continual learning. In Proc. NeurIPS, pp.\ 67625--67642, 2023 a

    2023

  52. [60]

    Diffusion models: A comprehensive survey of methods and applications

    Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56 0 (4): 0 1--39, 2023 b

    2023

  53. [61]

    Bayes conditional distribution estimation for knowledge distillation based on conditional mutual information

    Linfeng Ye, Shayan Mohajer Hamidi, Renhao Tan, and En-Hui YANG. Bayes conditional distribution estimation for knowledge distillation based on conditional mutual information. In Proc. ICLR, 2024. URL https://openreview.net/forum?id=yV6wwEbtkR

    2024

  54. [62]

    Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective

    Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. In Proc. NeurIPS, 2023. URL https://openreview.net/forum?id=5Fgdk3hZpb

    2023

  55. [63]

    A comprehensive survey to dataset distillation

    Ruonan Yu, Songhua Liu, and Xinchao Wang. A comprehensive survey to dataset distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46 0 (1): 0 150--170, 2023

    2023

  56. [64]

    Cutmix: Regularization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proc. ICCV, pp.\ 6022--6031, 2019

    2019

  57. [65]

    Dataset condensation with gradient matching

    Bo Zhao and Hakan Bilen. Dataset condensation with gradient matching. In Proc. ICLR, pp.\ 1--20, 2021

    2021

  58. [66]

    Synthesizing informative training samples with gan

    Bo Zhao and Hakan Bilen. Synthesizing informative training samples with gan. In Proc. NeurIPS Workshop, pp.\ 1--13, 2022

    2022

  59. [67]

    Dataset condensation with gradient matching

    Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In Proc. ICLR, 2021. URL https://openreview.net/forum?id=mSAKhLYLSsl

    2021

  60. [68]

    Improved distribution matching for dataset condensation

    Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proc. CVPR, pp.\ 7856--7865, 2023

    2023

  61. [69]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.