Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

CaO$_2$: Rectifying Inconsistencies in Diffusion-Based Dataset Distillation

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CaO2 is a two-stage diffusion distillation method that claims a 2.3% average accuracy gain on ImageNet-family datasets by removing Objective and Condition Inconsistencies.

desk verdict CaO2 is a genuinely useful two-stage heuristic for diffusion-based dataset distillation, with solid empirical gains, but the conceptual framing overreaches and the reported margins should be read with a cautious eye. read the letter →

arxiv 2506.22637 v2 pith:UB77K2KL submitted 2025-06-27 cs.CV

Haoxuan Wang , Zhenghao Zhao , Junyi Wu , Yuzhang Shang , Gaowen Liu , Yan Yan This is my paper
classification cs.CV
keywords datasetdistillationdiffusionmodelsconditionallikelihoodsampleselectionlatentoptimizationimageclassificationNetclassifier-freeguidance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CaO2 is a two-stage, training-free refinement method for diffusion-based dataset distillation. It claims that current diffusion-based distillation methods fail at evaluation time for two identifiable reasons — Objective Inconsistency and Condition Inconsistency — and that fixing both improves downstream classification on compact synthetic datasets. Concretely, the paper reports an average 2.3% accuracy gain over the best baselines on ImageNet and its subsets, with gains on ImageNette reaching 4.3% on average. A reader should care because dataset distillation is how large image collections are compressed for efficient training; making the distilled images carry their labels more faithfully raises the ceiling of every downstream model trained on them.

What carries the argument

The carrying mechanism is the two-stage refinement loop. Stage 1 generates an image pool of size $m \times \text{IPC}$ per class from a frozen conditional diffusion model and keeps only samples a lightweight pre-trained classifier assigns to the conditioned class, choosing the most confident correct samples at low IPC and the least confident correct samples at high IPC. Stage 2 fixes the diffusion model and optimizes each selected latent $x$ against the objective $\min_x \mathbb{E}_{t,\varepsilon}[\|\epsilon_\theta(x_t, \hat{c}, t) - \varepsilon\|_2^2 + \lambda\|\epsilon_\theta(x_t, \hat{c}, t) - \varepsilon\|_\infty]$, where $x_t = \sqrt{\alpha_t} x + \sqrt{1-\alpha_t}\varepsilon$ is a mildly perturbed latent and the condition $\hat{c}$ is chosen as the true class for easy tasks or the unconditional guidance label for hard tasks. This design avoids training or finetuning the generative backbone, which is why it can be transplanted onto DiT, Minimax Diffusion, and masked autoregressive backbones.

What would settle it

Two observations would settle the claim: measuring each selected latent's conditional likelihood or diffusion-classifier accuracy before and after Stage 2 — if it does not rise while downstream classifier accuracy rises, the stated mechanism is not driving the gain; or replacing Stage 2 with equal-magnitude random latent perturbations and checking whether the accuracy gain disappears.

Watch

Extended reading notes

Core claim

The paper's central claim is that current diffusion-based dataset distillation overlooks the evaluation objective, creating two mismatches: Objective Inconsistency, where generated images can be classified into a different class than their conditioning label, and Condition Inconsistency, where an empirical diffusion model assigns nonzero likelihood to a sample under conditions other than its own. The proposed remedy is a two-stage pipeline: first, generate an image pool and select, per class, the samples a lightweight pre-trained classifier confidently assigns to the conditioned class; second, keep the diffusion model frozen and optimize each selected image latent to minimize the denoising loss under a chosen condition, thereby improving the sample's conditional likelihood. The paper establishes this largely through experiments on ImageNet, ImageNet-100, ImageWoof, and ImageNette across ResNet-18/50/101, EfficientNet-B0, and MobileNet-V2, where CaO2 outperforms the best baselines by an average of 2.3% accuracy.

Load-bearing premise

The load-bearing premise is that minimizing the denoising loss of a latent under a fixed condition increases that sample's conditional likelihood, and that this makes the sample a better training example for a downstream classifier; if that link fails, Stage 2 adds nothing beyond Stage 1.

Editorial extensions

If this is right

  • Distilled datasets produced by CaO2 train classifiers to higher accuracy than datasets produced by Minimax Diffusion, D4M, RDED, and SRe2L across ImageNet and its subsets, at several images-per-class budgets.
  • The method improves both hard-label and soft-label evaluation settings, reporting the better of the two per configuration rather than relying on a single evaluation paradigm.
  • Because the generative backbone is frozen, CaO2 acts as a plug-and-play post-processing step: applying its two stages to a Minimax Diffusion or MAR generation pipeline yields consistent accuracy gains.
  • The easy-versus-hard condition choice suggests that task difficulty, not just class identity, should determine how synthetic samples are selected and refined during distillation.
  • The method is computationally light enough to run on a single GPU, making the reported gains available without large-scale optimization or additional model training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The latent-refinement step behaves like a lightweight label-alignment pass and should in principle transfer to any distilled dataset whose images were produced by a denoiser, not just to the two generative backbones tested.
  • Editorial inference: The task-dependent condition switch in Eq. (5) suggests a finer-grained recipe — choosing conditions or noise budgets per class or per difficulty cluster — that the paper does not test.
  • Editorial inference: Because the reported gains are often largest where samples are scarce, the 2.3% headline may understate the method's value at extreme compression ratios such as IPC=1.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes CaO2, a two-stage diffusion-based dataset distillation method. Stage 1 generates a pool of class-conditioned images and selects samples using a lightweight classifier's confidence, with the selection criterion (easiest vs hardest correct samples) chosen according to task difficulty. Stage 2 optimizes selected latents to minimize a denoising loss under a chosen condition, with an L-infinity regularizer, intended to increase the conditional likelihood of the sample. The authors frame the work around two 'inconsistencies' in diffusion-based distillation (Objective Inconsistency and Condition Inconsistency) and report state-of-the-art accuracy on ImageNet and its subsets, with an average improvement of 2.3% over baselines. The method is also integrated with a masked autoregressive backbone (MAR) and with Minimax Diffusion.

Significance. Strengths: the paper delivers extensive experiments across multiple datasets, architectures, and IPC settings; the gains over baselines are consistent though often modest; code is released; the method is simple, efficient, and works as a plug-and-play module on different backbones. The ablations cover pool size, selection strategy, condition choice, regularization form, stage ordering, and noise level. These are real empirical contributions. However, the conceptual claims about 'rectifying inconsistencies' are not rigorously established. Definition 2 of Condition Inconsistency is vacuous for continuous densities, and the Stage 2 likelihood-maximization mechanism is asserted rather than verified. The evaluation protocol that reports the higher of hard-label and soft-label accuracy for every method is non-standard and weakens the headline SOTA claim. The empirical method is likely useful, but the paper needs substantial revision to support its conceptual framing and to make the comparisons interpretable.

major comments (5)
  1. [Sec. 4.1 and Tabs. 1-2] The evaluation protocol 'report the best result for each method' between hard-label and soft-label paradigms is not a fixed benchmark. Because each reported cell is a maximum over two evaluation protocols, this can systematically inflate accuracy, and the magnitude of the inflation may differ across methods. The average +2.3% claim therefore is not interpretable without showing both hard-label and soft-label numbers separately for every method and cell. Please report both paradigms separately (or pre-register one protocol), and analyze how the ranking changes under each.
  2. [Sec. 3.3, Definition 2] Definition 2 states that Condition Inconsistency holds when for every i there exists j != i with p_theta(x_i0 | c_j) > 0. For any continuous density with overlapping class-conditional supports, this condition is true even for a perfectly trained diffusion model, since the likelihood of a sample under a wrong class label is generically nonzero. The definition therefore does not characterize a deficiency of empirical diffusion models or a 'mismatch' between condition and image. A quantitative definition (e.g., log-likelihood ratio p_theta(x|c_i)/p_theta(x|c_j), or the Bayes posterior p_theta(c_i|x)) is needed, or the paper should reframe this as 'suboptimal conditional likelihood' rather than an inconsistency.
  3. [Sec. 3.3, Eq. (4)] The mechanism claim underlying Stage 2 is not established. Minimizing the denoising loss with respect to x increases the ELBO lower bound in Eq. (2) for the current model, but it does not guarantee an increase in p_theta(x0|c) and, more importantly, no argument links the ELBO to the risk of a downstream discriminative classifier. The ablations in Tabs. 3-5 show accuracy gains, but they do not include a perturbation-matched control where x is randomly perturbed within the same norm budget. Such a control is needed to rule out the alternative explanation that the gains come from a small regularizing or data-augmentation effect rather than from likelihood alignment. Please add this control and, if possible, report a likelihood-based evaluation of the optimized samples.
  4. [Sec. 3.3, text vs Eq. (4)] The text introduces a constraint ||x - x||_inf <= eta, but Eq. (4) uses an L_infinity norm of the denoising error as a penalty term; no projection or clipping step is described. As written, the optimization does not enforce the stated perturbation bound. This is a mismatch between the stated design and the implemented objective, and it affects reproducibility and the interpretation of the 'imperceptible changes' in Fig. 4. Please either enforce the hard constraint explicitly (with projection) or clearly state that the L_infinity error penalty is a proxy and remove the unused eta notation.
  5. [Sec. 3.4 and Sec. 4.3] The choice of condition c_hat (true vs null) and the selection criterion (most vs least confident) are made per dataset based on validation accuracy, and the paper's ablations show this choice matters. This is effectively a task-level tuning knob. The paper should state explicitly how this choice is made in a new task without peeking at test accuracy, for example by a validation-set rule or a fixed default, and should report sensitivity of the main results to this choice.
minor comments (6)
  1. [Sec. 3.4] 'we use the size of the distilled dataset as an evidence' should read 'as evidence'.
  2. [Appendix Sec. 9] 'We induce that' should be 'We infer that'.
  3. [Appendix Tab. 7] Ours(w/o SS) is below DiT-IGD on ImageWoof IPC=10 (65.0 vs 67.7); the sentence claiming 'standalone effectiveness' of the single-stage CaO2 should be qualified to reflect this exception.
  4. [Algorithm 1, line 10] 'Update x using Eq. (4), Eq. (5)' is ambiguous; Eq. (5) defines the condition choice rather than an update rule. Please rephrase.
  5. [References] References [20] and [21] appear to be the same paper ('Prioritize Alignment in Dataset Distillation') and should be merged.
  6. [Sec. 4.5, Fig. 4] The statement that latent optimization introduces 'imperceptible changes' is qualitative; please include a quantitative distance metric (e.g., mean absolute difference or PSNR) to support this claim.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central claim is an empirical accuracy comparison against external baselines; the two 'inconsistencies' are heuristic motivations, not derived predictions.

full rationale

The paper's central result—state-of-the-art distillation accuracy—is measured empirically against external baselines (SRe2L, Minimax Diffusion, D4M, RDED) on held-out ImageNet validation data, not derived from a fitted parameter. Stage 1 (sample selection) uses a pretrained classifier to filter generated images; this is a legitimate data-selection heuristic, and the reported accuracy is obtained by training new models from scratch on the selected images and testing on real validation images, so the evaluation does not reduce to the selection criterion by construction. Stage 2 (latent optimization) minimizes the diffusion denoising loss w.r.t. the latent (Eq. 4); while the paper's interpretation that this maximizes conditional likelihood is not rigorously justified (a lower-bound increase need not increase the exact likelihood, and Definition 2's Condition Inconsistency is trivially satisfied for continuous densities), this is a gap between motivation and mechanism, not a circular reduction. The self-citations ([22], [46], [47]) are minor and non-load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. The use of validation accuracy to choose the condition in Eq. (5) is a hyperparameter-selection concern, not circularity, because the final numbers are still empirical measurements rather than algebraic consequences of the input. Overall, the paper's derivation chain is not circular; it is a heuristic framework validated by external benchmarks.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities; the two 'inconsistencies' are conceptual framings rather than quantities. The free parameters are standard ML hyperparameters, each ablated and tuned on validation accuracy. The axioms are domain assumptions about the reliability of diffusion generation and the transfer from generative likelihood to discriminative utility.

free parameters (8)
  • pool scaling factor m = 2 or 4
    Multiplier for the generated image pool size per class; chosen via ablation (Fig. 3b) based on validation accuracy.
  • regularization strength lambda = 10
    Coefficient for the L_infinity term in Eq. (4); tuned via ablation (Tab. 4).
  • latent optimization learning rate = 0.0006
    Adam learning rate for Stage 2; fixed across experiments.
  • optimization iterations = 100
    Number of gradient steps per image in Stage 2; fixed across experiments.
  • max noise time step T_hat = T/8 (ImageWoof IPC=10), T/2 (ImageNette IPC=10)
    Noise perturbation level in Eq. (3); selected per dataset via ablation (Tab. 10).
  • condition choice c_hat = true class label or null label depending on task difficulty
    Eq. (5); selected using validation accuracy for each dataset (Sec. 3.4).
  • selection confidence mode = most confident (low IPC) or least confident (high IPC)
    Chosen based on IPC following the curriculum insight of prior work and validation accuracy (Fig. 3c).
  • MAR-specific hyperparameters = max masking ratio 0.25, learning rate 0.0001, 100 steps
    Hyperparameters for the MAR backbone extension (Sec. 3.5 and Sec. 4.6).
assumptions (5)
  • domain assumption A pre-trained DiT trained on ImageNet provides accurate conditional density estimates p_theta(x0|c) and generates representative image pools.
    The entire pipeline depends on DiT generations; stated in Sec. 3.1 and used in Sec. 4.1.
  • domain assumption Minimizing the denoising loss with respect to a latent (Eq. 4) increases the conditional likelihood of that latent and improves its utility for downstream discriminative training.
    Stage 2's mechanism; the paper provides ablations but no formal derivation linking Eq. (4) to classification accuracy.
  • domain assumption The confidence of a pre-trained ResNet-18 classifier on a generated image is a valid proxy for the sample's value under the evaluation objective.
    Stage 1 selection criterion; motivated by diffusion-classifier results in [18] but not proven optimal for distillation.
  • domain assumption Reporting the higher of hard-label and soft-label evaluation accuracy is a fair comparison protocol.
    Sec. 4.1; this choice may inflate absolute accuracies but is applied uniformly to baselines.
  • ad hoc to paper Task difficulty can be estimated by validation accuracy, and the optimal condition (true vs null) can be selected accordingly.
    Sec. 3.4; this introduces a dataset-specific tunable choice that is not derived from theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CaO$_2$: Rectifying Inconsistencies in Diffusion-Based Dataset Distillation." pith.science (2026). https://pith.science/paper/UB77K2KL

@misc{pith2026250622637,
  author       = {Pith},
  title        = {Pith review of: CaO$_2$: Rectifying Inconsistencies in Diffusion-Based Dataset Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UB77K2KL}},
  note         = {Machine review of arXiv:2506.22637}
}
abstract

The recent introduction of diffusion models in dataset distillation has shown promising potential in creating compact surrogate datasets for large, high-resolution target datasets, offering improved efficiency and performance over traditional bi-level/uni-level optimization methods. However, current diffusion-based dataset distillation approaches overlook the evaluation process and exhibit two critical inconsistencies in the distillation process: (1) Objective Inconsistency, where the distillation process diverges from the evaluation objective, and (2) Condition Inconsistency, leading to mismatches between generated images and their corresponding conditions. To resolve these issues, we introduce Condition-aware Optimization with Objective-guided Sampling (CaO$_2$), a two-stage diffusion-based framework that aligns the distillation process with the evaluation objective. The first stage employs a probability-informed sample selection pipeline, while the second stage refines the corresponding latent representations to improve conditional likelihood. CaO$_2$ achieves state-of-the-art performance on ImageNet and its subsets, surpassing the best-performing baselines by an average of 2.3% accuracy.

Figures

Figures reproduced from arXiv: 2506.22637 by the authors.

Figure 1
Figure 1. Comparison of validation accuracy and distillation time for different methods under different IPCs on Image￾Woof. Our two-stage method is more efficient and can obtain bet￾ter performance compared with other SOTA approaches. the target dataset and a student model trained on the syn￾thetic dataset [10, 24, 36, 41]. However, these matching￾based methods either struggle to scale up to larger, higher￾resolution datasets… view at source ↗
Figure 2
Figure 2. Comparison and illustration of previous diffusion-based methods and our approach. (a) Current diffusion-based methods facilitate efficient distillation using a conditional diffusion model G. However, their proposed distillation processes overlook the evaluation objective, resulting in Objective Inconsistency (OI) between distillation and evaluation, as well as Condition Inconsistency (CI) between the input condition… view at source ↗
Figure 3
Figure 3. Ablation studies on different components. (a) illustrates the cross-architecture performance for selector and evaluation models, showing that a lightweight ResNet-18 generally leads to better performance in most cases. (b) examines the effect of the scaling factor, showing that a larger pool size is not always optimal. As a result, we adopt a pool size of 2×IPC or 4×IPC. (c) compares different selection strategies, … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Visualization of distilled images. We compare the dis￾tilled images from Minimax Diffusion with the two stages of our method, also highlighting the differences introduced by latent op￾timization, where white color denotes unchanged regions. Interestingly, latent optimi…
Figure 5
Figure 5. Figure 5: Pipeline of our method when applied to the Masked [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: More examples of our distilled images on ImageWoof. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Comparison of using classifier-guidance or not. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Examples of our distilled images on ImageNette. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Examples of our distilled images when using the Mini [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Examples of our distilled images when using MAR as [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hard Labels In! Rethinking the Role of Hard Labels in Mitigating Local Semantic Drift

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A soft-hard-soft training schedule uses hard labels as an intermediate anchor to correct local semantic drift and improves accuracy under 100x-reduced soft-label storage.

Reference graph

Works this paper leans on

55 extracted references · 44 canonical work pages · cited by 1 Pith paper

  1. [1]

    Curriculum learning

    Yoshua Bengio, J ´erˆome Louradour, Ronan Collobert, and Ja- son Weston. Curriculum learning. In International Confer- ence on Machine Learning, 2009. 4

    work page 2009

  2. [2]

    Efros, and Jun-Yan Zhu

    George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 1, 2, 6

    work page 2022

  3. [3]

    Efros, and Jun-Yan Zhu

    George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Generalizing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 1, 2

    work page 2023

  4. [4]

    Robust clas- sification via a single diffusion model

    Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang, Chengqi Duan, Hang Su, and Jun Zhu. Robust clas- sification via a single diffusion model. arXiv preprint arXiv:2305.15241, 2023. 3, 4, 8

    arXiv 2023

  5. [5]

    Influence-guided diffusion for dataset distillation

    Mingyang Chen, Jiawei Du, Bo Huang, Yi Wang, Xiaobo Zhang, and Wei Wang. Influence-guided diffusion for dataset distillation. In The Thirteenth International Conference on Learning Representations, 2025. 2, 6

    work page 2025

  6. [6]

    Scaling up dataset distillation to imagenet-1k with constant memory

    Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet-1k with constant memory. In International Conference on Machine Learning , pages 6565–6590. PMLR, 2023. 2, 6

    work page 2023

  7. [7]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 5

    2009

  8. [8]

    Minimizing the accumulated trajectory error to improve dataset distillation

    Jiawei Du, Yidi Jiang, Vincent YF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3749–3758, 2023. 2

    work page 2023

Show all 55 references
  1. [9]

    Efficient dataset distillation via minimax diffusion

    Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Hao- nan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 1, 2, 5, 6, 7, 8

    2024

  2. [10]

    Towards lossless dataset distillation via difficulty-aligned trajectory matching

    Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. In The Twelfth In- ternational Conference on Learning Representations , 2024. 1, 2, 4, 6

    2024

  3. [11]

    Deep residual learning for image recognition, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. 6

    2015

  4. [12]

    You only condense once: Two rules for pruning condensed datasets,

    Yang He, Lingao Xiao, and Joey Tianyi Zhou. You only condense once: Two rules for pruning condensed datasets,

  5. [13]

    Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco An- dreetto, and Hartwig Adam

    Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco An- dreetto, and Hartwig Adam. Mobilenets: Efficient convolu- tional neural networks for mobile vision applications, 2017. 6

    2017

  6. [14]

    Imagenette: A smaller subset of 10 easily classified classes from imagenet, 2019

    Jeremy Howard. Imagenette: A smaller subset of 10 easily classified classes from imagenet, 2019. 5

    2019

  7. [15]

    Imagewoof: a subset of 10 classes from imagenet that aren’t so easy to classify, 2019

    Jeremy Howard. Imagewoof: a subset of 10 classes from imagenet that aren’t so easy to classify, 2019. 5

    2019

  8. [16]

    Dataset condensation via efficient synthetic- data parameterization

    Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic- data parameterization. In International Conference on Ma- chine Learning (ICML), 2022. 2

    2022

  9. [17]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. 6

    2017

  10. [18]

    Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak

    Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , pages 2206–2217, 2023. 3

    2023

  11. [19]

    Autoregressive image generation without vec- tor quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vec- tor quantization. arXiv preprint arXiv:2406.11838, 2024. 2, 5, 8, 1

    2024 arXiv

  12. [20]

    Prioritize alignment in dataset distillation, 2024

    Zekai Li, Ziyao Guo, Wangbo Zhao, Tianle Zhang, Zhi-Qi Cheng, Samir Khaki, Kaipeng Zhang, Ahmad Sajedi, Kon- stantinos N Plataniotis, Kai Wang, and Yang You. Prioritize alignment in dataset distillation, 2024. 4

    2024

  13. [21]

    Prioritize alignment in dataset distillation, 2024

    Zekai Li, Ziyao Guo, Wangbo Zhao, Tianle Zhang, Zhi-Qi Cheng, Samir Khaki, Kaipeng Zhang, Ahmad Sajedi, Kon- stantinos N Plataniotis, Kai Wang, and Yang You. Prioritize alignment in dataset distillation, 2024. 6

    2024

  14. [22]

    Plataniotis, Bo Zhao, Zhangyang Wang, Yang You, and Kai Wang

    Zekai Li, Xinhao Zhong, Zhiyuan Liang, Yuhao Zhou, Mingjia Shi, Ziqiao Wang, Wangbo Zhao, Xuanlei Zhao, Haonan Wang, Ziheng Qin, Dai Liu, Kaipeng Zhang, Tianyi Zhou, Zheng Zhu, Kun Wang, Guang Li, Junhao Zhang, Ji- awei Liu, Yiran Huang, Lingjuan Lyu, Jiancheng Lv, Yaochu Jin,...

    2024

  15. [23]

    Darts: Differentiable architecture search, 2019

    Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, 2019. 1

    2019

  16. [24]

    Dream: Efficient dataset distillation by rep- resentative matching, 2023

    Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by rep- resentative matching, 2023. 1, 2

    2023

  17. [25]

    Reducing catas- trophic forgetting with learning on synthetic data, 2020

    Wojciech Masarczyk and Ivona Tautkute. Reducing catas- trophic forgetting with learning on synthetic data, 2020. 1

    2020

  18. [26]

    Moser, Federico Raue, Sebastian Palacio, Stanislav Frolov, and Andreas Dengel

    Brian B. Moser, Federico Raue, Sebastian Palacio, Stanislav Frolov, and Andreas Dengel. Latent dataset distillation with diffusion models, 2024. 1

    2024

  19. [27]

    Deep learning on a data diet: Finding important exam- ples early in training, 2023

    Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziu- gaite. Deep learning on a data diet: Finding important exam- ples early in training, 2023. 7 9

    2023

  20. [28]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022. 1, 6

    2022 arXiv

  21. [29]

    High-resolution image syn- thesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models, 2021. 1

    2021

  22. [30]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge,

  23. [31]

    Generalized large-scale data condensa- tion via various backbone and statistical matching, 2024

    Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensa- tion via various backbone and statistical matching, 2024. 2, 6

    2024

  24. [32]

    Elucidating the design space of dataset condensation,

    Shitong Shao, Zikai Zhou, Huanran Chen, and Zhiqiang Shen. Elucidating the design space of dataset condensation,

  25. [33]

    Dˆ4m: Dataset distillation via disentangled diffu- sion model

    Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. Dˆ4m: Dataset distillation via disentangled diffu- sion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5809–5818, 2024. 1, 2, 6

    2024

  26. [34]

    On the diversity and realism of distilled dataset: An efficient dataset distilla- tion paradigm

    Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distilla- tion paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 5, 6, 7, 2

    2024

  27. [35]

    Mingxing Tan and Quoc V . Le. Efficientnet: Rethinking model scaling for convolutional neural networks, 2020. 6

    2020

  28. [36]

    Cafe: Learning to condense dataset by align- ing features

    Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by align- ing features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12196– ...

    2022

  29. [37]

    Dataset dis- tillation with neural characteristic function: A minmax per- spective, 2025

    Shaobo Wang, Yicun Yang, Zhiyuan Liu, Chenghao Sun, Xuming Hu, Conghui He, and Linfeng Zhang. Dataset dis- tillation with neural characteristic function: A minmax per- spective, 2025. 4

    2025

  30. [38]

    Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation, 2020. 1

    2020

  31. [39]

    Distill gold from mas- sive ores: Bi-level data pruning towards efficient dataset dis- tillation, 2024

    Yue Xu, Yong-Lu Li, Kaitong Cui, Ziyu Wang, Cewu Lu, Yu-Wing Tai, and Chi-Keung Tang. Distill gold from mas- sive ores: Bi-level data pruning towards efficient dataset dis- tillation, 2024. 1

    2024

  32. [40]

    Dataset distillation in large data era

    Zeyuan Yin and Zhiqiang Shen. Dataset distillation in large data era. arXiv preprint arXiv:2311.18838, 2023. 2

    2023 arXiv

  33. [41]

    Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective

    Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 1, 2, 6

    2023

  34. [42]

    Dataset dis- tillation: A comprehensive review

    Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset dis- tillation: A comprehensive review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 1

    2023

  35. [43]

    Dataset condensation with dis- tribution matching

    Bo Zhao and Hakan Bilen. Dataset condensation with dis- tribution matching. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision , pages 6514–6523, 2023. 2

    2023

  36. [44]

    Dataset condensation with gradient matching

    Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Con- ference on Learning Representations, 2021. 2

    2021

  37. [45]

    Im- proved distribution matching for dataset condensation

    Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Im- proved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 7856–7865, 2023. 2

    2023

  38. [46]

    Dataset quantization with active learning based adaptive sampling

    Zhenghao Zhao, Yuzhang Shang, Junyi Wu, and Yan Yan. Dataset quantization with active learning based adaptive sampling. arXiv preprint arXiv:2407.07268, 2024. 1

    2024 arXiv

  39. [47]

    Distilling long-tailed datasets, 2024

    Zhenghao Zhao, Haoxuan Wang, Yuzhang Shang, Kai Wang, and Yan Yan. Distilling long-tailed datasets, 2024. 1

    2024

  40. [48]

    Self-supervised dataset distillation: A good compression is all you need, 2024

    Muxin Zhou, Zeyuan Yin, Shitong Shao, and Zhiqiang Shen. Self-supervised dataset distillation: A good compression is all you need, 2024. 2 10 CaO2: Rectifying Inconsistencies in Diffusion-Based Dataset Distillation Supplementary Material The supplementary material is organized...

    2024

  41. [49]

    5 shows the pipeline of how we utilize the MAR [19] backbone for our framework

    Generalizing to MAR Fig. 5 shows the pipeline of how we utilize the MAR [19] backbone for our framework. The process differs from the DiT-based pipeline in two aspects: (1) Instead of perturb- ing the input latent using Gaussian noise w.r.t. random time steps, we perturb the i...

  42. [50]

    Quantitative Comparison with DD Methods IPC 10 50 G-VBSM 31.4±0.5 51.8±0.4 EDC 48.6±0.3 58.0±0.2 DiT-IGD 45.5±0.5 59.8±0.3 Ours 46.1±0.2 60.0±0.0 Table 6

    More Baseline Comparisons 7.1. Quantitative Comparison with DD Methods IPC 10 50 G-VBSM 31.4±0.5 51.8±0.4 EDC 48.6±0.3 58.0±0.2 DiT-IGD 45.5±0.5 59.8±0.3 Ours 46.1±0.2 60.0±0.0 Table 6. Baseline comparison on ImageNet-1K. Woof IPC=10 IPC=50 DiT-IGD 67.7±0.3 81.0±0.7 Ours(w/o S...

  43. [51]

    Though we ablated on the various hyperparameters, only a few need to be tuned

    More Ablations Hyperparameter Recommendations. Though we ablated on the various hyperparameters, only a few need to be tuned. It is recommended to always use L∞ with λ = 10, and pool size of 2/4× IPC. Time Cost Comparison. We provide the quantitative re- sults for Fig. 1 here:...

  44. [52]

    Influence of different evaluation paradigms We compare the popularly used hard-label [9] and soft-label

  45. [53]

    13, using distilled images from Minimax Diffusion as an example

    evaluation metrics in Tab. 13, using distilled images from Minimax Diffusion as an example. From the table, we show that neither of the two approaches can always obtain better performance. Setting IPC=1 WoofIPC=10 IPC=50IPC=1 NetteIPC=10 IPC=50 Hard-label [9]19.9±0.236.2±0.2 5...

    2025

  46. [54]

    Distilled images of ImageWoof and ImageNette

    Additional Visualizations We provide more visualization results here for a compre- hensive analysis of our method. Distilled images of ImageWoof and ImageNette. Fig. 6 and 8 show examples of distilled images under IPC=10 for ImageNette and ImageWoof. Three samples are shown fo...

  47. [55]

    DiTs can only deal with ImageNet classes)

    Limitations and Potential Improvements Although diffusion-based methods demonstrate strong per- formance, their applicability is constrained by the limited conditions these models can handle (e.g. DiTs can only deal with ImageNet classes). Employing text-to-image models such a...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.