Pith. sign in

REVIEW 3 major objections 4 minor 65 references

Dataset Distillation as Data Compression: A Rate-Utility Perspective

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Dataset distillation should be optimized as a compression problem—minimize bits and utility loss together—and doing so produces synthetic datasets up to 170x smaller at comparable or better accuracy.

desk verdict A solid, influential rate-utility framework for dataset distillation whose reported bitrate numbers need validation against actual coding before being taken at face value. read the letter →

arxiv 2507.17221 v1 pith:LTXO2WML submitted 2025-07-23 cs.LG cs.CV

Youneng Bao , Yiping Liu , Zhuo Chen , Yongsheng Liang , Mu Li , Kede Ma This is my paper
classification cs.LGcs.CV
keywords datasetdistillationrate-utilityoptimizationneuraldatacompressionentropycodingbitsperclasslatentcodestrajectorymatchingquantization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dataset distillation replaces a large training set with a small set of synthetic examples, and this paper argues that the right way to build that set is to treat it as a compression problem: minimize storage cost and task-utility loss together, not sequentially. The authors parameterize synthetic images as multiscale quantized latent codes decoded by extremely lightweight per-sample networks, estimate the bitrate of those codes with a context-aware entropy model, and combine that rate with any existing distillation loss in a single Lagrange-multiplier objective. To make methods comparable they introduce bits per class (bpc), which counts latent bits, label bits, and decoder-parameter bits per class. On CIFAR-10, CIFAR-100, and ImageNet-128, the resulting method reports better accuracy than prior work at equal or smaller storage, including up to 170x compression relative to standard trajectory matching at comparable accuracy. If valid, this turns dataset distillation into a rate-utility trade-off problem whose Pareto frontier can be explored with off-the-shelf distillation losses.

What carries the argument

The central mechanism is a hybrid synthetic-dataset parameterization with a differentiable rate estimate. Each synthetic image is produced by upsampling and concatenating latent codes at $L$ scales and passing them through a very small decoder network, so the storage cost splits into latent bits, label bits, and parameter bits. The rate of the quantized latents is modeled as $$r(Z;\Phi)=-\frac{1}{N}\sum_{i}\sum_{l,m}\log_2 P(\bar{z}^{(i)}_{l,m}\mid \bar{c}^{(i)}_{l,m})$$ under a conditional Laplace distribution whose location and scale are predicted by a causal-context entropy network, with an assumed independence across channels and scales. The trade-off is then produced by minimizing $r(S)+\lambda\ell(D;S)$ end-to-end, and the new bits-per-class metric $\mathrm{bpc}(S)=\#\mathrm{bits}(S)/K$ converts the resulting dataset into a single storage number that is comparable across very different distillation designs.

What would settle it

Take any fully trained TM-RUO synthetic dataset at a reported bpc, run a genuine range or arithmetic coder over the quantized latents and quantized network weights using the same causal context model, and compare the actual file size with the sum $r(Z)+r(\Phi)+r(\Psi)$ used in the paper. If the true bitstream is materially larger, each rate-utility curve shifts right and the reported Pareto dominance at tight budgets would need to be re-measured.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that jointly optimizing a differentiable rate term and a differentiable utility term dominates optimizing either alone. Synthetic samples are stored as quantized latents at several scales plus lightweight per-sample decoder and entropy networks, and the full objective is $r(S)+\lambda\ell(D;S)$, with $r(S)=r(Z)+r(Y)+r(\Phi)+r(\Psi)$. Using an autoregressive Laplace entropy model for the latents and a Shannon-entropy bound for hard labels, the method reaches 79.7% on CIFAR-10, 49.2% on CIFAR-100, and 76.5% on ImageNet-Nette at per-class budgets of 246 kB, 53 kB, and 179.7 kB respectively, outperforming the strongest neural-field baseline, which reaches 72.0% on Nette at 192 kB. The rate term is loss-agnostic: plugging gradient matching or distribution matching into the same objective lifts their accuracy from 28.1% to 49.5% and from 25.7% to 59.1% on the ImageNet subsets at 192 kB per class. The paper's framing implies that any future distillation method can be graded by its position on a bpc-accuracy curve rather than by images-per-class alone.

Load-bearing premise

The argument stands on the premise that the entropy model's predicted bit count matches what a real coder would emit: the paper assumes independence across channels and scales in Eq. (13) and never encodes an actual arithmetic-coded bitstream, so every reported bpc is only as good as that Laplace context model.

Editorial extensions

If this is right

  • At equal bit budgets, the method reports higher downstream accuracy than prior distilled datasets on CIFAR-10, CIFAR-100, and all six ImageNet-128 subsets; for example, 76.5% versus 72.0% on Nette at about 180–192 kB per class.
  • The same joint objective improves gradient-matching and distribution-matching distillation as well as trajectory matching, so the rate term acts as a general add-on rather than a loss-specific trick.
  • Storage can be cut by up to 170x relative to vanilla trajectory-matching distillation while keeping accuracy, meaning synthetic datasets of a few hundred kilobytes can train competitive classifiers.
  • The bpc metric makes rate-utility curves the natural comparison tool, exposing that prior images-per-class budgets hid large differences in true storage cost.
  • Distilled datasets optimized with the method transfer across several classifier architectures better than prior baselines on average, with 50.7% versus 45.2% mean accuracy, so the improvements are not tied to the architecture used during distillation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors do not draw is that, because the reported rate comes from a surrogate entropy model rather than a real bitstream, the bpc numbers are optimistic lower bounds; re-encoding with an actual arithmetic or range coder would test how much of the claimed 170x survives in true file sizes.
  • The paper's bit-allocation analysis shows that implicit decoder bits dominate at low budgets and explicit latent bits at high budgets; a natural extension would make the Lagrange multiplier per-class or per-sample so the frontier adapts instead of using one global $\lambda$.
  • The bpc normalization by class count makes it possible to compare methods with very different numbers of synthetic samples, which ipc cannot do; the paper proposes bpc for evaluation but does not speculate on it becoming a standard reporting unit in the field.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reframes dataset distillation as a joint rate-utility optimization problem. Synthetic samples are parameterized as multiscale latent codes decoded by lightweight per-class networks, and the storage cost is measured through a learned context-aware entropy model over quantized latents, plus the costs of labels and decoder/entropy network parameters. A Lagrangian objective trades this rate against an arbitrary distillation loss (trajectory, gradient, or distribution matching). The authors introduce a bits-per-class (bpc) metric to compare methods with different storage designs, and report experiments on CIFAR-10, CIFAR-100, and six 128x128 ImageNet subsets showing improved rate-utility trade-offs over existing methods including TM, FreD, DDiF, and others.

Significance. If the reported storage numbers are faithful, this is a valuable contribution: it connects dataset distillation to the mature toolbox of neural lossy compression, makes the rate term differentiable and jointly optimizable with a plug-and-play utility loss, and proposes a unified storage metric (bpc) that is missing from the field. The experiments are extensive, with means and standard deviations over five trials, and cover multiple losses and architectures. The paper also ships a project URL, aiding reproducibility. However, the central quantitative claim depends on an unvalidated surrogate bitrate: the reported bpc values are cross-entropy estimates under a learned entropy model rather than measured arithmetic-coded bitstream lengths. Since every rate-utility comparison is plotted against bpc, the strength and even the sign of the claimed improvements hinge on how close the surrogate is to a realizable code length.

major comments (3)
  1. [Sec. 3.3, Eq. (13) and Sec. 4.2] The rate r(Z) is computed as a negative log-likelihood under a conditional Laplace model, not as the length of an actual entropy-coded bitstream. The paper never runs an arithmetic coder or any other real entropy coder on the quantized latents, so there is no evidence that the assumed model (independence across channels and scales, raster-scan causal context, discretized Laplace) can be realized at the reported bpc. This is load-bearing because all rate-utility curves (Figs. 1 and 3; Tables A3-A6) use bpc as the x-axis; if the true code length is higher, every TM-RUO point moves right and the claimed Pareto improvements may shrink or disappear. The authors should implement an arithmetic coder that uses exactly the quantized entropy network and context model described in Eqs. (11)-(13), report actual bitstream lengths, and compare them to the surrogate rate. At minimum, they should bound the gap, e.g., by comparing against a standard arithmetic coder with a simple but correct model.
  2. [Sec. 3.3, Eq. (14) and Appendix A2] The post-quantization of the entropy network parameters (step size Qe) is selected by grid search on reconstruction MSE of synthetic images, not on rate fidelity. The rate r(Z) in Eq. (13) is computed with the full-precision entropy network, but the deployed decoder would have to use the quantized weights phi_bar; the quantized network can predict different (mu, sigma) and thus change the cross-entropy. The paper does not show that r(Z) is stable under quantization, nor does it recompute r(Z) with phi_bar when reporting bpc. The authors should either (a) recompute the rate with post-quantized entropy networks during the grid search and select Qe based on the resulting rate-utility trade-off, or (b) empirically demonstrate that the rate estimate changes by a negligible amount. The same concern applies to the decoder quantization step Qd, which is also selected without explicit rate fidelity.
  3. [Footnote 4 and Tables A3-A4] The bpc metric charges raw-pixel baselines at 32 bits per pixel. This is a strong accounting assumption: many dataset distillation pipelines store synthetic images at 8-bit or 16-bit precision in practice, and a baseline like TM (Vanilla) could be quantized to 8-bit (48 kB for a 128x128x3 ipc=1 image) without necessarily sacrificing much accuracy. Because the paper's headline '170x compression' and the Pareto comparisons are computed against 32-bit baselines, the claimed advantage is sensitive to this choice. The authors should justify the 32-bit assumption, or report rate-utility curves under alternative bit-depth assumptions (e.g., 8-bit raw pixels) to show that the qualitative and quantitative conclusions are robust. Without this sensitivity analysis, the rate-utility comparisons are not as 'fair' as claimed.
minor comments (4)
  1. [Sec. 4.2, Table 2] The sentence 'TM-RUO attains the highest accuracies across almost all architectures' is contradicted by Table 2, where DDiF outperforms TM-RUO on AlexNet (49.3 vs. 46.7) and ViT (43.5 vs. 43.4). The claim should be revised to reflect that TM-RUO wins on VGG-11 and ResNet-18 but not on the other two architectures.
  2. [Sec. 3.5, Eq. (21)] The bpc metric is defined as total bits divided by the number of classes K, but the text and figures interchangeably use 'bits per class' and 'kB'. This unit confusion should be cleaned up: bpc is a bit count per class, while Table A3 reports it in kilobytes. The definition should be stated explicitly and used consistently.
  3. [Sec. 3.3, Eq. (13)] The notation r(Z; Phi) and the dependence on the entropy network parameters Phi are not fully consistent: Eq. (13) averages over N samples but the total bitrate in Eq. (9) is later used as an additive term with r(Y) and r(Psi). It should be clarified whether r(Z) is the total bits for all synthetic samples or an average-per-sample quantity, since the bpc calculation depends on this.
  4. [Appendix A1, Eq. (A4)] The upper bound on soft-label bitrate is derived under uniform quantization of the simplex, but the text says 'on the order of' bins and then precisely states the bound. The approximation 'Vol(Omega_i) ≈ epsilon^{K-1}' should be made rigorous or explicitly stated as an asymptotic formula; otherwise the numerical illustration for K=1000 (15,456 bits) is not a guaranteed upper bound.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the rate and utility terms are independent quantities, and the authors' self-citation is not load-bearing.

full rationale

The derivation chain is self-contained. The rate term r(Z;Phi) in Eq. (13) is the negative log-likelihood of quantized latents under a learned conditional Laplace model with causal context; it is not defined in terms of the utility loss or of the reported accuracy. The utility term is a standard distillation loss (gradient, trajectory, or distribution matching), and the reported accuracy is obtained by training independent classifiers on the decoded synthetic samples, so the claimed rate-utility trade-off is an empirical measurement rather than a tautology. The bitrate terms r(Phi) and r(Psi) in Eqs. (15) and (18) are computed by entropy-coding the quantized network weights under fitted Laplace priors; they are additive constants excluded from the joint optimization, not fitted values renamed as predictions. The bpc metric in Eq. (21) is a unit-conversion of total bit count per class and does not presuppose the method's success. The only self-citation by the present authors is reference [32] (Li, Ma, You, Zhang, Zuo), which appears in a related-work sentence about discretized mixture-of-likelihood contexts and is not used to justify any load-bearing step of the method; the entropy model actually follows the external C3 work [26]. The post-quantization step-size grid search in Appendix A2 is a hyperparameter choice, and while it is selected to maximize rate-utility performance at the target budget, that does not make the measured accuracy equal to the optimization objective by construction. The unvalidated surrogate bitrate (no arithmetic-coded bitstream comparison) is a correctness and reproducibility risk, but it is not a circularity: the rate estimate remains an independent quantity whose realizability can be checked externally.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced. The central claim rests on the fidelity of the learned entropy model, the independence assumption across channels and scales, and the accounting convention of charging raw-pixel baselines at 32-bit depth. Free parameters include the Lagrange multiplier lambda, the pretraining weight beta, quantization step sizes and MSE thresholds selected by grid search, and architecture choices (context length, decoder capacity).

free parameters (7)
  • Rate-utility Lagrange multiplier lambda = dataset- and loss-specific, with two-stage schedule (e.g., {2x10^1, 10^2} for CIFAR-10 TM)
    Balances r(Z) and l(D;S) in Eq (20); the two-stage schedule is tuned per dataset and loss to impose bitrate constraints.
  • Pretraining distortion weight beta = 10 for TM on ImageNet; 10^6 for GM and DM
    Weight in the overfitted initialization objective Eq (22); chosen per dataset and distillation loss.
  • Quantization step sizes Qe and Qd = grid-searched
    Uniform step sizes for post-training quantization of entropy network and decoder weights; selected to balance bitrate and reconstruction error (Sec 3.3).
  • Post-quantization MSE threshold = selected from {5x10^-5, 5x10^-6, 5x10^-7, 5x10^-8}
    Threshold chosen to maximize rate-utility performance at the target bpc budget (Appendix A2).
  • Entropy network context length C = values in {8, 16, 24, 32, 64}
    Causal neighborhood size for the autoregressive entropy model; tuned per dataset (Appendix A2).
  • Decoder configuration and slice size = v4-40 to v5-320 with slice size per class
    Decoder depth/channels and number of samples per entropy/decoder instance are scaled with spc and dataset, affecting both rate and utility.
  • Target bpc budget = e.g., 192 kB for ImageNet comparisons
    Reported results are evaluated at chosen budgets; each budget corresponds to a specific run configuration with corresponding spc and quantization choices.
assumptions (5)
  • domain assumption Quantized latent codes are well-modeled by a conditional Laplace distribution with causal context.
    The rate in Eq (13) is the negative log-likelihood under this model; miscalibration would bias all reported bpc values.
  • domain assumption Independence of latent codes across channels and scales.
    Stated in Sec 3.3 before Eq (13); ignored correlations could make the true coding cost higher than the estimate.
  • standard math Straight-through estimation or additive uniform noise provides usable gradients through quantization.
    Standard neural-compression practice (refs [5,7]) used to train through round() in Eq (10).
  • ad hoc to paper Per-class sample-specific decoders initialized by overfitted compression (C3 objective, Eq (22)) form a good starting point for the joint optimization.
    The method's practical success depends on this initialization; no ablation or theoretical justification is provided.
  • domain assumption Raw-pixel baselines are charged 32 bits per pixel.
    Footnote 4 sets this convention; common image storage is 8 bits per channel, so the convention inflates relative compression ratios by about 4x.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dataset Distillation as Data Compression: A Rate-Utility Perspective." pith.science (2026). https://pith.science/paper/LTXO2WML

@misc{pith2026250717221,
  author       = {Pith},
  title        = {Pith review of: Dataset Distillation as Data Compression: A Rate-Utility Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LTXO2WML}},
  note         = {Machine review of arXiv:2507.17221}
}
abstract

Driven by the ``scale-is-everything'' paradigm, modern machine learning increasingly demands ever-larger datasets and models, yielding prohibitive computational and storage requirements. Dataset distillation mitigates this by compressing an original dataset into a small set of synthetic samples, while preserving its full utility. Yet, existing methods either maximize performance under fixed storage budgets or pursue suitable synthetic data representations for redundancy removal, without jointly optimizing both objectives. In this work, we propose a joint rate-utility optimization method for dataset distillation. We parameterize synthetic samples as optimizable latent codes decoded by extremely lightweight networks. We estimate the Shannon entropy of quantized latents as the rate measure and plug any existing distillation loss as the utility measure, trading them off via a Lagrange multiplier. To enable fair, cross-method comparisons, we introduce bits per class (bpc), a precise storage metric that accounts for sample, label, and decoder parameter costs. On CIFAR-10, CIFAR-100, and ImageNet-128, our method achieves up to $170\times$ greater compression than standard distillation at comparable accuracy. Across diverse bpc budgets, distillation losses, and backbone architectures, our approach consistently establishes better rate-utility trade-offs.

Figures

Figures reproduced from arXiv: 2507.17221 by the authors.

Figure 1
Figure 1. Comparison of the rate-utility curves on the Nette sub [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. System diagram of our proposed joint rate-utility optimization method for DD. “FC” denotes a fully connected layer whose [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the rate–utility curves on CIFAR-10 and CIFAR-100. Our proposed joint rate–utility optimization method [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualization of synthetic samples on the Nette subset of ImageNet. The first row presents initialization by “overfitted” image [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Bit allocation analysis across bpc regimes, with post [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 60 canonical work pages

  1. [1]

    Soft-to-hard vector quantization for end-to-end learn- ing compressible representations

    Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Luc Van Gool. Soft-to-hard vector quantization for end-to-end learn- ing compressible representations. In Advances in Neural In- formation Processing Systems, pages 1141–1151, 2017. 5

    work page 2017

  2. [2]

    Multi-realism image compression with a conditional generator

    Eirikur Agustsson, David Minnen, George Toderici, and Fabian Mentzer. Multi-realism image compression with a conditional generator. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 22324–22333,

    work page

  3. [3]

    Nasir Ahmed, Raj Natarajan, and Kamisetty R. Rao. Dis- crete cosine transfom. IEEE Transactions on Computers, 23 (1):90–93, 1974. 3

    work page 1974

  4. [4]

    Simoncelli

    Johannes Ball ´e, Valero Laparra, and Eero P. Simoncelli. Density modeling of images using a generalized normaliza- tion transformation. In International Conference on Learn- ing Representations, 2016. 3

    work page 2016

  5. [5]

    Simoncelli

    Johannes Ball ´e, Valero Laparra, and Eero P. Simoncelli. End-to-end optimized image compression. In International Conference on Learning Representations, 2017. 2, 3, 5

    work page 2017

  6. [6]

    Variational image compres- sion with a scale hyperprior

    Johannes Ball ´e, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compres- sion with a scale hyperprior. In International Conference on Learning Representations, 2018. 3

    work page 2018

  7. [7]

    Courville

    Yoshua Bengio, Nicholas L ´eonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. ArXiv preprint arXiv:1308.3432, 2013. 5

    arXiv 2013

  8. [8]

    Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. In Advances in Neural Information Processing Systems, pages 1877–1901, 2020. 1

    work page 1901

Show all 65 references
  1. [9]

    Efros, and Jun-Yan Zhu

    George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 4749–4758, 2022. 1, 2, 4, 6, 7, 5

    2022

  2. [10]

    Efros, and Jun-Yan Zhu

    George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Generalizing dataset dis- tillation via deep generative prior. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3739– 3748, 2023. 1, 2, 6, 7, 4

    2023

  3. [11]

    Influence-guided diffusion for dataset distillation

    Mingyang Chen, Jiawei Du, Bo Huang, Yi Wang, Xiaobo Zhang, and Wei Wang. Influence-guided diffusion for dataset distillation. In International Conference on Learning Repre- sentations, 2025. 2

    2025

  4. [12]

    Cover and Joy A

    Thomas M. Cover and Joy A. Thomas. Elements of Informa- tion Theory. Wiley-Interscience, USA, 2006. 2, 3

    2006

  5. [13]

    Scaling up dataset distillation to ImageNet-1K with constant mem- ory

    Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to ImageNet-1K with constant mem- ory. In International Conference on Machine Learning , pages 6565–6590, 2023. 5

    2023

  6. [14]

    Remember the past: Distilling datasets into addressable memories for neural net- works

    Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural net- works. In Advances in Neural Information Processing Sys- tems, 2022. 2, 6, 5

    2022

  7. [15]

    NICE: Non-linear independent components estimation

    Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. In Interna- tional Conference on Learning Representations, 2015. 3

    2015

  8. [16]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. InInternational ...

    2021

  9. [17]

    COIN: Compression with im- plicit neural representations

    Emilien Dupont, Adam Golinski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. COIN: Compression with im- plicit neural representations. In International Conference on Learning Representations Workshops, 2021. 3

    2021

  10. [18]

    Embarrassingly simple dataset distillation

    Yunzhen Feng, Ramakrishna Vedantam, and Julia Kempe. Embarrassingly simple dataset distillation. In International Conference on Learning Representations, 2024. 1, 2, 3

    2024

  11. [19]

    Robert M. Gray. Vector quantization. IEEE ASSP Magazine, 1(2):4–29, 1984. 5

    1984

  12. [20]

    Borgwardt, Malte J

    Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Sch¨olkopf, and Alexander J. Smola. A kernel two- sample test. Journal of Machine Learning Research, 13:723– 773, 2012. 4

    2012

  13. [21]

    Hern´andez-Lobato

    Zongyu Guo, Gergely Flamich, Jiajun He, Zhibo Chen, and Jos´e M. Hern´andez-Lobato. Compression with Bayesian im- plicit neural representations. In Advances in Neural Infor- mation Processing Systems, 2023. 3

    2023

  14. [22]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Con- ference on Computer Vision and Pattern Recognition, pages 770–778, 2016. 3, 7

    2016

  15. [23]

    Diamos, Heewoo Jun, Hassan Kianinejad, Mostofa Patwary, Yang Yang, and Yanqi Zhou

    Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianinejad, Mostofa Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. ArXiv preprint arXiv:1712.00409,

    arXiv

  16. [24]

    David A. Huffman. A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 40(9): 1098–1101, 1952. 2

    1952

  17. [25]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv preprint arXiv:2001.08361, 2020. 1

    2001 arXiv

  18. [26]

    Schwarz, and Emilien Dupont

    Hyunjik Kim, Matthias Bauer, Lucas Theis, Jonathan R. Schwarz, and Emilien Dupont. C3: High-performance and low-complexity neural compression from a single image or video. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9347–9358, 2024. 2, 3, 5, 6, 1

    2024

  19. [27]

    Dataset condensation via efficient synthetic- data parameterization

    Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic- data parameterization. In International Conference on Ma- chine Learning, pages 11102–11118, 2022. 1, 6, 7, 4, 5

    2022

  20. [28]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding varia- tional Bayes. In International Conference on Learning Rep- resentations, 2014. 3 9

    2014

  21. [29]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convolutional neural net- works. In Advances in Neural Information Processing Sys- tems, pages 1106–1114, 2012. 7

    2012

  22. [30]

    COOL-CHIC: Coordinate-based low complexity hierarchical image codec

    Th ´eo Ladune, Pierrick Philippe, F´elix Henry, Gordon Clare, and Thomas Leguay. COOL-CHIC: Coordinate-based low complexity hierarchical image codec. In IEEE/CVF Interna- tional Conference on Computer Vision, pages 13469–13476,

  23. [31]

    A comprehensive survey of dataset distillation

    Shiye Lei and Dacheng Tao. A comprehensive survey of dataset distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(1):17–32, 2023. 1

    2023

  24. [32]

    Efficient and effective context-based convolutional en- tropy modeling for image compression

    Mu Li, Kede Ma, Jane You, David Zhang, and Wangmeng Zuo. Efficient and effective context-based convolutional en- tropy modeling for image compression. IEEE Transactions on Image Processing, 29:5900–5911, 2020. 3

    2020

  25. [33]

    The evolution of dataset distillation: Toward scalable and generalizable solutions

    Ping Liu and Jiawei Du. The evolution of dataset distillation: Toward scalable and generalizable solutions. ArXiv preprint arXiv:2502.05673, 2025. 1, 5

    2025 arXiv

  26. [34]

    Dataset distillation via factorization

    Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xin- chao Wang. Dataset distillation via factorization. In Ad- vances in Neural Information Processing Systems , 2022. 2, 4, 6, 7, 5

    2022

  27. [35]

    Hasani, Mathias Lechner, and Daniela Rus

    Noel Loo, Ramin M. Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. In International Conference on Machine Learning , pages 22649–22674, 2023. 1

    2023

  28. [36]

    High-fidelity generative image compres- sion

    Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity generative image compres- sion. In Advances in Neural Information Processing Systems,

  29. [37]

    Joint autoregressive and hierarchical priors for learned image compression

    David Minnen, Johannes Ball ´e, and George Toderici. Joint autoregressive and hierarchical priors for learned image compression. In Advances in Neural Information Process- ing Systems, pages 10794–10803, 2018. 3

    2018

  30. [38]

    Dataset meta-learning from kernel ridge-regression

    Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. In International Conference on Learning Representations, 2021. 1

    2021

  31. [39]

    Dataset distillation with infinitely wide convolutional networks

    Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In Advances in Neural Information Processing Systems, pages 5186–5198, 2021. 2, 3

    2021

  32. [40]

    Oord, Nal Kalchbrenner, and Koray Kavukcuoglu

    A ¨aron V . Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pages 1747–1756, 2016. 1

    2016

  33. [41]

    Claude E. Shannon. A mathematical theory of communi- cation. The Bell System Technical Journal , 27(3):379–423,

  34. [42]

    Claude E. Shannon. Coding theorems for a discrete source with a fidelity criterion. IRE National Convention Record, 4 (1):142–163, 1959. 3

    1959

  35. [43]

    Fre- quency domain-based dataset distillation

    Donghyeok Shin, Seungjae Shin, and Il-Chul Moon. Fre- quency domain-based dataset distillation. In Advances in Neural Information Processing Systems , 2023. 1, 2, 6, 7, 4, 5

    2023

  36. [44]

    Distilling dataset into neural field

    Donghyeok Shin, HeeSun Bae, Gyuwon Sim, Wanmo Kang, and Il-Chul Moon. Distilling dataset into neural field. In In- ternational Conference on Learning Representations , 2025. 2, 6, 7, 4, 5

    2025

  37. [45]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. In In- ternational Conference on Learning Representations , 2015. 7

    2015

  38. [46]

    D4M: Dataset distillation via disentangled diffusion model

    Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. D4M: Dataset distillation via disentangled diffusion model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5809–5818, 2024. 2

    2024

  39. [47]

    RNADE: The real-valued neural autoregressive density-estimator

    Benigno Uria, Iain Murray, and Hugo Larochelle. RNADE: The real-valued neural autoregressive density-estimator. In Advances in Neural Information Processing Systems , pages 2175–2183, 2013. 3

    2013

  40. [48]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008, 2017. 3

    2017

  41. [49]

    Dataset distillation

    Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. ArXiv preprint arXiv:1811.10959, 2018. 1, 2

    2018 arXiv

  42. [50]

    Sparse parameterization for epitomic dataset distillation

    Xing Wei, Anjia Cao, Funing Yang, and Zhiheng Ma. Sparse parameterization for epitomic dataset distillation. In Ad- vances in Neural Information Processing Systems , 2023. 1, 6, 7, 4, 5

    2023

  43. [51]

    Paul J. Werbos. Backpropagation through time: What it does and how to do it. Proceedings of the IEEE , 78(10):1550– 1560, 1990. 1

    1990

  44. [52]

    Witten, Radford M

    Ian H. Witten, Radford M. Neal, and John G. Cleary. Arith- metic coding for data compression. Communications of the ACM, 30(6):520–540, 1987. 2

    1987

  45. [53]

    Lossy image compression with conditional diffusion models

    Ruihan Yang and Stephan Mandt. Lossy image compression with conditional diffusion models. In Advances in Neural Information Processing Systems, 2023. 3

    2023

  46. [54]

    Neural spectral decompo- sition for dataset distillation

    Shaolei Yang, Shen Cheng, Mingbo Hong, Haoqiang Fan, Xing Wei, and Shuaicheng Liu. Neural spectral decompo- sition for dataset distillation. In European Conference on Computer Vision, pages 275–290, 2024. 1, 6, 7, 4, 5

    2024

  47. [55]

    An introduc- tion to neural data compression

    Yibo Yang, Stephan Mandt, and Lucas Theis. An introduc- tion to neural data compression. Foundations and Trends® in Computer Graphics and Vision, 15(2):113–200, 2023. 3

    2023

  48. [56]

    Xing, and Zhiqiang Shen

    Zeyuan Yin, Eric P. Xing, and Zhiqiang Shen. Squeeze, re- cover and relabel: Dataset condensation at ImageNet scale from a new perspective. In Advances in Neural Information Processing Systems, 2023. 2, 3, 5, 8

    2023

  49. [57]

    Teddy: Efficient large-scale dataset distillation via Taylor- approximated matching

    Ruonan Yu, Songhua Liu, Jingwen Ye, and Xinchao Wang. Teddy: Efficient large-scale dataset distillation via Taylor- approximated matching. In European Conference on Com- puter Vision, pages 1–17, 2024. 2, 3

    2024

  50. [58]

    TD3: Tucker decomposition based dataset distillation method for sequential recommendation

    Jiaqing Zhang, Mingjia Yin, Hao Wang, Yawen Li, Yuyang Ye, Xingyu Lou, Junping Du, and Enhong Chen. TD3: Tucker decomposition based dataset distillation method for sequential recommendation. In Proceedings of the ACM on Web Conference, page 3994–4003, 2025. 1, 2

    2025

  51. [59]

    Synthesizing informative train- ing samples with GAN

    Bo Zhao and Hakan Bilen. Synthesizing informative train- ing samples with GAN. In Advances in Neural Information Processing Systems Workshops, 2022. 2 10

    2022

  52. [60]

    Dataset condensation with distri- bution matching

    Bo Zhao and Hakan Bilen. Dataset condensation with distri- bution matching. In IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 6514–6523, 2023. 1, 2, 4, 7, 6

    2023

  53. [61]

    Dataset condensation with gradient matching

    Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Con- ference on Learning Representations, 2021. 1, 2, 4, 7, 6

    2021

  54. [62]

    Mao, Chaowei Xiao, and Atul Prakash

    Haizhong Zheng, Jiachen Sun, Shutong Wu, Bhavya Kailkhura, Zhuo M. Mao, Chaowei Xiao, and Atul Prakash. Leveraging hierarchical feature sharing for efficient dataset condensation. In European Conference on Computer Vision, pages 166–182, 2024. 6, 5

    2024

  55. [63]

    Hierarchical features mat- ter: A deep exploration of GAN priors for improved dataset distillation

    Xinhao Zhong, Hao Fang, Bin Chen, Xulin Gu, Tao Dai, Meikang Qiu, and Shu-Tao Xia. Hierarchical features mat- ter: A deep exploration of GAN priors for improved dataset distillation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 2, 6, 7, 4

    2025

  56. [64]

    Dataset distillation using neural feature regression

    Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. In Advances in Neural Information Processing Systems, 2022. 6, 7, 4, 5

    2022

  57. [65]

    slice size

    Jacob Ziv and Abraham Lempel. A universal algorithm for sequential data compression. IEEE Transactions on Infor- mation Theory, 23(3):337–343, 1977. 2 11 Appendix This appendix elaborates on 1) the information-theoretic cost of encoding soft labels, 2) full hyperparameter and ...

    1977

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.