{"id":"e1034e35-ebce-495f-9978-2ec6877f7936","arxiv_id":"2507.17221","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A new method frames dataset distillation as joint rate-utility optimization, storing distilled images as entropy-coded latent codes plus tiny decoders, and it reports better storage-accuracy trade-offs than prior methods.","lead":"Researchers built a way to compress training image datasets into tiny synthetic datasets by borrowing ideas from data compression. The method stores small numerical codes and tiny decoders, fitting more accuracy into fewer bits than prior dataset distillation methods on three benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified surrogate bitrate: reported bpc is a cross-entropy estimate under a post-quantized entropy model, never validated against an actual arithmetic-coded bitstream; every rate-utility comparison hinges on this.","rationale":"I agree with the reader's identification of the entropy-model fidelity as the weakest load-bearing assumption. The other reservations in the reader's report—32-bit versus 8-bit raw-pixel costing, post-hoc lambda schedules and grid searches, and the omission of r(Phi) and r(Psi) from Eq. (20)—are real but secondary: they affect the framing of the method as joint optimization or the size of the headline 170x number, not the validity of the bpc axis itself. The entropy-model issue is more fundamental because every rate-utility curve, every table with bpc budgets, and the central claim of better rate-utility trade-offs depend on the reported bpc being an achievable storage cost. The paper computes r(Z) as a training-time cross-entropy under a model whose parameters are later quantized, and it never validates that an actual arithmetic coder can realize this rate. In principle the cross-entropy of a known conditional model is achievable by arithmetic coding, so this is an implementation gap rather than a mathematical impossibility; but the gap is material because the deployed entropy network is quantized, and the selection criterion for Qe is image reconstruction MSE, not coding-rate fidelity. The concrete test I propose—running an actual arithmetic coder on the final quantized latents with the final quantized entropy network—would settle the question directly. If the measured bitrate is close to the reported bpc, the conditional verdict can stand and the concern is resolved; if it is materially larger, the bpc values need revision and the rate-utility comparisons must be redone. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":21569,"tokens_out":16491,"duration_ms":184090,"concrete_test":"Take the ImageNet Nette spc=51 and spc=102 trained TM-RUO models. After post-quantization, recompute r(Z;Phi) using the quantized entropy network on the final quantized latents, then implement a range arithmetic coder with the identical raster-scan context and discretized Laplace probability tables, encode all six scales, and compare total bytes (including any header and termination overhead) with the reported 101.7 kB and 179.7 kB. Also decode the bitstream and verify that the reconstructed latents match exactly. If the measured bpc exceeds the reported value by more than 2%, replot the rate-utility curves with measured bitrates and recheck whether TM-RUO still dominates DDiF at matched budgets; if the excess is greater than 10%, the central quantitative claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that TM-RUO attains better rate-utility trade-offs, with all comparisons plotted against bpc. The bpc values are not measured bitstream lengths: r(Z) in Eq. (13) is the negative log-likelihood of quantized latents under a learned conditional Laplace model, and r(Phi), r(Psi) in Eqs. (15) and (18) are computed under fitted Laplace priors. The paper never runs an actual entropy coder. The estimate would be realizable only if an arithmetic coder can use exactly the same context model and probabilities at decode time. Two things can break this: (i) the entropy network used to compute r(Z) is trained in full precision, while the deployed one is post-quantized (Eq. 14) using a grid search whose selection criterion is reconstruction MSE of synthetic images, not rate fidelity, so the quantized phi can change the predicted (mu, sigma) and hence the cross-entropy; (ii) the model assumptions (fixed context length C, independence across channels and scales, discretized Laplace) may be a poor match to the actual quantized latents, and arithmetic coding with this model realizes its cross-entropy only if the coder's contexts are exactly those used in Eq. (13). If the realized bitstream is longer than the surrogate, every TM-RUO point moves right on the rate axis; the headline comparison in Table A3 (76.5% at 179.7 kB vs DDiF 72.0% at 192 kB) and the claimed Pareto improvements over DDiF, IDC, FreD, and others are all x-axis dependent. This is the single most load-bearing unverified element: it is the only link between the optimized surrogate loss and the storage costs the paper claims to minimize.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reframes dataset distillation as a joint rate-utility optimization problem. Synthetic samples are parameterized as multiscale latent codes decoded by lightweight per-class networks, and the storage cost is measured through a learned context-aware entropy model over quantized latents, plus the costs of labels and decoder/entropy network parameters. A Lagrangian objective trades this rate against an arbitrary distillation loss (trajectory, gradient, or distribution matching). The authors introduce a bits-per-class (bpc) metric to compare methods with different storage designs, and report experiments on CIFAR-10, CIFAR-100, and six 128x128 ImageNet subsets showing improved rate-utility trade-offs over existing methods including TM, FreD, DDiF, and others.","tokens_in":21857,"tokens_out":6798,"duration_ms":73017,"significance":"If the reported storage numbers are faithful, this is a valuable contribution: it connects dataset distillation to the mature toolbox of neural lossy compression, makes the rate term differentiable and jointly optimizable with a plug-and-play utility loss, and proposes a unified storage metric (bpc) that is missing from the field. The experiments are extensive, with means and standard deviations over five trials, and cover multiple losses and architectures. The paper also ships a project URL, aiding reproducibility. However, the central quantitative claim depends on an unvalidated surrogate bitrate: the reported bpc values are cross-entropy estimates under a learned entropy model rather than measured arithmetic-coded bitstream lengths. Since every rate-utility comparison is plotted against bpc, the strength and even the sign of the claimed improvements hinge on how close the surrogate is to a realizable code length.","major_comments":[{"comment":"The rate r(Z) is computed as a negative log-likelihood under a conditional Laplace model, not as the length of an actual entropy-coded bitstream. The paper never runs an arithmetic coder or any other real entropy coder on the quantized latents, so there is no evidence that the assumed model (independence across channels and scales, raster-scan causal context, discretized Laplace) can be realized at the reported bpc. This is load-bearing because all rate-utility curves (Figs. 1 and 3; Tables A3-A6) use bpc as the x-axis; if the true code length is higher, every TM-RUO point moves right and the claimed Pareto improvements may shrink or disappear. The authors should implement an arithmetic coder that uses exactly the quantized entropy network and context model described in Eqs. (11)-(13), report actual bitstream lengths, and compare them to the surrogate rate. At minimum, they should bound the gap, e.g., by comparing against a standard arithmetic coder with a simple but correct model.","section":"Sec. 3.3, Eq. (13) and Sec. 4.2"},{"comment":"The post-quantization of the entropy network parameters (step size Qe) is selected by grid search on reconstruction MSE of synthetic images, not on rate fidelity. The rate r(Z) in Eq. (13) is computed with the full-precision entropy network, but the deployed decoder would have to use the quantized weights phi_bar; the quantized network can predict different (mu, sigma) and thus change the cross-entropy. The paper does not show that r(Z) is stable under quantization, nor does it recompute r(Z) with phi_bar when reporting bpc. The authors should either (a) recompute the rate with post-quantized entropy networks during the grid search and select Qe based on the resulting rate-utility trade-off, or (b) empirically demonstrate that the rate estimate changes by a negligible amount. The same concern applies to the decoder quantization step Qd, which is also selected without explicit rate fidelity.","section":"Sec. 3.3, Eq. (14) and Appendix A2"},{"comment":"The bpc metric charges raw-pixel baselines at 32 bits per pixel. This is a strong accounting assumption: many dataset distillation pipelines store synthetic images at 8-bit or 16-bit precision in practice, and a baseline like TM (Vanilla) could be quantized to 8-bit (48 kB for a 128x128x3 ipc=1 image) without necessarily sacrificing much accuracy. Because the paper's headline '170x compression' and the Pareto comparisons are computed against 32-bit baselines, the claimed advantage is sensitive to this choice. The authors should justify the 32-bit assumption, or report rate-utility curves under alternative bit-depth assumptions (e.g., 8-bit raw pixels) to show that the qualitative and quantitative conclusions are robust. Without this sensitivity analysis, the rate-utility comparisons are not as 'fair' as claimed.","section":"Footnote 4 and Tables A3-A4"}],"minor_comments":[{"comment":"The sentence 'TM-RUO attains the highest accuracies across almost all architectures' is contradicted by Table 2, where DDiF outperforms TM-RUO on AlexNet (49.3 vs. 46.7) and ViT (43.5 vs. 43.4). The claim should be revised to reflect that TM-RUO wins on VGG-11 and ResNet-18 but not on the other two architectures.","section":"Sec. 4.2, Table 2"},{"comment":"The bpc metric is defined as total bits divided by the number of classes K, but the text and figures interchangeably use 'bits per class' and 'kB'. This unit confusion should be cleaned up: bpc is a bit count per class, while Table A3 reports it in kilobytes. The definition should be stated explicitly and used consistently.","section":"Sec. 3.5, Eq. (21)"},{"comment":"The notation r(Z; Phi) and the dependence on the entropy network parameters Phi are not fully consistent: Eq. (13) averages over N samples but the total bitrate in Eq. (9) is later used as an additive term with r(Y) and r(Psi). It should be clarified whether r(Z) is the total bits for all synthetic samples or an average-per-sample quantity, since the bpc calculation depends on this.","section":"Sec. 3.3, Eq. (13)"},{"comment":"The upper bound on soft-label bitrate is derived under uniform quantization of the simplex, but the text says 'on the order of' bins and then precisely states the bound. The approximation 'Vol(Omega_i) ≈ epsilon^{K-1}' should be made rigorous or explicitly stated as an asymptotic formula; otherwise the numerical illustration for K=1000 (15,456 bits) is not a guaranteed upper bound.","section":"Appendix A1, Eq. (A4)"}],"recommendation":"major_revision","confidential_remarks":"The surrogate-rate concern is the central issue. It is not a circularity or a fabricated result; it is a standard proxy in neural compression. But in neural compression the proxy is always validated against an actual bitstream. Here the entire rate-utility comparison is built on the proxy, so the fix is essential before publication. The paper is otherwise well-structured and the experiments are carefully run. I recommend major revision rather than rejection: adding an arithmetic-coder validation and a bit-depth sensitivity analysis is within the manuscript's scope and should settle the question. I would also suggest the editor ask the authors to release the exact bpc computation details (including how r(Z) in Eq. (13) is converted to a per-class bit count), as this is currently difficult to verify from the text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my honest read on 2507.17221.\n\nThis is a paper worth engaging with. It introduces something genuinely new: a rate-utility formulation for dataset distillation, where the storage cost of synthetic latents is estimated via a learned entropy model and traded off against any existing distillation loss. The bits-per-class (bpc) metric, which accounts for sample, label, and decoder parameters, is a sensible standardization that the field could adopt. The experiments are extensive, with means and standard deviations over five trials across three datasets, multiple distillation losses, and four architectures. The gains over DDiF and other strong baselines are consistent and credible.\n\nTwo soft spots, one of them load-bearing. First, the reported bpc values are cross-entropy estimates under a conditional Laplace model, not measured bitstream lengths. The paper never runs an actual arithmetic coder. The stress-test note is correct that the entropy network is post-quantized via grid search on reconstruction MSE, not rate fidelity, so the deployed model's probabilities may differ from the ones used for the reported rates. If the true bitstreams are longer, every rate-utility point moves right, and the headline comparison in Table A3 weakens. This is fixable: compute the cross-entropy under the post-quantized model or actually code a few bitstreams and report the gap. Without that, the numbers are estimates, and the paper should say so more clearly.\n\nSecond, the baselines are charged 32 bits per pixel (footnote 4). That's a nonstandard convention for raw images, which are almost always 8-bit. It inflates the apparent compression advantage. The authors should justify this choice or report results under both conventions.\n\nMinor: 'joint' optimization is a bit stronger than what's implemented, since Eq. 20 treats label, entropy-network, and decoder bitrates as constants. The authors explain why, so it's a wording issue, not a deep flaw.\n\nOverall, the central idea is sound and likely to influence how dataset distillation is compared. The missing bitrate validation is the main thing I'd want resolved before trusting the exact numbers, but it's not a desk-reject-level problem. This deserves serious peer review.","headline":"A solid, influential rate-utility framework for dataset distillation whose reported bitrate numbers need validation against actual coding before being taken at face value.","tokens_in":22504,"tokens_out":4089,"would_cite":true,"duration_ms":42824,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dataset distillation should be optimized as a compression problem—minimize bits and utility loss together—and doing so produces synthetic datasets up to 170x smaller at comparable or better accuracy.","keywords":["dataset distillation","rate-utility optimization","neural data compression","entropy coding","bits per class","latent codes","trajectory matching","quantization"],"falsifier":"Take any fully trained TM-RUO synthetic dataset at a reported bpc, run a genuine range or arithmetic coder over the quantized latents and quantized network weights using the same causal context model, and compare the actual file size with the sum $r(Z)+r(\\Phi)+r(\\Psi)$ used in the paper. If the true bitstream is materially larger, each rate-utility curve shifts right and the reported Pareto dominance at tight budgets would need to be re-measured.","tokens_in":21265,"feed_emoji":"🗜️","tokens_out":9091,"duration_ms":94264,"temperature":0.7,"pith_summary":"Dataset distillation replaces a large training set with a small set of synthetic examples, and this paper argues that the right way to build that set is to treat it as a compression problem: minimize storage cost and task-utility loss together, not sequentially. The authors parameterize synthetic images as multiscale quantized latent codes decoded by extremely lightweight per-sample networks, estimate the bitrate of those codes with a context-aware entropy model, and combine that rate with any existing distillation loss in a single Lagrange-multiplier objective. To make methods comparable they introduce bits per class (bpc), which counts latent bits, label bits, and decoder-parameter bits per class. On CIFAR-10, CIFAR-100, and ImageNet-128, the resulting method reports better accuracy than prior work at equal or smaller storage, including up to 170x compression relative to standard trajectory matching at comparable accuracy. If valid, this turns dataset distillation into a rate-utility trade-off problem whose Pareto frontier can be explored with off-the-shelf distillation losses.","feed_headline":"Distilled datasets get up to 170x smaller at same accuracy","feed_subtitle":"Storing synthetic images as quantized latent codes gets more accuracy per stored bit.","key_machinery":"The central mechanism is a hybrid synthetic-dataset parameterization with a differentiable rate estimate. Each synthetic image is produced by upsampling and concatenating latent codes at $L$ scales and passing them through a very small decoder network, so the storage cost splits into latent bits, label bits, and parameter bits. The rate of the quantized latents is modeled as $$r(Z;\\Phi)=-\\frac{1}{N}\\sum_{i}\\sum_{l,m}\\log_2 P(\\bar{z}^{(i)}_{l,m}\\mid \\bar{c}^{(i)}_{l,m})$$ under a conditional Laplace distribution whose location and scale are predicted by a causal-context entropy network, with an assumed independence across channels and scales. The trade-off is then produced by minimizing $r(S)+\\lambda\\ell(D;S)$ end-to-end, and the new bits-per-class metric $\\mathrm{bpc}(S)=\\#\\mathrm{bits}(S)/K$ converts the resulting dataset into a single storage number that is comparable across very different distillation designs.","core_discovery":"On its own terms, the central discovery is that jointly optimizing a differentiable rate term and a differentiable utility term dominates optimizing either alone. Synthetic samples are stored as quantized latents at several scales plus lightweight per-sample decoder and entropy networks, and the full objective is $r(S)+\\lambda\\ell(D;S)$, with $r(S)=r(Z)+r(Y)+r(\\Phi)+r(\\Psi)$. Using an autoregressive Laplace entropy model for the latents and a Shannon-entropy bound for hard labels, the method reaches 79.7% on CIFAR-10, 49.2% on CIFAR-100, and 76.5% on ImageNet-Nette at per-class budgets of 246 kB, 53 kB, and 179.7 kB respectively, outperforming the strongest neural-field baseline, which reaches 72.0% on Nette at 192 kB. The rate term is loss-agnostic: plugging gradient matching or distribution matching into the same objective lifts their accuracy from 28.1% to 49.5% and from 25.7% to 59.1% on the ImageNet subsets at 192 kB per class. The paper's framing implies that any future distillation method can be graded by its position on a bpc-accuracy curve rather than by images-per-class alone.","pith_inferences":["An implication the authors do not draw is that, because the reported rate comes from a surrogate entropy model rather than a real bitstream, the bpc numbers are optimistic lower bounds; re-encoding with an actual arithmetic or range coder would test how much of the claimed 170x survives in true file sizes.","The paper's bit-allocation analysis shows that implicit decoder bits dominate at low budgets and explicit latent bits at high budgets; a natural extension would make the Lagrange multiplier per-class or per-sample so the frontier adapts instead of using one global $\\lambda$.","The bpc normalization by class count makes it possible to compare methods with very different numbers of synthetic samples, which ipc cannot do; the paper proposes bpc for evaluation but does not speculate on it becoming a standard reporting unit in the field."],"forward_implications":["At equal bit budgets, the method reports higher downstream accuracy than prior distilled datasets on CIFAR-10, CIFAR-100, and all six ImageNet-128 subsets; for example, 76.5% versus 72.0% on Nette at about 180–192 kB per class.","The same joint objective improves gradient-matching and distribution-matching distillation as well as trajectory matching, so the rate term acts as a general add-on rather than a loss-specific trick.","Storage can be cut by up to 170x relative to vanilla trajectory-matching distillation while keeping accuracy, meaning synthetic datasets of a few hundred kilobytes can train competitive classifiers.","The bpc metric makes rate-utility curves the natural comparison tool, exposing that prior images-per-class budgets hid large differences in true storage cost.","Distilled datasets optimized with the method transfer across several classifier architectures better than prior baselines on average, with 50.7% versus 45.2% mean accuracy, so the improvements are not tied to the architecture used during distillation."],"supporting_citations":[{"why":"Supplies the default trajectory-matching utility loss and the vanilla trajectory-matching baseline that the method must beat.","marker":"[9]"},{"why":"Provides the per-sample overfitted-compression recipe, entropy-model defaults, and the initialization objective used in Eq. (22).","marker":"[26]"},{"why":"Establishes the end-to-end rate-distortion optimization and additive-noise quantization that the joint objective adapts.","marker":"[5]"},{"why":"Contributes the hybrid multiscale latent-plus-lightweight-decoder parameterization used for the synthetic dataset.","marker":"[30]"},{"why":"Supports optimizing latents and decoders together, the hybrid scheme underlying the synthetic-data parameterization.","marker":"[34]"},{"why":"Is the strongest neural-field baseline whose per-class storage and accuracy the experiments directly compare against.","marker":"[44]"},{"why":"Supplies the frequency-domain baseline and the downstream classifier training protocol used in evaluation.","marker":"[43]"},{"why":"Underpins the Shannon-entropy bound used to price class labels and justify hard-label storage.","marker":"[41]"}],"fun_headline_variants":["170x smaller distilled datasets via joint rate-utility optimization","Distillation as compression: 170x smaller at same accuracy","Joint rate-utility optimization squeezes distilled data 170x","Better rate-utility trade-offs in dataset distillation","Compress distilled data 170x with rate-utility tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument stands on the premise that the entropy model's predicted bit count matches what a real coder would emit: the paper assumes independence across channels and scales in Eq. (13) and never encodes an actual arithmetic-coded bitstream, so every reported bpc is only as good as that Laplace context model.","fun_headline_variants_meta":{"raw":{"variants":["170x smaller distilled datasets via joint rate-utility optimization","Distillation as compression: 170x smaller at same accuracy","Joint rate-utility optimization squeezes distilled data 170x","Better rate-utility trade-offs in dataset distillation","Compress distilled data 170x with rate-utility tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1668,"prompt_tokens":1027,"completion_tokens":641,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":559}},"tokens_in":643,"tokens_out":641,"duration_ms":6106,"temperature":1.0,"reasoning_tokens":559,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:55:18.451809+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any fully trained TM-RUO synthetic dataset at a reported bpc, run a genuine range or arithmetic coder over the quantized latents and quantized network weights using the same causal context model, and compare the actual file size with the sum $r(Z)+r(\\Phi)+r(\\Psi)$ used in the paper. If the true bitstream is materially larger, each rate-utility curve shifts right and the reported Pareto dominance at tight budgets would need to be re-measured.","supporting_citations":[{"cited_title":"Efros, and Jun-Yan Zhu","cited_arxiv_id":null,"evidence_quote":"Supplies the default trajectory-matching utility loss and the vanilla trajectory-matching baseline that the method must beat."},{"cited_title":"Schwarz, and Emilien Dupont","cited_arxiv_id":null,"evidence_quote":"Provides the per-sample overfitted-compression recipe, entropy-model defaults, and the initialization objective used in Eq. (22)."},{"cited_title":"Simoncelli","cited_arxiv_id":null,"evidence_quote":"Establishes the end-to-end rate-distortion optimization and additive-noise quantization that the joint objective adapts."},{"cited_title":"COOL-CHIC: Coordinate-based low complexity hierarchical image codec","cited_arxiv_id":null,"evidence_quote":"Contributes the hybrid multiscale latent-plus-lightweight-decoder parameterization used for the synthetic dataset."},{"cited_title":"Dataset distillation via factorization","cited_arxiv_id":null,"evidence_quote":"Supports optimizing latents and decoders together, the hybrid scheme underlying the synthetic-data parameterization."},{"cited_title":"Distilling dataset into neural field","cited_arxiv_id":null,"evidence_quote":"Is the strongest neural-field baseline whose per-class storage and accuracy the experiments directly compare against."},{"cited_title":"Fre- quency domain-based dataset distillation","cited_arxiv_id":null,"evidence_quote":"Supplies the frequency-domain baseline and the downstream classifier training protocol used in evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Underpins the Shannon-entropy bound used to price class labels and justify hard-label storage."}],"review_version":1}