{"id":"904ed7ac-3ab0-4fb4-b1e1-21eea45125df","arxiv_id":"2501.04202","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A generative dataset distillation method that replaces MSE logits matching with standardized KL-divergence distribution matching, reporting improved accuracy on three benchmarks.","lead":"This paper proposes a generative dataset distillation method that uses self-knowledge distillation and logits standardization to align synthetic and original data distributions. It reports accuracy improvements over several existing dataset distillation methods on MNIST, FashionMNIST, and CIFAR-10.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains over DiM may reflect GAN/pool differences rather than the proposed loss; no controlled ablation isolates the self-knowledge distillation component.","rationale":"The reader's weakest assumption focuses on the adequacy of a three-model pool for cross-architecture generalization. That is a legitimate concern, but the paper's own cross-architecture results (Table 2) already show transfer to AlexNet and VGG11, which are not in the pool, providing some evidence that the pool size is not fatal. The more load-bearing issue is attribution: the paper compares the full method to DiM without controlling for the generator architecture, discriminator, pool training, or schedule. Because the proposed method changes the matching loss while also implicitly using a different generative setup, the observed accuracy advantage over DiM could be due to the GAN backbone rather than to self-knowledge distillation or standardization. This is a concrete, technically checkable gap: a controlled ablation that swaps only the matching objective would settle whether the proposed components are the actual source of improvement. This concern does not overturn the reader's CONDITIONAL verdict—it reinforces the need for additional experiments and reporting before the claims are accepted at face value. Hence I recommend leaving the verdict unchanged, with the concrete ablation as a required addition. I mark agreement as 'partial' because the reader identified a related but different experimental-design weakness (pool composition) while I emphasize the missing isolation of the matching loss from GAN/pool confounds.","tokens_in":8214,"tokens_out":9001,"duration_ms":92718,"concrete_test":"Re-implement the proposed framework with the exact same generator, discriminator, model pool (same trained weights and pool composition), and training schedule, then run four variants: (i) LCGAN only; (ii) LCGAN + lambda * ||xO - xS||^2 (DiM-style logits MSE with the same pool); (iii) LCGAN + LSKD without standardization; (iv) the full method with standardization. Compare Table 1 entries for CIFAR-10 IPC=10 and IPC=50, and Table 2 cross-architecture results. If variant (ii) achieves accuracy within 1% of variant (iv), the proposed loss contributes little beyond the GAN/pool setup, and the central claim should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the proposed self-knowledge distillation plus logits standardization yields superior distillation performance. The evidence in Tables 1 and 2 compares the full method to DiM and other baselines, but the comparison is confounded: the proposed method differs from DiM in at least two dimensions simultaneously — (i) the matching objective (KL distribution matching after standardization vs. DiM's MSE logits matching) and (ii) the underlying generative model and model-pool training details. Section 3.2 only states that a pool of three networks (ConvNet3, ResNet10, ResNet18) is used, but it never specifies how these networks are trained (e.g., on the original dataset, with what optimizer, for how many epochs). Likewise, the GAN architecture and training are not described beyond 'conditional GAN.' The ablation 'Ours (No Stand.)' vs. 'Ours' isolates the standardization step only; it does not isolate the contribution of LSKD relative to a simple GAN or to DiM-style logits MSE under identical generator, discriminator, pool, and schedule. If the accuracy advantage over DiM comes primarily from a stronger GAN or a particular pool initialization, then the central claim that the proposed losses are responsible for the improvement is not supported. This is load-bearing because the paper's novelty and abstract claim rest on attributing the empirical gains to the proposed components.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a generative dataset distillation method. A conditional GAN is trained to produce synthetic images, and the generator is further optimized by matching the softmax distributions of standardized logits from a randomly selected model in a pool, using a KL-divergence-based 'self-knowledge distillation' loss. Experiments on MNIST, FashionMNIST, and CIFAR-10 report accuracy improvements over DiM and other baselines, plus cross-architecture results on CIFAR-10.","tokens_in":8432,"tokens_out":3687,"duration_ms":35785,"significance":"If the reported gains are robust, the combination of logits standardization and KL-based distribution matching would be a simple, useful modification to existing generative dataset distillation (e.g., DiM). The paper ships a clear loss formulation and reproducible-looking experimental protocol. However, the empirical evidence currently does not isolate the contribution of the proposed self-knowledge distillation loss, because the main comparison with DiM differs in both the matching objective and the underlying generator/pool setup. The paper is nonetheless a plausible incremental contribution that could become acceptable after a controlled ablation study and additional baselines.","major_comments":[{"comment":"The comparison with DiM is confounded. The proposed method changes the matching objective (KL divergence on standardized softmax outputs instead of DiM's MSE on logits) and also uses a different generator, discriminator, and model-pool training protocol. The 'Ours (No Stand.)' ablation removes only the standardization step; there is no condition that replaces LSKD with DiM-style logits MSE while keeping the generator, discriminator, pool, and schedule identical. As a result, the accuracy advantage over DiM in Tables 1 and 2 cannot be attributed to the self-knowledge distillation loss, which is the central novelty claimed in the abstract and introduction.","section":"Section 2.2, Table 1"},{"comment":"The model pool used for random model selection is underspecified. The paper states that a model is 'randomly selected from a pool of models' consisting of ConvNet3, ResNet10, and ResNet18, but it does not state how these networks are trained (on which dataset, with what optimizer, for how many epochs, with which augmentation). Without this information the cross-architecture generalization claimed in Section 3.3 is not reproducible, and the reader cannot assess whether the pool composition or the loss is responsible for the cross-architecture performance.","section":"Section 3.2"},{"comment":"The abstract claims the method 'outperforms existing state-of-the-art methods', but the reported results do not support this uniformly. On MNIST IPC=50, Ours (99.1±0.1) is below DiM (99.2±0.2), and on FashionMNIST IPC=50 Ours ties DiM at 89.8. The claim should be qualified to reflect the settings where the improvement is consistent, or the discrepancy should be discussed.","section":"Table 1"},{"comment":"The authors' own related generative dataset distillation method, reference [17] (CVPR Workshop 2024), is cited in the bibliography but never compared in the experiments. Since [17] addresses the same task and is likely closely related, a direct empirical comparison is necessary to establish the novelty and incremental benefit of the proposed self-knowledge distillation and standardization components.","section":"Section 3.1 (missing reference [17])"},{"comment":"The use of LCGAN in the total loss is ambiguous. Equation (1) defines a min-max GAN objective, but Equation (7) treats LCGAN as an additive term in Ltotal, which is then said to be minimized with respect to the generator parameters W. The paper should specify whether the discriminator is fixed when W is updated, how the adversarial term is computed (e.g., using the standard generator loss rather than the full min-max objective), and how the gradient flows through the GAN loss during the distribution-matching stage. This is essential for reproducibility.","section":"Equations (1) and (7)"}],"minor_comments":[{"comment":"The word 'comparation' in the table captions should be 'comparison'.","section":"Tables 1 and 2"},{"comment":"The sentence 'IPC values were set to 1, 10, and 50, while batch sizes were 32, 64, and 128' should specify which batch size corresponds to which IPC value, since the pairing is needed to interpret the experiments.","section":"Section 3.2"},{"comment":"The standardization in Equation (3) computes mean and std over the logits vector for each input sample; this should be stated explicitly, and the paper should discuss how zero standard deviation (e.g., for very confident or degenerate outputs) is handled.","section":"Equation (3)"},{"comment":"The term 'self-knowledge distillation' is used for a KL divergence between the softmax outputs of the same model on original and synthetic data; this is essentially distribution matching (as in DM) applied to logits. The paper should relate the term to prior usage of self-knowledge distillation and clarify the difference from simple logits-matching in DiM.","section":"Remark on terminology"}],"recommendation":"major_revision","confidential_remarks":"The paper is short and the core idea is straightforward, but the experimental validation needs a controlled ablation to support the central claim. The omission of the authors' own prior work [17] from the comparisons is a notable gap. I would not reject the paper outright, but the current evidence base is insufficient to accept the claimed superiority of the proposed loss."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a modest incremental step on DiM: replace the MSE logits matching with a KL divergence between softmax distributions of standardized logits, run the same generative pipeline, and report a few points of gain. The idea is clearly presented, the loss is simple, and the standardization ablation (Ours vs. Ours (No Stand.)) is genuinely informative. Over MNIST/FashionMNIST/CIFAR-10 and IPC 1/10/50, the method mostly edges out DiM, and the cross-architecture results are consistent, though the margins are small — 1–2 points, with a couple of ties/losses (MNIST IPC=50, FashionMNIST IPC=50). So the headline claim of \"superior performance\" is overstated.\n\nThe soft spot is load-bearing. The comparison to DiM changes two things at once: the matching loss and the whole underlying GAN/pool/schedule setup. The paper never says how the three pool networks are trained, what the generator architecture is, or the optimizer details. The \"No Stand.\" ablation isolates standardization, but no experiment isolates the KL-vs-MSE choice. Until I see the same generator, same pool, same schedule with only the loss swapped, I can't attribute the gains to the proposed self-knowledge distillation term. It could just be a stronger GAN.\n\nThe baseline list also omits the authors' own CVPRW 2024 generative distillation paper and the diffusion-based one from their group — the closest related work. A referee should ask for those comparisons. And lambda_SKD and tau are dataset-specific with no sensitivity analysis; no code is released either.\n\nThat said, it's not a rejection-level problem. The method is plausible, the experiments are directionally right, and the writing doesn't hide the ties. I'd send it to review with a request for a controlled ablation and the missing baselines. For your reading group it's a maybe — useful as a data point on loss design in generative distillation, but not a landmark. I'd probably cite it as a recent variant if I needed one.\n\nRecommendation: conditional accept with major changes; the key experiment is the controlled loss-swap.","headline":"A modest DiM variant with real but small gains; the claimed attribution to the KL loss needs a controlled ablation.","tokens_in":8974,"tokens_out":4177,"would_cite":true,"duration_ms":37998,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A generative dataset distillation method that standardizes prediction logits and matches their probability distributions through self-knowledge distillation reports higher accuracy than existing distillation baselines on MNIST…","keywords":["generative dataset distillation","self-knowledge distillation","logits standardization","distribution matching","dataset condensation","cross-architecture generalization","conditional GAN"],"falsifier":"Run the method with a model pool containing only ConvNet3 and evaluate the distilled generator on VGG11; if cross-architecture accuracy falls to roughly the DiM level, the random-pool assumption, rather than standardization, is responsible for the generalization gain. Equivalently, sweep the temperature $\\tau$ over a wide range; if the standardization benefit disappears at some values of $\\tau$, the claim that standardization itself is the cause of the improvement would need to be revised.","tokens_in":1526,"feed_emoji":"🧠","tokens_out":1898,"duration_ms":63972,"temperature":0.7,"pith_summary":"Dataset distillation compresses a large training set into a small synthetic one, and generative variants train a GAN to produce that synthetic set on demand. This paper proposes a generative distillation method whose core claim is that matching the probability distributions of prediction logits, rather than matching raw logits, makes the synthetic data capture the original data's structure more faithfully. To make the matching stable, the paper standardizes the logits before the softmax, so that original and synthetic predictions are placed on the same scale. The authors report that this combination beats existing dataset distillation methods on three benchmark datasets and that the gains persist when the distilled generator is evaluated on architectures not used during training.","feed_headline":"Standardized logits push dataset distillation past SOTA","feed_subtitle":"Matching standardized prediction distributions lifts accuracy on MNIST, Fashion-MNIST, and CIFAR-10.","key_machinery":"The machinery is the self-knowledge distillation loss $L_{\\text{SKD}}$, which matches the class-probability distributions of original and synthetic logits through a KL-style divergence instead of minimizing the mean squared error between raw logits. The standardization step $Z(x;\\tau)$ removes scale and shift differences between the two logit vectors before the softmax, which the paper argues prevents the matching from being misled by similar ranges of inaccurate predictions. At each matching step, a model is randomly selected from a pool of ConvNet3, ResNet10, and ResNet18, so the generator is not optimized against a single fixed architecture. The generator itself is trained with the conditional GAN loss $L_{\\text{CGAN}}$, and the distillation loss is added with a weighted temperature-scaled term to form the total objective.","core_discovery":"The central discovery is that replacing plain logits matching with a self-knowledge-distillation loss, applied to standardized logits, improves the accuracy of the synthetic data produced by the generator. Formally, the paper computes $Z(x;\\tau) = \\frac{x - \\text{mean}(x)}{\\text{std}(x)} \\times \\tau$, feeds the standardized logits through softmax to obtain $d(x_O)$ and $d(x_S)$, and then uses $L_{\\text{SKD}} = \\sum_{k=1}^{K} d(x_O)^{(k)} \\log \\frac{d(x_O)^{(k)}}{d(x_S)^{(k)}}$. The total training loss is $L_{\\text{total}} = L_{\\text{CGAN}} + \\lambda_{\\text{SKD}} \\tau^2 L_{\\text{SKD}}$, where $L_{\\text{CGAN}}$ is the conditional GAN loss. The paper's empirical claim is that this design outperforms DiM and other state-of-the-art condensation methods on MNIST, Fashion-MNIST, and CIFAR-10 at images-per-class values of 1, 10, and 50, and that the standardization step alone produces consistent accuracy gains.","pith_inferences":["The paper does not isolate how much of the gain comes from the random model pool versus the standardization step; a natural next experiment is to hold the pool fixed while toggling standardization, and vice versa, on a held-out architecture.","Because standardization removes logits scale and shift, the same normalization could likely be inserted into other logits-comparing distillation losses, such as trajectory matching or gradient matching, with similar stabilizing effects.","The fixed temperature $\\tau = 2$ and per-dataset weight $\\lambda_{\\text{SKD}}$ suggest a tuning interaction worth probing; per-dataset temperature schedules might widen the reported margins beyond the fixed settings used here."],"forward_implications":["Standardizing logits before distribution matching produces consistent accuracy gains over the same method without standardization, across all three datasets and all images-per-class settings tested.","Randomly drawing the matching model from a small pool of architectures gives synthetic data that transfers to unseen architectures such as AlexNet and VGG11, with the standardized version reporting the best cross-architecture results.","Distribution matching of standardized probability vectors captures global structure and inter-class relationships better than direct logits matching, which is why the generator retains more representative information from the original dataset.","The proposed method remains accurate even at very low compression budgets such as one image per class, and it improves further as the budget grows to 10 and 50 images per class."],"supporting_citations":[{"why":"The DiM generative dataset distillation baseline that this method extends and the main comparison for the reported accuracy improvements.","marker":"[16]"},{"why":"The GAN formulation whose generator and adversarial loss $L_{\\text{CGAN}}$ form the base of the synthetic-data generator.","marker":"[19]"},{"why":"The conditional GAN variant that supplies the generator and discriminator structure used in the proposed method.","marker":"[28]"},{"why":"The distribution-matching idea that the paper adapts from MSE logits matching to distribution matching of standardized logits.","marker":"[24]"},{"why":"The gradient-matching DC baseline and the source of the random-selection baseline in Table 1.","marker":"[13]"},{"why":"The trajectory-matching MTT baseline compared in both the benchmark and cross-architecture results.","marker":"[14]"},{"why":"The neural feature regression FRePo baseline compared in both the benchmark and cross-architecture results.","marker":"[27]"}],"fun_headline_variants":["Standardized logits boost dataset distillation accuracy","Self-knowledge distillation sharpens synthetic data","Logit standardization is key to better distilled datasets","Generative distillation improved via standardized logits"],"cache_read_input_tokens":11136,"weakest_assumption_plain":"The load-bearing premise is that a matching model randomly drawn from a pool of just three architectures, ConvNet3, ResNet10, and ResNet18, provides a distribution-matching signal that transfers to other architectures; the paper does not show why three models suffice or how sensitive the results are to the pool's composition.","fun_headline_variants_meta":{"raw":{"variants":["Standardized logits boost dataset distillation accuracy","Self-knowledge distillation sharpens synthetic data","Logit standardization is key to better distilled datasets","Generative distillation improved via standardized logits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1359,"prompt_tokens":908,"completion_tokens":451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":395}},"tokens_in":524,"tokens_out":451,"duration_ms":4850,"temperature":1.0,"reasoning_tokens":395,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:38:30.663867+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the method with a model pool containing only ConvNet3 and evaluate the distilled generator on VGG11; if cross-architecture accuracy falls to roughly the DiM level, the random-pool assumption, rather than standardization, is responsible for the generalization gain. Equivalently, sweep the temperature $\\tau$ over a wide range; if the standardization benefit disappears at some values of $\\tau$, the claim that standardization itself is the cause of the improvement would need to be revised.","supporting_citations":[{"cited_title":"Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data,","cited_arxiv_id":null,"evidence_quote":"The DiM generative dataset distillation baseline that this method extends and the main comparison for the reported accuracy improvements."},{"cited_title":"Importance-aware adaptive dataset distillation,","cited_arxiv_id":null,"evidence_quote":"The GAN formulation whose generator and adversarial loss $L_{\\text{CGAN}}$ form the base of the synthetic-data generator."},{"cited_title":"Dataset condensation with distri- bution matching,","cited_arxiv_id":null,"evidence_quote":"The conditional GAN variant that supplies the generator and discriminator structure used in the proposed method."},{"cited_title":"Super-samples from kernel herding,","cited_arxiv_id":null,"evidence_quote":"The distribution-matching idea that the paper adapts from MSE logits matching to distribution matching of standardized logits."},{"cited_title":"Dataset distillation for medical dataset sharing,","cited_arxiv_id":null,"evidence_quote":"The gradient-matching DC baseline and the source of the random-selection baseline in Table 1."},{"cited_title":"An efficient dataset condensation plugin and its application to continual learning,","cited_arxiv_id":null,"evidence_quote":"The trajectory-matching MTT baseline compared in both the benchmark and cross-architecture results."},{"cited_title":"Dataset condensation with dif- ferentiable siamese augmentation,","cited_arxiv_id":null,"evidence_quote":"The neural feature regression FRePo baseline compared in both the benchmark and cross-architecture results."}],"review_version":1}