{"id":"f0656725-179d-4fdc-86e5-e1f90bfa7b75","arxiv_id":"2506.24125","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A dataset distillation method combining data-level residual connections, mixed precision, and multi-resolution optimization achieves new state-of-the-art accuracy with roughly half the compute.","lead":"FADRM is a method for dataset distillation, the task of compressing a large image dataset into a small set of synthetic images that trains models almost as well. It adds data-level residual connections, mixed-precision computation, and multi-resolution optimization, reporting higher accuracy with roughly half the compute and memory on benchmarks like ImageNet-1K.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed SOTA margin over EDC is small and rests on best-of-four hyperparameter reporting with no error bars; the unverified condition (11) behind Theorem 2 is a real but secondary concern.","rationale":"The reader's weakest_assumption correctly flags Theorem 2's condition (11) as unverified, and the proof's algebraic slip is real. However, the stated central claim is empirical: FADRM establishes a new SOTA and cuts cost by 50%. For that claim, the evaluation protocol is more load-bearing than the theorem. The paper's own ablations show accuracy varying by up to two points with α (45.7–47.7 in Table 5), while the final margin over EDC is only about 1.4–2.3 points. With best-of-four post-evaluation schedules and no error bars, the reported margin is within plausible noise. The theorem issue matters for the paper's 'theoretically grounded' framing and should be fixed or qualified, but it does not by itself overturn the experiments. Since the reader already assigned a CONDITIONAL verdict, this assessment does not change the verdict.","tokens_in":19883,"tokens_out":12175,"duration_ms":137479,"concrete_test":"Fix the final FADRM+ configuration (α=0.5, k=3, Dds=200, ensemble of ResNet18, DenseNet121, ShuffleNetV2, MobileNetV2) and the EDC baseline; rerun ImageNet-1K IPC=10 distillation and post-evaluation under identical S1–S4 schedules with at least three independent seeds. If the mean FADRM+−EDC gap is ≤1 accuracy point or the confidence intervals overlap, the paper should not claim a new SOTA over EDC without a paired multi-seed comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—FADRM+ beating EDC on ImageNet-1K IPC=10 by +1.4% in the abstract (50.0 vs 48.6) or by +2.3% in Table 1—is the most load-bearing part of the paper. The ablation selects α=0.5 from a grid (Table 5), and post-evaluation selects among four schedules S1–S4 (Tables 11, 13, 15, 17, 18), with no repeated runs or error bars. For large-scale distillation, seed-to-seed and schedule-to-schedule variation is typically at least ±1%; a +1.4% margin against a baseline that is itself reported as the best of two protocols (Appendix E) is therefore fragile. This is not an accusation of cherry-picking; it is a statement that the evidence as reported cannot yet distinguish FADRM+ from EDC. The theory-level concern is also valid: Theorem 2's condition (11) is never verified, and the proof has a notational slip ('L + h' in Eq. 49, which should be Lh). If (11) fails for actual synthetic data, the claimed theoretical grounding for ARC disappears. But the empirical SOTA claim would still stand or fall on the evaluation protocol, which is why I treat the evaluation issue as load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FADRM, a dataset distillation method that combines three components: mixed-precision training (MPT), multi-resolution optimization (MRO), and adjustable residual connections (ARC). Synthetic images are initialized from real patches, optimized against a pretrained model at alternating resolutions, and periodically fused with resampled original patches via a merge ratio α. The authors report state-of-the-art results on CIFAR-100, Tiny-ImageNet, ImageNet-1K and its subsets, including 47.7% (single-model) and 50.0% (multi-model) top-1 accuracy on ImageNet-1K at IPC=10 with ResNet-18, alongside roughly 50% reductions in generation time and peak GPU memory. Two theoretical results are stated: Theorem 1 bounds the mutual information between uni-level synthetic data and the original dataset, and Theorem 2 gives a generalization-bound argument for ARC. The appendix provides proofs, extensive hyperparameter tables, ablations, cross-architecture generalization results, efficiency comparisons, and a continual learning application.","tokens_in":20205,"tokens_out":7134,"duration_ms":74882,"significance":"If the empirical claims hold, FADRM is a strong practical contribution: it is simple, efficient, and shows consistent gains across several datasets and architectures, with concrete reductions in time and memory. The paper also documents its hyperparameters carefully and makes code available, which are positive reproducibility features. However, the central SOTA claim is currently supported by an evaluation protocol that selects the best of four post-evaluation schedules and reports no error bars, and the theoretical justification of ARC depends on an unverified condition with a proof error. These issues are load-bearing for the paper's two headline claims (state-of-the-art accuracy and theoretically grounded residual connections), so the present evidence is suggestive rather than conclusive.","major_comments":[{"comment":"The reported FADRM/FADRM+ numbers in Table 1 are the best of four post-evaluation settings S1–S4 described in Appendix F, and the mapping from each result to a specific setting is not given. No repeated runs or error bars are reported. For ImageNet-1K IPC=10 with ResNet-18, the abstract and Table 2 report FADRM+ at 50.0%, while Table 1 reports 50.9%, without explanation of which configuration produced which number. Since the claimed margin over EDC is only +1.4 to +2.3 percentage points, and schedule-to-schedule or seed-to-seed variation in large-scale distillation can plausibly account for that difference, the current evidence does not establish a SOTA margin over EDC. The authors should prespecify a single evaluation protocol, report means and variances over multiple seeds, and reconcile the inconsistent 50.0/50.9 numbers.","section":"§4.2, Table 1, Appendix F"},{"comment":"Theorem 2's conclusion is conditioned on inequality (11), Rn(H∘O) − Rn(H∘C̃res) < −LhΔ(Ll+2Bα)/(2B), but the paper never verifies this condition for any dataset, hypothesis class, or trained hypothesis, and gives no argument that it should hold in practice. Without that verification, the theorem does not establish that ARC improves generalization. In addition, the proof contains an algebraic error: Eq. (49) writes 'L + hα(1−α)·Δ' where dimensional consistency requires Lh, and the step from Eqs. (44) and (48) to Eq. (49) is not derived. Because the paper explicitly advertises the method as 'theoretically grounded,' this theorem needs to be corrected and its condition either verified or replaced by an unconditional analysis.","section":"§3.4, Eq. (11), Appendix A.3"},{"comment":"The efficiency comparisons mix two different method variants: FADRM uses a single pretrained model (ResNet-18), while FADRM+ uses an ensemble of four models (Table 8), yet Table 1 and the cross-architecture table compare FADRM+ against single-model baselines such as RDED. The claimed 50% reduction in time and peak memory is presented for both variants, but Table 4 reports FADRM+ peak memory as 12 GB while Table 2 reports 11.0 GB for the same variant, and the per-image time of 1.09 s for FADRM+ presumably includes all four teachers while FADRM's 0.47 s uses one. The comparison should report which variant, which teacher set, and which compute budget underlies each efficiency claim so that the gains are attributable to the method and not to an asymmetric comparison.","section":"§3.2, §4.2, Tables 2 and 4"}],"minor_comments":[{"comment":"The text contains a typo: 'see Theroem 2' should read 'see Theorem 2'.","section":"§3.4"},{"comment":"Theorem 1's statement uses H(fθ) both as the maximum per-sample entropy of the network output and, in the proof, as the entropy of the model fθ; these are different quantities and should be disambiguated.","section":"§3 (Theorem 1) and Appendix A.2"},{"comment":"The baseline 'SRe2L++' is cited as reference [6], but reference [6] is the CV-DD paper; SRe2L++ is reference [43]. This citation error appears in both tables.","section":"Table 2 (left) and Table 4"},{"comment":"The saved-computation formula in Eq. (8) uses '⌊k/2⌋' without derivation; it is not immediately clear which stages of Algorithm 1 run at downsampled resolution for odd versus even k, so a short derivation or a precise counting of stages would improve clarity.","section":"Eq. (8)"},{"comment":"The post-evaluation hyperparameter tables list which settings S1–S4 are used for each dataset and model, but Table 1's cells do not indicate which setting produced each reported accuracy; a mapping table from Table 1 entries to S1–S4 would make the evaluation reproducible.","section":"Appendix F"},{"comment":"The diagram is dense and does not label the resolution at every stage; adding the current spatial resolution and the merge step at each data residual block would make the multi-resolution schedule easier to follow.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core idea: instead of optimizing synthetic images in isolation, periodically fuse them with resized real patches via a convex combination. That is simple and new in this subfield. The paper backs it with a large experimental sweep—five datasets, multiple architectures—and reports consistent gains over RDED, CV-DD, and EDC, plus about 50% reductions in time and peak memory. The efficiency numbers are plausible given MPT and MRO, and the ablation is honest: MRO alone hurts, ARC at α=0.5 is what makes the framework work.\n\nThe soft spots are real. The main SOTA claim on ImageNet-1K IPC=10 is FADRM+ at 50.0 vs EDC at 48.6, a +1.4% margin. That margin is smaller than typical seed-to-seed variation in large-scale distillation, and the paper reports no repeated runs, no error bars, and evaluates four post-training schedules (S1–S4) picking the best per model/dataset. The baseline numbers are also best-of-two protocols. So as reported, the evidence cannot distinguish FADRM+ from EDC. This is the load-bearing weakness, because the abstract's 'state-of-the-art' claim rests on that comparison. The gains over RDED (+5.7) and CV-DD (+4.0) are bigger, but those baselines are known to be weaker, so the comparison that matters is EDC.\n\nThe theory is a secondary issue. Theorem 2's condition (11) is never verified, and the proof contains 'L + h' where it should be 'Lh'. If the inequality doesn't hold, the ARC's theoretical justification collapses. But the empirical story doesn't depend on the theorem, and the mechanism is intuitive enough to stand on its own. The 'first to introduce data-level residual connections' claim is defensible within dataset distillation, though the idea is an obvious analogue of model-level residuals.\n\nThere is also some tuning on the target benchmark: α, k, Dds, and the schedule choice are all selected using the test set. That reduces the significance of the reported numbers, but it's a common practice in this subfield, not a fatal flaw.\n\nWho should read this: anyone working on scalable dataset condensation. The engineering contribution is solid, the code is out, and the efficiency gains are likely reproducible. It deserves serious refereeing, but the evaluation needs to be much more rigorous. If I were the editor, I would send it out with a request for repeated trials, confidence intervals, and a direct like-for-like comparison against EDC under the same protocol. The theory should be either fixed or removed.\n\nAll in all: a useful paper, mildly overclaimed, and one that will get sharper with revision.","headline":"The data-residual idea is worth taking seriously, but the paper's headline margin over EDC rests on best-of-four evaluation with no error bars, so the SOTA claim is not yet trustworthy.","tokens_in":20675,"tokens_out":3068,"would_cite":true,"duration_ms":34058,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By moving residual skip connections from networks to the data itself, FADRM claims a dataset-distillation state of the art: 47.7% and 50.0% top-1 accuracy on ImageNet-1K at 0.8% compression, with compute roughly halved.","keywords":["dataset distillation","data residual matching","data-level skip connections","information vanishing","uni-level optimization","mixed precision training","multi-resolution optimization","ImageNet-1K"],"falsifier":"Run the paper's own ImageNet-1K IPC=10 recipe (ResNet-18 teacher, $\\alpha = 0.5$, $k = 3$, $D_{ds} = 200$) and empirically estimate the two terms of condition (11): the Rademacher complexity of the hypothesis class on the original patches versus on the optimized-only synthetic images, and the average discrepancy $\\Delta$ between them. If the left-hand side of the inequality is not negative enough, the theorem's conclusion does not follow from its stated assumptions. Independently, the information-vanishing mechanism is directly testable: feature-level entropy of the distilled images should fall during optimization and be restored at each residual injection, which a reader could reproduce with any pretrained feature extractor.","tokens_in":19714,"feed_emoji":"🎯","tokens_out":13375,"duration_ms":129275,"temperature":0.7,"pith_summary":"Dataset distillation compresses a huge training set into a tiny synthetic one, but the cheapest large-scale methods optimize the synthetic images without ever looking at the originals, so information about the real data steadily fades — the paper calls this 'information vanishing'. FADRM's central proposal is to carry residual connections over to the data itself: periodically blend the intermediate synthetic image with resized patches of the original images during optimization, so the distilled set keeps both newly learned detail and the original's local structure. Because the blending is cheap and the optimization can be run at reduced resolution with mixed-precision arithmetic, the method claims roughly a 50% cut in training time and peak GPU memory. If the results hold, the payoff is that large-scale, high-quality distilled datasets become substantially cheaper to produce, with a compact set of a few images per class training a model to 47.7-50.0% top-1 accuracy on ImageNet-1K.","feed_headline":"Data residuals push distilled ImageNet past 50% accuracy","feed_subtitle":"A data-level skip connection stops information vanishing while halving the time and memory cost of distillation.","key_machinery":"The central object is the Adjustable Residual Connection (ARC), a data-level skip connection that at each injection stage replaces the intermediate synthetic image with a convex blend of itself and the resampled original patch: $\\tilde{x}_{ib} = \\alpha\\,\\tilde{x}_{ib} + (1-\\alpha)\\,\\mathrm{Resample}(P_s, D_{ib})$. It is the mechanism supposed to stop information vanishing and to deliver the tighter generalization bound, with $\\alpha$ controlling how much original-detail is retained. Two efficiency components carry the cost claims: Multi-Resolution Optimization (MRO), which runs roughly half of the optimization steps at a downsampled resolution such as 200x200 instead of 224x224 and then rescales back, and Mixed-Precision Training (MPT), which computes logits and cross-entropy in FP16 while keeping the divergence-to-statistics loss and the gradients with respect to the image in FP32.","core_discovery":"FADRM claims that data-level residual matching solves the information-vanishing problem of uni-level dataset distillation, where synthesis is driven by matching a pretrained network's running BatchNorm statistics and prediction alignments without direct access to the original data. The paper shows (Theorem 1) that such pipelines are fundamentally information-starved: the mutual information between the synthetic set and the original set is bounded by $|C|\\cdot H(f_\\theta)$, the synthetic-set size times the maximum per-sample entropy of the pretrained network, so an over-confident model caps how much the distilled set can retain. Against this, FADRM injects the original data through an adjustable residual connection (ARC) that repeatedly fuses the optimized image with resampled original patches weighted by a merge ratio $\\alpha \\approx 0.5$, and the paper argues (Theorem 2) that this blending tightens a Rademacher-complexity generalization bound — a measure of how expressive the hypothesis class is — relative to optimizing alone. Empirically the package of ARC plus multi-resolution optimization and mixed-precision training reports state-of-the-art accuracy on CIFAR-100, Tiny-ImageNet, ImageNet subsets, and ImageNet-1K, with headline numbers of 47.7% (single-model) and 50.0% (multi-model) top-1 accuracy on ImageNet-1K at IPC=10 using ResNet-18, together with roughly 50% reductions in optimization time and peak GPU memory.","pith_inferences":["A direct testable extension of the paper's logic: replace the real patch in the residual blend with its pixel-shuffled or Fourier-shuffled version; if accuracy holds, the residual's benefit is statistical (color, energy, texture) rather than semantic, which would tell practitioners what ARC is actually preserving.","The Theorem 1 bound suggests a pragmatic diagnostic for any distillation run: measure the per-class output entropy of the pretrained teacher on the synthetic set; runs sitting near the bound $|C|\\cdot H(f_\\theta)$ are prime candidates for residual injection regardless of the distillation objective used.","Using the paper's own quantities, one can empirically test Theorem 2's condition (11): estimate the Rademacher-complexity difference between the original patch set and the optimized-only synthetic set on a real distillation run; if the inequality fails, the empirical gains stand but the theory, as written, does not yet explain them.","The paper itself flags that data-level residual signals may not transfer to non-visual modalities such as text or time series; applying ARC to those domains, or to iterative synthesis loops like generative replay in continual learning, would clarify how general the mechanism is."],"forward_implications":["If FADRM's claims hold, ImageNet-scale dataset distillation becomes feasible on a single consumer GPU: generating the 50-IPC ImageNet-1K set would take about 54 hours less than EDC and roughly 28 hours less than SRe2L++, with peak memory below 3 GB for the single-model variant.","A distilled set built from about 0.8% of ImageNet-1K (10 images per class) would train ResNet-18 to 47.7% top-1 accuracy, and an ensemble of four teachers (FADRM+) would push that to 50.0%; the same data transfers across eleven tested architectures, from ShuffleNetV2 at 30.2% to DenseNet201 at 59.7%.","Data-level residual connections become a reusable ingredient: any uni-level distillation objective, whether statistics matching, prediction matching, or their combination, can be run with periodic residual injection at negligible extra cost, since ARC is a per-element tensor fusion of complexity $O(H_t W_t C)$.","Information vanishing has a measurable signature — feature-level entropy of the distilled images falls as optimization proceeds and the residual connection restores it — so the mechanism is checkable independently of final accuracy.","The ablations put the optimal settings at $\\alpha = 0.5$ and $k = 3$ residual injections, so the reported headline gains are tied to these specific hyperparameters rather than to the residual idea alone."],"supporting_citations":[{"why":"RDED baseline; supplies the practice of selecting and resampling cropped real patches that FADRM inherits for initialization and residual injection, and the single-model accuracy it must beat.","marker":"[32]"},{"why":"EDC baseline; its uni-level optimization setup and training configuration define the comparison protocol and the efficiency target FADRM claims to halve.","marker":"[30]"},{"why":"CV-DD and SRe2L++ baselines; the committee-voting ensemble and the efficiency numbers that FADRM+ is compared against.","marker":"[6]"},{"why":"ResNet is both the pretrained teacher/student architecture for the headline results and the origin of the residual-connection concept the paper re-targets at data.","marker":"[11]"},{"why":"Mixed precision training; supplies the FP16/FP32 scheme behind the MPT efficiency component.","marker":"[23]"},{"why":"Rademacher complexity generalization bound (Lemma 2) that Theorem 2 builds on.","marker":"[2]"},{"why":"Data processing inequality used in Theorem 1 to bound the information the distilled set can carry about the original set.","marker":"[3]"},{"why":"SRe2L; the uni-level statistics-matching framework whose information-vanishing limitation motivates FADRM.","marker":"[43]"}],"fun_headline_variants":["Data residuals push distilled ImageNet to 50% accuracy","FADRM: residual matching halves distillation time and memory","Data-level skip connections stop info vanishing in distillation","FADRM sets new record with 50% accuracy on distilled ImageNet"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is Theorem 2's condition (11), an inequality requiring that blending in the original patches shrink the model's effective hypothesis class by more than a threshold set by the Lipschitz constants, the loss bound, the merge ratio, and the discrepancy between optimized and real images. The paper never verifies this condition on any actual data or model, and the derivation leading to it contains a slip where '$L + h$' appears in place of the product $Lh$; until the condition is checked and that slip is corrected, the theory alone does not carry the claim that the residual connection improves generalization.","fun_headline_variants_meta":{"raw":{"variants":["Data residuals push distilled ImageNet to 50% accuracy","FADRM: residual matching halves distillation time and memory","Data-level skip connections stop info vanishing in distillation","FADRM sets new record with 50% accuracy on distilled ImageNet"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1670,"prompt_tokens":1098,"completion_tokens":572,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":714,"completion_tokens_details":{"reasoning_tokens":503}},"tokens_in":714,"tokens_out":572,"duration_ms":6292,"temperature":1.0,"reasoning_tokens":503,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:22:55.832834+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's own ImageNet-1K IPC=10 recipe (ResNet-18 teacher, $\\alpha = 0.5$, $k = 3$, $D_{ds} = 200$) and empirically estimate the two terms of condition (11): the Rademacher complexity of the hypothesis class on the original patches versus on the optimized-only synthetic images, and the average discrepancy $\\Delta$ between them. If the left-hand side of the inequality is not negative enough, the theorem's conclusion does not follow from its stated assumptions. Independently, the information-vanishing mechanism is directly testable: feature-level entropy of the distilled images should fall during optimization and be restored at each residual injection, which a reader could reproduce with any pretrained feature extractor.","supporting_citations":[{"cited_title":"Rademacher and gaussian complexities: Risk bounds and structural results","cited_arxiv_id":null,"evidence_quote":"Rademacher complexity generalization bound (Lemma 2) that Theorem 2 builds on."}],"review_version":1}