{"id":"d7427d05-b9a3-40af-b568-0a67b21a8e23","arxiv_id":"2505.14751","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A self-distillation framework that aligns features of original and gradient-perturbed inputs reports an accuracy jump from 22.93% to 41.99% on CIFAR-100 and improved SSIM/FID on CUB.","lead":"This paper proposes a training method that repeatedly perturbs inputs in the direction that reduces the model's loss, then uses features of perturbed inputs as targets for the original inputs via self-distillation. Reported gains on CIFAR-100 and CUB are large, but the evidence lacks error bars, a validation split, and code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Most load-bearing untested assumption: ICP-perturbed features (Eq. 2) are treated as better distillation targets in Eq.","rationale":"The reader's weakest assumption identifies exactly the point I find most load-bearing: there is no independent verification that features from ICP-perturbed inputs are improved supervision targets. My reading of Eqs. (2)-(3) confirms this is an assumption, not a derived property. The paper's empirical evidence is suggestive but confounded: the ICP methods change the training schedule, add input-gradient computation, and select hyperparameters on the test set, so the reported accuracy gap does not isolate the proposed mechanism. The duplicated table entry and lack of error bars further reduce confidence. The proposed concrete test—comparing ICP against random, adversarial, and null perturbations while controlling for schedule—would settle whether the 'constructive' direction is what matters, or whether the gain comes from a generic regularizer or from extra compute. Because the reader already reached a CONDITIONAL verdict and my concern reinforces rather than overturns that judgment, I recommend leaving the verdict unchanged. I agree with the reader's weak-assumption identification and do not see a different, more central flaw that would justify rejection; the method is coherent and the experiments are reproducible in principle, but the mechanism remains unproven.","tokens_in":7492,"tokens_out":3933,"duration_ms":41605,"concrete_test":"On CIFAR-100 with the same k=25, T=5, Weighted=True setup, compare ICP against three matched controls: (a) random Gaussian perturbation with the same L2 norm as the ICP perturbation, (b) gradient-ascent (i-FGSM) perturbation of the same magnitude, and (c) self-distillation with the unperturbed input (F'_i = F_i). If any control reaches within a few points of 41.99%, the constructive direction of the perturbation is not the cause. Also train a k=25 baseline and continue plain training to epoch 100 with no distillation to separate the schedule change from the proposed loss, and report a linear-probe accuracy (or similar) of F'_i versus F_i on held-out data to verify that perturbed features are actually better targets. Run 3 seeds with standard deviations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2.1-3.2.2 defines I' by T steps of gradient descent on the current model's task loss (Eq. 2), then sets L_dist^i = MSE(F_i, F'_i) (Eq. 3). This makes F'_i the 'teacher'. However, F'_i is computed by the same network on an input attracted toward lower loss under the current parameters; it is not an independent or better hypothesis. The only claimed justification is the toy illustration in Figure 1, which is qualitative, uses a different model and data distribution than the CIFAR-100 experiments, and does not measure feature quality. Consequently, the observed CIFAR-100 improvement (41.99 vs 22.93) could equally be explained by (i) the schedule change after k=25 baseline epochs, (ii) the extra per-batch input-gradient computation, or (iii) implicit regularization from MSE feature alignment, without the input perturbation being 'constructive'. The table also contains a duplicated value (26.27 appears for both AdEMAMix-ICP k=50/T=10/Weighted=True and k=75/T=5/Weighted=False), and hyperparameters are selected using the test set, which further weakens causal attribution. As written, the central mechanism claim—that minimizing task loss on inputs produces better supervision targets—is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Iterative Constructive Perturbation (ICP), which applies several gradient-descent steps to the input in the direction that minimizes the model's task loss, then feeds the perturbed input through the network and uses the resulting intermediate feature maps as self-distillation targets for the original input's features. The total loss is a cosine-decayed mixture of the task loss and layer-wise MSE feature-alignment losses. Experiments on CIFAR-100 classification with a modified ResNet20 and on CUB image generation with a small VAE report consistent improvements over a 100-epoch control, with the best configuration (k=25, T=5, Weighted=True, AdEMAMix-ICP) achieving 41.99% accuracy versus 22.93% for the control and SSIM 0.3893 versus 0.2580 on CUB.","tokens_in":7790,"tokens_out":5776,"duration_ms":57790,"significance":"The proposed idea—using input-space gradient descent to create feature targets for self-distillation—is potentially interesting and would be a useful addition to the self-distillation literature if the mechanism were convincingly established. The paper deserves credit for comparing three input optimizers (SGD-, Adam-, and AdEMAMix-ICP), reporting wall-clock times, and for attempting a transfer to a generative task. However, the current evidence does not isolate the proposed mechanism from several confounds: the teacher features are generated by the same network on its own gradient-adjusted inputs, hyperparameters are selected on the test set, every reported number is a single run with no error bars, and Table 1 contains a duplicated value. As written, the central claim that minimizing the task loss on inputs produces better supervision targets is not supported. The results are promising enough to warrant a major revision with controlled ablations and more rigorous evaluation, rather than outright rejection.","major_comments":[{"comment":"The central mechanism is not independently verified. The teacher features F'_i are computed by the same network on an input I' that has been moved along the network's own task-loss gradient, so the distillation targets are self-generated rather than externally validated. The only support offered is Figure 1, which is a toy two-dimensional MLP example on isotropic Gaussian clusters and does not measure feature quality or transfer to the CIFAR-100/ResNet20 setting. Please add ablations that replace the ICP perturbation with (i) random Gaussian input noise, (ii) adversarial (loss-increasing) perturbations, and (iii) a different self-distillation target such as features from a differently augmented view of the same input or from a stale copy of the network, while keeping the loss schedule and feature-alignment loss identical. Without these controls, the observed improvement could be due to the extra input-gradient computation or to the MSE feature-alignment regularizer rather than to 'constructive' perturbations.","section":"Section 3.2.2, Eq. (3)"},{"comment":"The main experimental table does not support the headline claim because hyperparameters are selected on the test set. The text states that the optimal configuration (k=25, T=5, Weighted=True) was determined from Table 1, but no validation split is described; this makes 41.99% a test-set-selected result and risks overfitting to the test set. Please use a separate validation split for selecting k, T, and the Weighted flag, and report test accuracy with means and standard deviations over at least three seeds. In addition, Table 1 contains duplicated entries: AdEMAMix-ICP with k=50, T=10, Weighted=True and with k=75, T=5, Weighted=False both report 26.27% accuracy and 0.249 F1. This strongly suggests a data-entry error and must be corrected before the table can be used as evidence.","section":"Section 4.2, Table 1"},{"comment":"The comparison against the 100-epoch control confounds the method with the training schedule. The control is trained with alpha_e = 1 for all 100 epochs, while every ICP run trains for k baseline epochs and then switches to the cosine-decayed combination of task loss and feature-alignment loss. The reported improvement over the control could therefore be caused by the change in the objective after epoch k (including the MSE alignment term or the cosine schedule) rather than by the input perturbation itself. Please include a baseline that uses the same alpha_e schedule and the same feature-alignment loss but with the perturbed input replaced by an unperturbed or randomly perturbed input, and report that comparison. This is the minimal experiment needed to attribute the observed gain specifically to ICP.","section":"Section 3.3, Eqs. (4)-(5)"},{"comment":"Key implementation details needed for reproducibility are missing. The ICP step size epsilon for the CIFAR-100 experiments is never given (epsilon=0.002 is mentioned only for the toy example in Figure 1), the identity and number of distilled layers are unspecified, the 'modified ResNet20' architecture is not described, and the VAE architecture and its loss weighting are not provided. Without these details, the experiments cannot be reproduced, and the sensitivity of the method to epsilon and layer choice remains unknown. Please report all hyperparameters, specify the network layers used for distillation, describe the VAE architecture, and ideally release code. This is load-bearing because the method's only empirical support is these experiments.","section":"Sections 4.1 and 4.3"}],"minor_comments":[{"comment":"The phrase '19.06% more than the control baseline' is misleading; the reported numbers are percentage points (41.99% minus 22.93% equals 19.06 percentage points), not a relative 19.06% improvement.","section":"Section 4.2"},{"comment":"There is a missing space in 'andsgn' after 'and'.","section":"Eq. (1)"},{"comment":"The citations for BYOT and Born Again Neural Networks appear mismatched: reference [25] is cited as BYOT but is actually 'Deep Mutual Learning', while 'Born Again Neural Networks' is reference [4]. Please correct the citation mapping.","section":"Introduction and References"},{"comment":"The Adam-ICP and AdEMAMix-ICP update rules are typeset ambiguously; the bias-correction factors and the division by sqrt(v) should be written with explicit fractions so the update equations are unambiguous.","section":"Eqs. (8) and (11)"}],"recommendation":"major_revision","confidential_remarks":"The paper's main evidence is a single table with one run per configuration and no validation set; the duplicated row suggests data entry problems. The editor may wish to require the authors to provide code, multi-seed results, and controlled ablations (random-noise and adversarial-perturbation baselines) before resubmission. I do not see an internal inconsistency in the derivation, but the empirical support is currently too weak for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a plausible but under-evidenced paper. The central trick—take T gradient-descent steps on the input to lower the current model's loss, then use the resulting feature maps as self-distillation targets—is a natural combination of FGSM-style input perturbation and self-distillation, and I don't see it in the cited literature. The authors also report wall-clock times and try a VAE transfer task, which is more than many such papers do. The toy example in Figure 1 is a nice sanity check that ICP moves points away from boundaries, though it is qualitative and not the CIFAR setting.\n\nThe soft spots are real and load-bearing. First, the mechanism claim is untested: nothing in the paper shows that I' features are better targets than the original features. The only motivation is that the input was moved along the negative task-loss gradient. That could help, but it could also just be a consistency regularizer plus an extra gradient computation. The loss schedule change after k=25 and the extra per-batch input-gradient cost are confounds; the observed gains don't isolate ICP. Second, the evidence table is thin: one run per configuration, no error bars, no validation set, hyperparameters chosen on the test set. The baseline control at 22.93% on CIFAR-100 is far below what a standard ResNet20 should reach, so the reported 19-point gain is measured from an unusually weak reference point. Third, there is an apparent data-entry error in Table 1: AdEMAMix-ICP k=50/T=10/Weighted=True and k=75/T=5/Weighted=False both show Acc 26.27 and F1 0.249. That kind of duplication makes it hard to trust the rest of the table.\n\nThe VAE experiment is better than nothing, but the FID gains are small (161.8 to 157.6) and the authors admit there's little visual difference. Not a strong independent check.\n\nSo: is this worth a serious referee? I think yes, conditionally. The idea is concrete, the formulation is clear, and the flaws are fixable. But I would not accept the paper in this form. The authors need to release code, run multiple seeds, use a proper validation split, report standard deviations, fix the table, and include a control that isolates ICP from the schedule and from plain feature alignment. For a reading group, it's a useful case study in how a plausible idea can outrun its evidence. I wouldn't cite it yet.","headline":"Plausible new self-distillation trick, but the central 'constructive' claim is untested and the evidence table is too thin to support the headline 19-point gain.","tokens_in":8309,"tokens_out":2436,"would_cite":false,"duration_ms":24862,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a cyclic optimization scheme in which inputs are iteratively perturbed to reduce the current task loss and then used as self-distillation teachers for their own features, and reports large accuracy gains on CIFAR-100…","keywords":["iterative constructive perturbation","self-distillation","input optimization","feature alignment","CIFAR-100","image generation","AdEMAMix","knowledge distillation"],"falsifier":"Run the exact k=25, T=5, weighted-feature protocol on CIFAR-100, but replace the ICP update with (a) a fixed random perturbation of matched norm per step, or (b) an i-FGSM adversarial perturbation of matched magnitude. If either replacement reproduces the 41.99% accuracy, the constructive (loss-minimizing) direction of ICP is not what carries the benefit; if neither does, the gradient-descent direction is doing the work. A second check is to record the task loss on the perturbed input across the T steps and verify it actually decreases as the constructive claim requires.","tokens_in":7322,"feed_emoji":"📈","tokens_out":9482,"duration_ms":81657,"temperature":0.7,"pith_summary":"The paper proposes a training scheme in which the input data and the model are optimized alternately: for the first k epochs the model trains normally, then each input is repeatedly nudged in the direction that lowers the current task loss, and the intermediate features of this nudged input are used as a self-distillation target for the original input's features. The authors claim this closes part of the fitting-generalization gap without a teacher network, extra parameters, or architectural changes. Their main evidence is that on CIFAR-100, with k=25 baseline epochs and T=5 perturbation steps, AdEMAMix-ICP reaches 41.99% accuracy against 22.93% for a 100-epoch control, and that the same recipe improves a VAE's SSIM and FID on CUB. The reason to care is that the method is orthogonal to architecture and optimizer choices, so if the claim holds it gives practitioners a cheap, plug-in way to improve existing training runs.","feed_headline":"Input tweaks lift CIFAR-100 accuracy from 23% to 42%","feed_subtitle":"Same 100-epoch budget, no teacher, no extra parameters: classification and generation improve in the paper's tests.","key_machinery":"The load-bearing mechanism is the Iterative Constructive Perturbation (ICP) update, a gradient-descent step on the input instead of on the weights: $x_t = x_{t-1} - \\epsilon \\nabla_{x_{t-1}} J(\\theta, x_{t-1}, y)$. This reverses FGSM's logic in two ways: it uses the full gradient rather than its sign, and it minimizes the loss rather than maximizing it, over $T$ refinement steps. The refined input is then passed through the model a second time, and the mean-squared error between original and refined feature maps at selected layers, $\\mathrm{MSE}(F_i, F'_i)$, is added to the task loss with a cosine-decayed mixing weight $\\alpha_e$. The paper also makes the perturbation step optimizer-dependent, replacing plain SGD with Adam-style moment estimates (Adam-ICP) and with AdEMAMix's extra momentum term (AdEMAMix-ICP), and reports that the choice of this inner optimizer changes the results enough to matter.","core_discovery":"The paper's central claim is that features extracted from a loss-minimizing perturbed version of an input are better supervision targets than the original input's own features, and that aligning the two with a layer-wise distillation loss is enough to improve task performance. Concretely, after k baseline epochs the combined training loss becomes $L_{\\mathrm{total}} = \\alpha_e L_{\\mathrm{task}} + (1-\\alpha_e) \\sum_i \\mathrm{MSE}(F_i, F'_i)$, where $F_i$ are features of the original input and $F'_i$ are features of the ICP-refined input, with $\\alpha_e$ cosine-decaying from 1 to 0 over the remaining epochs. The authors report that on CIFAR-100 the best configuration (k=25, T=5, weighted feature maps) lifts accuracy from 22.93% (100-epoch control) to 41.99% for AdEMAMix-ICP, a 19.06-percentage-point gain, with F1 rising from 0.229 to 0.414. In their constrained VAE experiment on CUB, the same configuration improves SSIM from 0.2580 to 0.3893 and FID from 161.830 to 157.604. The paper interprets these results as evidence that alternating optimization of weights and inputs reduces the gap between fitting and generalization.","pith_inferences":["A natural reading the paper does not spell out is that ICP's main effect is to act as a data-dependent regularizer: tying original features to features of a loss-minimizing perturbation discourages the model from relying on directions that are unstable under small input shifts.","A direct way to test this reading is to ablate the distillation term while keeping the ICP-refined input as the only change, or to replace the ICP direction with a random or adversarial perturbation of the same magnitude; the reported table does not isolate the distillation contribution on its own.","Because the authors fix the total epoch budget at 100, a comparison on larger models and datasets, for instance transformer-based vision models or ImageNet-scale inputs, would show whether the gain survives outside the small-model, small-data regime they tested."],"forward_implications":["With k=25 and T=5, AdEMAMix-ICP improves CIFAR-100 accuracy by 19.06 percentage points over a 100-epoch control run while keeping the same total epoch budget.","Weighting deeper layers more heavily in the feature-alignment loss yields better classification results than uniform weighting in the paper's ablation.","The same hyperparameter configuration transfers to image generation, improving both SSIM and FID for a VAE trained on CUB.","The method needs no teacher model and no extra parameters; the distillation target is produced by the same network from a perturbed copy of the input."],"supporting_citations":[{"why":"Supplies the gradient-based perturbation logic that ICP reverses, replacing sign-based loss maximization with full-gradient loss minimization.","marker":"[6]"},{"why":"Supplies the ResNet20 architecture used for the CIFAR-100 classification experiments.","marker":"[7]"},{"why":"Supplies the FID metric used to evaluate generated images.","marker":"[8]"},{"why":"Establishes the knowledge-distillation setting that the self-distillation objective extends.","marker":"[9]"},{"why":"Supplies the Adam moment estimates used to define the Adam-ICP perturbation variant.","marker":"[11]"},{"why":"Supplies the VAE objective used in the image-generation evaluation.","marker":"[12]"},{"why":"Supplies the CIFAR-100 benchmark on which the accuracy and ablation results are measured.","marker":"[13]"},{"why":"Motivates the use of a model's own features as a training signal by showing self-distillation amplifies regularization.","marker":"[17]"},{"why":"Supplies the AdEMAMix optimizer whose extra momentum term defines the AdEMAMix-ICP variant.","marker":"[18]"},{"why":"Supplies the CUB dataset used for the VAE image-generation experiment.","marker":"[20]"}],"fun_headline_variants":["ICP: Boost CIFAR-100 by 19 points via input self-distillation","Perturb then align: self-distillation lifts CIFAR-100 to 42%","No teacher, no extra params: ICP self-distillation gains 19 pts","Self-distill on perturbed inputs to lift CIFAR-100 to 42%","Perturbed inputs create better targets, lifting CIFAR-100 to 42%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the input produced by T steps of gradient descent on the model's task loss is a genuinely better view of the same input, with richer or more stable features, rather than just a sample that happens to lower the current loss; if that premise fails, the distillation term aligns the model with its own noise instead of with better supervision.","fun_headline_variants_meta":{"raw":{"variants":["ICP: Boost CIFAR-100 by 19 points via input self-distillation","Perturb then align: self-distillation lifts CIFAR-100 to 42%","No teacher, no extra params: ICP self-distillation gains 19 pts","Self-distill on perturbed inputs to lift CIFAR-100 to 42%","Perturbed inputs create better targets, lifting CIFAR-100 to 42%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000714,"raw_usage":{"total_tokens":3234,"prompt_tokens":995,"completion_tokens":2239,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":2126}},"tokens_in":611,"tokens_out":2239,"duration_ms":13436,"temperature":1.0,"reasoning_tokens":2126,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:35:55.889358+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact k=25, T=5, weighted-feature protocol on CIFAR-100, but replace the ICP update with (a) a fixed random perturbation of matched norm per step, or (b) an i-FGSM adversarial perturbation of matched magnitude. If either replacement reproduces the 41.99% accuracy, the constructive (loss-minimizing) direction of ICP is not what carries the benefit; if neither does, the gradient-descent direction is doing the work. A second check is to record the task loss on the perturbed input across the T steps and verify it actually decreases as the constructive claim requires.","supporting_citations":[{"cited_title":"Deep residual learning for image recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the ResNet20 architecture used for the CIFAR-100 classification experiments."},{"cited_title":"Learning multiple layers of features from tiny images.(2009), 2009","cited_arxiv_id":null,"evidence_quote":"Supplies the CIFAR-100 benchmark on which the accuracy and ablation results are measured."},{"cited_title":"Self-distillation amplifies regularization in hilbert space","cited_arxiv_id":null,"evidence_quote":"Motivates the use of a model's own features as a training signal by showing self-distillation amplifies regularization."}],"review_version":1}