{"id":"05204ba5-18d7-43ce-969f-60147ef890c2","arxiv_id":"2608.04442","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Shallow layers are most robust to corruptions early in training; freezing them (EPS) or rewinding them (AWR) reduces corruption error on several benchmarks, with caveats.","lead":"This paper reports that shallow layers of vision networks become more robust to image corruptions early in training and then lose that robustness, and proposes freezing or rewinding those layers to keep the robust state. It matters because the fix adds no learnable parameters and is claimed to improve corruption robustness across many tasks, but the supporting evidence is not yet conclusive.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Robustness-fading evidence does not yet exclude the trivial null that early shallow features are merely under-specialized: CKA/MI similarity and frozen-landscape flatness need matched under-specialization controls before the central mechanism can be accepted.","rationale":"The reader's weakest_assumption identifies exactly the confound I consider load-bearing: early-feature similarity and flatness may reflect under-specialization rather than learned robust structure. This is the right place to attack because the entire intervention framework is motivated by the Sec. 3 diagnosis; if high early CKA/MI and frozen-landscape flatness are trivial properties of under-trained features, then EPS/AWR are not 'preserving robust priors' but are simply early-stopping or regularizing the shallow layers, and the claimed mechanism is not established even if the empirical gains survive. I considered whether the more damaging issue is the confusing Table 2/3 reporting (mCE vs. accuracy) or the absence of code/data/seeds; those are genuine but they affect verifiability, not the central mechanism. The permutation control is a decisive and cheap experiment: it preserves all scale and under-specialization properties of early theta_s while destroying learned structure, so it isolates the causal contribution of learned configuration. The verdict should remain CONDITIONAL: the paper is not internally inconsistent, but the central claim needs this additional control plus released code and data before it can be accepted. Thus no change to the reader's verdict is required.","tokens_in":13855,"tokens_out":6915,"duration_ms":64506,"concrete_test":"Run a null control for Figs. 3 and 4 on the smallest reported setting: for ResNet-18/50 checkpoints at epochs {1, 5, 10, 20, 40, 90}, build a matched null by randomly permuting weights within each shallow layer (preserving per-layer L2 norm and channel-wise scale), then recompute clean-vs-corrupted CKA/MI and rerun the freezing protocol (freeze permuted vs. actual shallow weights, retrain deep, measure Delta L_max). If the permuted early checkpoints match or exceed the reported early CKA/MI and flatness, the early peak is an under-specialization artifact, not a learned robust prior; if actual checkpoints separate clearly from the null early and converge to it later, the diagnosis survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Sec. 1) is that shallow layers spontaneously develop robust representations and flat loss landscapes early in training but lose them during convergence. The load-bearing evidence is Figs. 3 and 4: clean-vs-corrupted CKA/MI curves and landscape flatness after freezing early shallow weights. Both are vulnerable to the same confound: early theta_s may not contain a learned robust prior at all; it may simply be under-specialized. Early features have low effective rank and small norms, so CKA between clean and corrupted inputs can be high for reasons unrelated to robustness, and as training specializes features CKA naturally drops even with no loss of robustness. The InfoNCE MI estimator is also sensitive to feature dimensionality and batch statistics, so cross-depth comparisons are not calibrated. The freezing experiment (Sec. 3.3) compares actual early checkpoints against actual late checkpoints, but not against a null control that preserves early-training scale and under-specialization while destroying learned structure, e.g., random permutation of the same weights. Without that control, the flatter landscape after freezing early weights is equally explained by early weights being closer to initialization and therefore acting as a regularizer that prevents deep layers from over-sharpening the loss, rather than by an intrinsic robust configuration. Since EPS and AWR are motivated entirely by this diagnosis, the central claim is not settled until these confounds are ruled out. The Table 2/3 mCE-vs-accuracy inconsistencies are genuine but secondary; the mechanism concern is primary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies the training dynamics of shallow-subnetwork robustness to natural corruptions. It claims that shallow layers acquire robust, corruption-stable representations and flat loss landscapes early in training, but that standard ERM convergence erodes these properties, a phenomenon the authors call 'robustness fading.' Based on this diagnosis, the paper proposes two intervention strategies: Early-Phase Stabilization (EPS), which freezes the shallow subnetwork after an early epoch, and Asymmetric Weight Reversion (AWR), which rewinds the shallow subnetwork to an early snapshot and reduces its learning rate while the deep subnetwork continues training. The authors report gains over representative baselines on ImageNet-C and related corruption benchmarks, across several architectures, and on downstream detection, segmentation, and test-time adaptation tasks. The paper is entirely empirical and does not provide code, seeds, or error bars; all headline results are reported as single numbers.","tokens_in":14194,"tokens_out":6575,"duration_ms":57596,"significance":"If the central finding were established, the paper would make a useful contribution: it connects training dynamics to corruption robustness and offers a simple, architecture-agnostic intervention with no architectural modifications or extra learnable parameters. The evaluation is broad, covering classification, dense prediction, and online TTA, and the freezing experiments in Sec. 3.3 are a reasonable causal probe. However, the central diagnostic evidence is currently confounded by the under-specialization of early features, and the headline claims are not fully supported by the reported numbers once clean-accuracy trade-offs and table inconsistencies are taken into account. The paper does not ship code or machine-checked proofs, so its contribution rests entirely on the empirical claims, which need stronger controls and more careful reporting.","major_comments":[{"comment":"The evidence for 'robustness fading' does not control for early-feature under-specialization. CKA and InfoNCE-based MI between clean and corrupted features can be high in early training simply because early features have low effective rank, small norms, and undeveloped selectivity; the subsequent decline can occur even without any loss of a learned robustness property. The InfoNCE estimator is also sensitive to feature dimensionality and batch statistics, so cross-depth comparisons are not calibrated. The freezing experiment in Sec. 3.3 has no null control: freezing early weights and retraining deep layers may act as a regularizer simply because early weights are close to initialization. A control that permutes weights within the early checkpoint, or otherwise destroys learned structure while preserving scale and under-specialization, is needed before the diagnosis can support the proposed interventions.","section":"Sec. 3.2, Figs. 3 and 4"},{"comment":"The text states that EPS and AWR 'consistently yield additive gains' across augmentation and regularization baselines, but clean Top-1 accuracy decreases in nearly every integrated setting. For example, CutMix clean accuracy drops from 76.9 to 74.3 (EPS) and 74.4 (AWR); Label Smoothing drops from 76.6 to 74.5 and 74.7; AutoAug drops from 76.4 to 74.3. The reported Avg. mCE reductions are small (0.5-1.9 points) and are given as single numbers without error bars or significance tests. The claim should be reframed as a clean-accuracy/robustness trade-off, and the mCE differences need variance estimates.","section":"Sec. 5.2, Table 3"},{"comment":"The methods are described as 'parameter-free,' but they depend on intervention hyperparameters: the EPS epoch tau_eps, the AWR snapshot epoch t_early, the AWR reversion epoch tau_awr, and the shallow learning rate eta_s. These hyperparameters are analyzed via sensitivity studies in Sec. 5.4 on the same corruption benchmarks (IN100-C, IN100-Cbar, IN100-3DCC) used for the headline comparisons. Unless the hyperparameters were fixed on a held-out set or selected with a principled criterion before seeing the evaluation sets, the reported gains are optimistic. The authors should clarify the selection protocol and replace the 'parameter-free' terminology with 'no learnable parameters.'","section":"Sec. 5.4 and Algorithm 1"},{"comment":"All experimental results are reported as single numbers with no seeds, standard deviations, or code release. Because the paper's contribution is empirical and several reported improvements are small (for example, Table 3 and Table 6), this level of reporting is insufficient to support the claim of consistent improvement. At minimum, the authors should provide seeds and confidence intervals for the main tables.","section":"Reproducibility, all experiments"},{"comment":"The main comparison tables are not self-contained. Table 1 lacks clear column labels and contains a stray token; Table 2 claims to report both Top-1 accuracy and mCE but does not clearly indicate which columns are which, and the MobileNetV2 row shows an IN-100-C value of 95.8 with clean accuracy 84.6, which cannot both be Top-1 accuracy; Table 3 mixes clean accuracy with mCE under a single header. A reader cannot verify the reported gains without this information.","section":"Tables 1-3"}],"minor_comments":[{"comment":"The axes and the precise definition of subnetwork indices (e.g., #1, #10, #40, #49) are not given in the main text, and the linear-probing setup is described only by a citation; please provide these details.","section":"Fig. 1"},{"comment":"The InfoNCE-based MI estimator is sensitive to feature dimensionality and batch statistics; the paper should justify why cross-depth comparisons are meaningful without calibration.","section":"Sec. 3.2"},{"comment":"The intervention operator I is introduced but never formally instantiated for EPS or AWR; the authors should show explicitly how Eq. (2) encodes Eqs. (3) and (4).","section":"Eq. (2)"},{"comment":"The text says the framework 'consistently boosts performance,' but clean mAP/mIoU drops in several rows, e.g., FasterRCNN COCO from 37.6 to 37.0 and DeepLabV3+ ADE20K S from 42.1 to 41.0; please report the clean/robust trade-off explicitly.","section":"Sec. 5.3, Tables 4 and 5"},{"comment":"The online TTA gains are small (0.1 to 2.6 points) and are reported without error bars; please add variance or state that differences are not statistically tested.","section":"Sec. 5.3, Table 6"},{"comment":"There are formatting typos such as 'T able 1', 'T able 2', and the stray '∗P' token in Table 1; please proofread the final version.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central idea is potentially publishable, but the diagnostic evidence needs a matched under-specialization control before the 'robustness fading' mechanism is accepted, and the evaluation tables need clearer labeling and error bars. The hyperparameter-selection concern is not fatal if the authors can show the intervention choices were made without peeking at the test sets. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWhat you should know: this paper has a useful empirical hook and a simple, parameter-free intervention pair, but its central diagnosis is not yet established. The authors show that if you freeze or rewind shallow-layer weights to an early-training state, you get consistent mCE improvements on corruption benchmarks across many architectures and tasks. That is a real, practical result worth taking seriously, and the breadth of evaluation (classification, detection, segmentation, TTA, ACDC) is a genuine strength. The 'robustness fading' framing is also nice: it connects critical learning periods and robust overfitting into a layer-wise story.\n\nWhat is actually new: the specific characterization that shallow-layer natural-corruption robustness peaks early and decays, plus the EPS/AWR intervention pair. The ingredients are close to progressive layer freezing and LTH rewinding, so novelty is moderate, but the combination aimed at corruption robustness is not in the cited literature.\n\nNow the soft spots, in proportion. First, the load-bearing mechanism evidence in Figs. 3 and 4 is vulnerable to the under-specialization confound. Early features are low-rank and small-norm; CKA and MI similarity between clean and corrupted inputs can be high simply because the features are not yet task-specific, not because they are robust. The freezing experiment lacks a null control (e.g., randomly permuted early weights) that preserves scale and under-specialization while destroying learned structure. The stress-test note lands: without that control, the flatness evidence is equally explained by early weights being near initialization and acting as a regularizer. So the paper's central claim, that 'shallow layers spontaneously develop robust representations,' is plausible but not proven.\n\nSecond, the reporting is sloppier than it should be. No code, seeds, or error bars; Table 2 mixes Top-1 and mCE without clear labels; Table 1 has a stray token. More substantively, Table 3 shows clean accuracy dropping by 2-3 points across the board while mCE gains are tiny (often below 1). That is a robustness-accuracy tradeoff the paper does not discuss. The text says 'consistently yield additive gains' but the clean accuracy losses are not negligible.\n\nThird, the intervention hyperparameters are tuned via sensitivity analyses on the same corruption benchmarks used for the headline comparisons, which inflates the apparent gains. A real evaluation would hold out the main benchmark or use fixed defaults.\n\nBottom line: the paper deserves a serious referee, but the mechansim needs controls, the tradeoff needs reporting, and the code/data need to be released. I'd send it to review with a request for major revision, and I'd focus the referee's attention on the freezing null control and the clean-accuracy loss.\n\nWould I bring it to reading group? Maybe, mostly to discuss the under-specialization confound and the tradeoff. Would I cite it? Not yet, until the mechanism holds up. But the EPS/AWR trick is the kind of thing that could be useful if it reproduces, so I would not desk-reject.","headline":"Plausible and practically motivated, but the central mechanism claim is under-supported because the early-robustness evidence lacks controls for under-specialization, and the reported gains come with clean-accuracy drops that the paper glosses over.","tokens_in":14701,"tokens_out":1851,"would_cite":false,"duration_ms":18533,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Shallow layers build their most robust representations early in training, but standard convergence destroys them.","keywords":["robustness fading","natural corruptions","training dynamics","shallow subnetwork","Early-Phase Stabilization","Asymmetric Weight Reversion","representation stability","loss landscape flatness"],"falsifier":"Train the same architectures while freezing a randomly initialized or adversarially shuffled shallow subnetwork at the same early epoch; if such a control matches the corruption robustness of EPS and AWR, then the reported gains come from stopping shallow updates per se, not from preserving specifically robust early weights. Alternatively, measure CKA and mutual information between clean and corrupted features for early shallow weights after the deep subnetwork has been retrained to convergence; if early features no longer look robust after deep retraining, then the freezing experiments are not isolating the effect of $\\theta_s$.","tokens_in":13651,"feed_emoji":"🛡️","tokens_out":6495,"duration_ms":51989,"temperature":0.7,"pith_summary":"This paper claims that the shallow layers of a deep network do their best corruption-robustness work very early in training, and that ordinary training to convergence actively erases that advantage. The authors call this \"robustness fading.\" They show with representation-similarity, mutual-information, and loss-landscape measurements that early shallow features are more stable under natural corruptions and sit in flatter minima than late-training shallow features. From that diagnosis, they propose two parameter-free interventions, Early-Phase Stabilization and Asymmetric Weight Reversion, that freeze or rewind the shallow subnetwork while letting deeper layers keep fitting, and report consistent gains in corruption robustness across architectures and downstream tasks. If true, the paper turns robustness from a property to be built into architectures into a resource already present early in training and worth deliberately preserving.","feed_headline":"Robustness emerges early in training, then fades","feed_subtitle":"Two parameter-free tweaks freeze or rewind early shallow weights, cutting corruption error across models and tasks.","key_machinery":"The load-bearing object is the decomposition of the network into a shallow subnetwork $\\theta_s$ and a deep subnetwork $\\theta_d$, together with the robustness-fading diagnosis that $\\theta_s$'s best configuration occurs early. The mechanism that carries the interventions is a trajectory operator that constrains $\\theta_s$ after a threshold epoch: EPS sets $\\Delta\\theta_s^{(t)} \\leftarrow 0$, freezing the early robust state, while AWR resets $\\theta_s$ to a snapshot from epoch $t_{\\mathrm{early}}$ and lowers its learning rate $\\eta_s$, letting $\\theta_d$ adapt around it. The diagnostics that carry the argument are layer-wise linear probing, CKA and InfoNCE-based mutual information between clean and corrupted features, and freezing experiments that plot the loss-landscape sharpness of models anchored at early versus late shallow weights. These measurements are what link early weight states to flat geometry and corruption stability, motivating the interventions.","core_discovery":"Under natural corruptions, the shallow subnetwork spontaneously develops robust representations and a flat loss landscape in the early phase of training, but these properties are not preserved during standard convergence. The evidence is layer-wise: CKA similarity and InfoNCE-based mutual information between clean and corrupted features decline in early layers across training, freezing shallow weights from early epochs and retraining the rest yields flatter loss surfaces, and later-phase shallow weights produce sharper landscapes. The paper therefore formulates robustness enhancement as a trajectory-intervention problem: modulate the shallow updates after an intervention epoch instead of modifying the architecture. Two parameter-free strategies instantiate this: EPS nullifies shallow updates after a chosen epoch, and AWR rewinds the shallow parameters to an early robust snapshot and shrinks their learning rate while deeper layers continue to adapt. On ImageNet-C, AWR with a strong augmentation recipe raises Top-1 accuracy from 39.2 to 46.5, and the interventions reduce mean Corruption Error across CNN, ViT, and Mamba backbones and improve detection, segmentation, and online test-time adaptation.","pith_inferences":["If robustness is an early-training resource, then late-phase fitting to clean data may be the main culprit behind its loss; a testable extension is whether learning-rate schedules that slow late-phase fitting reduce the need for explicit freezing or rewinding.","The same preserve-the-early-snapshot logic could apply to other distribution shifts, such as domain shift or label noise, although the paper only claims natural corruptions; applying EPS and AWR to domain-generalization benchmarks would test that.","Because the interventions add no learnable parameters, they could be combined with pruning or lottery-ticket-style rewinding to separate the robustness contribution of the shallow subnetwork from the trainability contribution of the deep subnetwork.","The early-peak-then-fade trajectory suggests a cheap diagnostic: monitoring CKA or loss sharpness of shallow layers during training could serve as an early-warning signal that final robustness will be poor, guiding training adjustments before corruption benchmarks are run."],"forward_implications":["Corruption robustness of a standard trained network is largely determined by whether the shallow subnetwork's early-phase configuration survives; deep-layer training alone cannot recover it once shallow robustness has faded.","EPS and AWR give consistent robustness gains on ImageNet-C and related benchmarks, with AWR reaching 46.5% Top-1 accuracy, a gain of 7.3 points over the baseline.","The gains transfer across tasks: object detection on COCO-C, semantic segmentation on Cityscapes-C and ADE20K-C, and several online test-time adaptation methods all improve when EPS or AWR is integrated.","The interventions are compatible with data augmentation and regularization, adding mean Corruption Error reductions on top of AugMix, AutoAug, CutMix, label smoothing, and dropout.","The mechanism works across CNN, ViT, and Mamba architectures, making it an architecture-agnostic way to improve corruption robustness without adding parameters."],"supporting_citations":[{"why":"Supplies the linear probing method used to show that shallow layers keep a large robustness gap across training methods.","marker":"[2]"},{"why":"Provides CKA, the similarity metric that tracks declining clean-corrupted feature alignment in shallow layers.","marker":"[30]"},{"why":"Provides InfoNCE-based mutual information estimates used to measure information preservation under corruption.","marker":"[45]"},{"why":"Gives the weight-rewinding mechanism that AWR adapts to revert shallow parameters to an early robust state.","marker":"[16]"},{"why":"Shows that a hard-coded V1-like front-end boosts robustness, supporting the premise that shallow layers have untapped robust capacity.","marker":"[11]"},{"why":"Documents adversarial robust overfitting, the early-peak-then-decline pattern that motivates studying natural-robustness dynamics.","marker":"[51]"},{"why":"Defines the ImageNet-C benchmark and corruption error, the main evaluation protocol for the paper's claims.","marker":"[22]"},{"why":"AugMix serves as a representative augmentation baseline whose shallow layers remain unimproved in the layer-wise probing comparison.","marker":"[23]"}],"fun_headline_variants":["Robustness fades in training; freeze shallow weights to keep it","Early robust features vanish; rewind shallow weights","Freeze early shallow weights to stop robustness fade","Robustness emerges early, fades later; two parameter-free fixes","Robustness fades during convergence; rewind shallow weights"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the high early CKA and mutual-information values and the flat frozen-landscape results really reflect robust priors in the early shallow weights, rather than trivial similarity among under-specialized features or an artifact of retraining the deep subnetwork; if that confound is large, the robustness-fading diagnosis and both interventions lose their motivation.","fun_headline_variants_meta":{"raw":{"variants":["Robustness fades in training; freeze shallow weights to keep it","Early robust features vanish; rewind shallow weights","Freeze early shallow weights to stop robustness fade","Robustness emerges early, fades later; two parameter-free fixes","Robustness fades during convergence; rewind shallow weights"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000981,"raw_usage":{"total_tokens":4133,"prompt_tokens":882,"completion_tokens":3251,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":3167}},"tokens_in":498,"tokens_out":3251,"duration_ms":21540,"temperature":1.0,"reasoning_tokens":3167,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:40:14.814792+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same architectures while freezing a randomly initialized or adversarially shuffled shallow subnetwork at the same early epoch; if such a control matches the corruption robustness of EPS and AWR, then the reported gains come from stopping shallow updates per se, not from preserving specifically robust early weights. Alternatively, measure CKA and mutual information between clean and corrupted features for early shallow weights after the deep subnetwork has been retrained to convergence; if early features no longer look robust after deep retraining, then the freezing experiments are not isolating the effect of $\\theta_s$.","supporting_citations":[{"cited_title":"In: International Conference on Learning Representations","cited_arxiv_id":null,"evidence_quote":"Gives the weight-rewinding mechanism that AWR adapts to revert shallow parameters to an early robust state."},{"cited_title":"Advances in Neural Information Processing Systems33, 13073–13087 (2020)","cited_arxiv_id":null,"evidence_quote":"Shows that a hard-coded V1-like front-end boosts robustness, supporting the premise that shallow layers have untapped robust capacity."},{"cited_title":"In: International conference on machine learning","cited_arxiv_id":null,"evidence_quote":"Documents adversarial robust overfitting, the early-peak-then-decline pattern that motivates studying natural-robustness dynamics."},{"cited_title":"In: International Conference on Learning Representations","cited_arxiv_id":null,"evidence_quote":"AugMix serves as a representative augmentation baseline whose shallow layers remain unimproved in the layer-wise probing comparison."}],"review_version":2}