Robustness Emerges Early in Training Dynamics, but Is Not Preserved

Jiangang Yang; Jian Liu; Jing Xing; Lu Hu; Wenhui Shi

REVIEW 5 major objections 6 minor 72 references

Robustness Emerges Early in Training Dynamics, but Is Not Preserved

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Shallow layers build their most robust representations early in training, but standard convergence destroys them.

desk verdict Plausible and practically motivated, but the central mechanism claim is under-supported because the early-robustness evidence lacks controls for under-specialization, and the reported gains come with clean-accuracy drops that the paper glosses over. read the letter →

arxiv 2608.04442 v1 pith:FPYW72FT submitted 2026-08-05 cs.LG cs.CV

Jiangang Yang , Wenhui Shi , Lu Hu , Jing Xing , Jian Liu This is my paper · ORCID

classification cs.LGcs.CV

keywords robustnessfadingnaturalcorruptionstrainingdynamicsshallowsubnetworkEarly-PhaseStabilizationAsymmetricWeightReversionrepresentationstabilitylosslandscapeflatness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the shallow layers of a deep network do their best corruption-robustness work very early in training, and that ordinary training to convergence actively erases that advantage. The authors call this "robustness fading." They show with representation-similarity, mutual-information, and loss-landscape measurements that early shallow features are more stable under natural corruptions and sit in flatter minima than late-training shallow features. From that diagnosis, they propose two parameter-free interventions, Early-Phase Stabilization and Asymmetric Weight Reversion, that freeze or rewind the shallow subnetwork while letting deeper layers keep fitting, and report consistent gains in corruption robustness across architectures and downstream tasks. If true, the paper turns robustness from a property to be built into architectures into a resource already present early in training and worth deliberately preserving.

What carries the argument

The load-bearing object is the decomposition of the network into a shallow subnetwork $\theta_s$ and a deep subnetwork $\theta_d$, together with the robustness-fading diagnosis that $\theta_s$'s best configuration occurs early. The mechanism that carries the interventions is a trajectory operator that constrains $\theta_s$ after a threshold epoch: EPS sets $\Delta\theta_s^{(t)} \leftarrow 0$, freezing the early robust state, while AWR resets $\theta_s$ to a snapshot from epoch $t_{\mathrm{early}}$ and lowers its learning rate $\eta_s$, letting $\theta_d$ adapt around it. The diagnostics that carry the argument are layer-wise linear probing, CKA and InfoNCE-based mutual information between clean and corrupted features, and freezing experiments that plot the loss-landscape sharpness of models anchored at early versus late shallow weights. These measurements are what link early weight states to flat geometry and corruption stability, motivating the interventions.

What would settle it

Train the same architectures while freezing a randomly initialized or adversarially shuffled shallow subnetwork at the same early epoch; if such a control matches the corruption robustness of EPS and AWR, then the reported gains come from stopping shallow updates per se, not from preserving specifically robust early weights. Alternatively, measure CKA and mutual information between clean and corrupted features for early shallow weights after the deep subnetwork has been retrained to convergence; if early features no longer look robust after deep retraining, then the freezing experiments are not isolating the effect of $\theta_s$.

Watch

Extended reading notes

Core claim

Under natural corruptions, the shallow subnetwork spontaneously develops robust representations and a flat loss landscape in the early phase of training, but these properties are not preserved during standard convergence. The evidence is layer-wise: CKA similarity and InfoNCE-based mutual information between clean and corrupted features decline in early layers across training, freezing shallow weights from early epochs and retraining the rest yields flatter loss surfaces, and later-phase shallow weights produce sharper landscapes. The paper therefore formulates robustness enhancement as a trajectory-intervention problem: modulate the shallow updates after an intervention epoch instead of modifying the architecture. Two parameter-free strategies instantiate this: EPS nullifies shallow updates after a chosen epoch, and AWR rewinds the shallow parameters to an early robust snapshot and shrinks their learning rate while deeper layers continue to adapt. On ImageNet-C, AWR with a strong augmentation recipe raises Top-1 accuracy from 39.2 to 46.5, and the interventions reduce mean Corruption Error across CNN, ViT, and Mamba backbones and improve detection, segmentation, and online test-time adaptation.

Load-bearing premise

The load-bearing premise is that the high early CKA and mutual-information values and the flat frozen-landscape results really reflect robust priors in the early shallow weights, rather than trivial similarity among under-specialized features or an artifact of retraining the deep subnetwork; if that confound is large, the robustness-fading diagnosis and both interventions lose their motivation.

Editorial extensions

If this is right

Corruption robustness of a standard trained network is largely determined by whether the shallow subnetwork's early-phase configuration survives; deep-layer training alone cannot recover it once shallow robustness has faded.
EPS and AWR give consistent robustness gains on ImageNet-C and related benchmarks, with AWR reaching 46.5% Top-1 accuracy, a gain of 7.3 points over the baseline.
The gains transfer across tasks: object detection on COCO-C, semantic segmentation on Cityscapes-C and ADE20K-C, and several online test-time adaptation methods all improve when EPS or AWR is integrated.
The interventions are compatible with data augmentation and regularization, adding mean Corruption Error reductions on top of AugMix, AutoAug, CutMix, label smoothing, and dropout.
The mechanism works across CNN, ViT, and Mamba architectures, making it an architecture-agnostic way to improve corruption robustness without adding parameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If robustness is an early-training resource, then late-phase fitting to clean data may be the main culprit behind its loss; a testable extension is whether learning-rate schedules that slow late-phase fitting reduce the need for explicit freezing or rewinding.
The same preserve-the-early-snapshot logic could apply to other distribution shifts, such as domain shift or label noise, although the paper only claims natural corruptions; applying EPS and AWR to domain-generalization benchmarks would test that.
Because the interventions add no learnable parameters, they could be combined with pruning or lottery-ticket-style rewinding to separate the robustness contribution of the shallow subnetwork from the trainability contribution of the deep subnetwork.
The early-peak-then-fade trajectory suggests a cheap diagnostic: monitoring CKA or loss sharpness of shallow layers during training could serve as an early-warning signal that final robustness will be poor, guiding training adjustments before corruption benchmarks are run.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

What you should know: this paper has a useful empirical hook and a simple, parameter-free intervention pair, but its central diagnosis is not yet established. The authors show that if you freeze or rewind shallow-layer weights to an early-training state, you get consistent mCE improvements on corruption benchmarks across many architectures and tasks. That is a real, practical result worth taking seriously, and the breadth of evaluation (classification, detection, segmentation, TTA, ACDC) is a genuine strength. The 'robustness fading' framing is also nice: it connects critical learning periods and robust overfitting into a layer-wise story.

What is actually new: the specific characterization that shallow-layer natural-corruption robustness peaks early and decays, plus the EPS/AWR intervention pair. The ingredients are close to progressive layer freezing and LTH rewinding, so novelty is moderate, but the combination aimed at corruption robustness is not in the cited literature.

Now the soft spots, in proportion. First, the load-bearing mechanism evidence in Figs. 3 and 4 is vulnerable to the under-specialization confound. Early features are low-rank and small-norm; CKA and MI similarity between clean and corrupted inputs can be high simply because the features are not yet task-specific, not because they are robust. The freezing experiment lacks a null control (e.g., randomly permuted early weights) that preserves scale and under-specialization while destroying learned structure. The stress-test note lands: without that control, the flatness evidence is equally explained by early weights being near initialization and acting as a regularizer. So the paper's central claim, that 'shallow layers spontaneously develop robust representations,' is plausible but not proven.

Second, the reporting is sloppier than it should be. No code, seeds, or error bars; Table 2 mixes Top-1 and mCE without clear labels; Table 1 has a stray token. More substantively, Table 3 shows clean accuracy dropping by 2-3 points across the board while mCE gains are tiny (often below 1). That is a robustness-accuracy tradeoff the paper does not discuss. The text says 'consistently yield additive gains' but the clean accuracy losses are not negligible.

Third, the intervention hyperparameters are tuned via sensitivity analyses on the same corruption benchmarks used for the headline comparisons, which inflates the apparent gains. A real evaluation would hold out the main benchmark or use fixed defaults.

Bottom line: the paper deserves a serious referee, but the mechansim needs controls, the tradeoff needs reporting, and the code/data need to be released. I'd send it to review with a request for major revision, and I'd focus the referee's attention on the freezing null control and the clean-accuracy loss.

Would I bring it to reading group? Maybe, mostly to discuss the under-specialization confound and the tradeoff. Would I cite it? Not yet, until the mechanism holds up. But the EPS/AWR trick is the kind of thing that could be useful if it reproduces, so I would not desk-reject.

Referee Report

5 major / 6 minor

Summary. The manuscript studies the training dynamics of shallow-subnetwork robustness to natural corruptions. It claims that shallow layers acquire robust, corruption-stable representations and flat loss landscapes early in training, but that standard ERM convergence erodes these properties, a phenomenon the authors call 'robustness fading.' Based on this diagnosis, the paper proposes two intervention strategies: Early-Phase Stabilization (EPS), which freezes the shallow subnetwork after an early epoch, and Asymmetric Weight Reversion (AWR), which rewinds the shallow subnetwork to an early snapshot and reduces its learning rate while the deep subnetwork continues training. The authors report gains over representative baselines on ImageNet-C and related corruption benchmarks, across several architectures, and on downstream detection, segmentation, and test-time adaptation tasks. The paper is entirely empirical and does not provide code, seeds, or error bars; all headline results are reported as single numbers.

Significance. If the central finding were established, the paper would make a useful contribution: it connects training dynamics to corruption robustness and offers a simple, architecture-agnostic intervention with no architectural modifications or extra learnable parameters. The evaluation is broad, covering classification, dense prediction, and online TTA, and the freezing experiments in Sec. 3.3 are a reasonable causal probe. However, the central diagnostic evidence is currently confounded by the under-specialization of early features, and the headline claims are not fully supported by the reported numbers once clean-accuracy trade-offs and table inconsistencies are taken into account. The paper does not ship code or machine-checked proofs, so its contribution rests entirely on the empirical claims, which need stronger controls and more careful reporting.

major comments (5)

[Sec. 3.2, Figs. 3 and 4] The evidence for 'robustness fading' does not control for early-feature under-specialization. CKA and InfoNCE-based MI between clean and corrupted features can be high in early training simply because early features have low effective rank, small norms, and undeveloped selectivity; the subsequent decline can occur even without any loss of a learned robustness property. The InfoNCE estimator is also sensitive to feature dimensionality and batch statistics, so cross-depth comparisons are not calibrated. The freezing experiment in Sec. 3.3 has no null control: freezing early weights and retraining deep layers may act as a regularizer simply because early weights are close to initialization. A control that permutes weights within the early checkpoint, or otherwise destroys learned structure while preserving scale and under-specialization, is needed before the diagnosis can support the proposed interventions.
[Sec. 5.2, Table 3] The text states that EPS and AWR 'consistently yield additive gains' across augmentation and regularization baselines, but clean Top-1 accuracy decreases in nearly every integrated setting. For example, CutMix clean accuracy drops from 76.9 to 74.3 (EPS) and 74.4 (AWR); Label Smoothing drops from 76.6 to 74.5 and 74.7; AutoAug drops from 76.4 to 74.3. The reported Avg. mCE reductions are small (0.5-1.9 points) and are given as single numbers without error bars or significance tests. The claim should be reframed as a clean-accuracy/robustness trade-off, and the mCE differences need variance estimates.
[Sec. 5.4 and Algorithm 1] The methods are described as 'parameter-free,' but they depend on intervention hyperparameters: the EPS epoch tau_eps, the AWR snapshot epoch t_early, the AWR reversion epoch tau_awr, and the shallow learning rate eta_s. These hyperparameters are analyzed via sensitivity studies in Sec. 5.4 on the same corruption benchmarks (IN100-C, IN100-Cbar, IN100-3DCC) used for the headline comparisons. Unless the hyperparameters were fixed on a held-out set or selected with a principled criterion before seeing the evaluation sets, the reported gains are optimistic. The authors should clarify the selection protocol and replace the 'parameter-free' terminology with 'no learnable parameters.'
[Reproducibility, all experiments] All experimental results are reported as single numbers with no seeds, standard deviations, or code release. Because the paper's contribution is empirical and several reported improvements are small (for example, Table 3 and Table 6), this level of reporting is insufficient to support the claim of consistent improvement. At minimum, the authors should provide seeds and confidence intervals for the main tables.
[Tables 1-3] The main comparison tables are not self-contained. Table 1 lacks clear column labels and contains a stray token; Table 2 claims to report both Top-1 accuracy and mCE but does not clearly indicate which columns are which, and the MobileNetV2 row shows an IN-100-C value of 95.8 with clean accuracy 84.6, which cannot both be Top-1 accuracy; Table 3 mixes clean accuracy with mCE under a single header. A reader cannot verify the reported gains without this information.

minor comments (6)

[Fig. 1] The axes and the precise definition of subnetwork indices (e.g., #1, #10, #40, #49) are not given in the main text, and the linear-probing setup is described only by a citation; please provide these details.
[Sec. 3.2] The InfoNCE-based MI estimator is sensitive to feature dimensionality and batch statistics; the paper should justify why cross-depth comparisons are meaningful without calibration.
[Eq. (2)] The intervention operator I is introduced but never formally instantiated for EPS or AWR; the authors should show explicitly how Eq. (2) encodes Eqs. (3) and (4).
[Sec. 5.3, Tables 4 and 5] The text says the framework 'consistently boosts performance,' but clean mAP/mIoU drops in several rows, e.g., FasterRCNN COCO from 37.6 to 37.0 and DeepLabV3+ ADE20K S from 42.1 to 41.0; please report the clean/robust trade-off explicitly.
[Sec. 5.3, Table 6] The online TTA gains are small (0.1 to 2.6 points) and are reported without error bars; please add variance or state that differences are not statistically tested.
[Throughout] There are formatting typos such as 'T able 1', 'T able 2', and the stray '∗P' token in Table 1; please proofread the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central observations are empirical measurements and the intervention results are evaluated against external benchmarks.

full rationale

The manuscript is an empirical study rather than a derivation chain. The robustness-fading claim is supported by direct measurements (CKA, InfoNCE MI, freezing experiments) and the interventions are then evaluated on external corruption benchmarks (ImageNet-C, COCO-C, ADE20K-C, Cityscapes-C, ACDC) that are not used in the diagnosis. No equation in the paper reduces a predicted quantity to a fitted input; the EPS and AWR hyperparameters are swept in Sec. 5.4 rather than fit to the headline numbers, and the reported gains are not equal by construction to the CKA/MI or flatness measurements. The paper cites only external prior work for robustness overfitting, sharpness, CKA, and related concepts; there is no load-bearing self-citation or imported uniqueness theorem. The 'robust priors' term is explicitly operationalized in Sec. 6 as 'empirically associated' with corruption robustness via representation similarity, information preservation, freezing interventions, and loss-landscape analyses, so the later association of EPS/AWR with smoother landscapes is a consistency check rather than a definitional equivalence. The main threats to the central claim are empirical confounds, such as early features being under-specialized or frozen early weights acting as a regularizer, but these are validity concerns that require matched controls, not circularity within the paper's argument.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claim rests on a small set of hand-tuned intervention hyperparameters and on the assumption that representation-similarity and freezing experiments measure robustness rather than feature collapse. No new physical entities are introduced beyond a conceptual label for the observed phenomenon.

free parameters (5)

EPS intervention epoch tau_eps = not reported in main text; tuned via sensitivity analysis (Fig. 5a)
Freezing time for shallow weights; chosen by hand on the target corruption benchmarks; no fixed value given for the main runs.
AWR snapshot epoch t_early = tuned via Fig. 5b
The epoch from which shallow weights are saved before reversion; the ablation claims insensitivity over [0,30], but the final choice is not stated.
AWR reversion epoch tau_awr = tuned via Fig. 5c; peak near epoch 20
The epoch at which shallow weights are reset to the early snapshot; the ablation shows a peak around 20 and a drop beyond 35.
AWR shallow learning rate eta_s = tuned via Fig. 5d; robust range 1e-6 to 1e-3
The reduced learning rate applied after reversion; a hand-selected hyperparameter that controls how much the restored features can drift.
shallow subnetwork layer split theta_s = not specified in main text
Architecture-dependent choice of which layers count as shallow; the paper does not define this split precisely or ablate over it, which is required to reproduce EPS/AWR.

assumptions (5)

domain assumption CKA similarity and InfoNCE MI between clean and corrupted features track corruption robustness (Sec. 3.2).
The paper interprets high early CKA/MI as robust priors, but these metrics could be confounded by feature collapse or by normal feature specialization during training.
ad hoc to paper Freezing theta_s at a given epoch and retraining theta_d isolates the robustness contribution of theta_s (Sec. 3.3, Fig. 4).
The re-optimized deep layers adapt to frozen features, so the comparison does not purely isolate the effect of the shallow weights; it also includes the effect of a different deep optimization run.
domain assumption Standard ERM with SGD and the described hyperparameters is a sufficient base for evaluating robustness interventions (Sec. 5.1).
The paper excludes augmentations to isolate effects, but the interaction with strong augmentation is not fully characterized, and clean accuracy often drops when combined with regularizers.
domain assumption ImageNet-C and its variants are valid proxies for natural corruption robustness (Sec. 5.1).
This is a standard benchmark assumption and not unique to this paper, but all headline claims are tied to these datasets.
domain assumption Local sharpness and gradient flatness along random perturbations measure robustness-relevant geometry (Sec. 5.5).
This is an adopted metric from prior work; the connection to natural-corruption robustness is asserted rather than derived or validated on held-out distributions.

invented entities (1)

early-emergent robust prior
purpose: A hypothesized internal state of shallow layers that the paper claims exists early in training and that EPS and AWR are designed to preserve.
The state is defined through the same CKA/MI and freezing measurements used to motivate the method, and the only evidence offered is the authors' own corruption-benchmark results. No external, independently validated handle is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Robustness Emerges Early in Training Dynamics, but Is Not Preserved." pith.science (2026). https://pith.science/paper/FPYW72FT

@misc{pith2026260804442,
  author       = {Pith},
  title        = {Pith review of: Robustness Emerges Early in Training Dynamics, but Is Not Preserved},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FPYW72FT}},
  note         = {Machine review of arXiv:2608.04442}
}

read the original abstract

Robustness to natural corruptions remains a fundamental challenge for deep neural networks. In this paper, we identify a robustness fading phenomenon where shallow layers spontaneously develop robust representations and flat loss landscapes in early training, yet these properties are not preserved during standard convergence. To address this, we propose a framework that performs strategic interventions on training dynamics to stabilize the empirically identified early-emergent robust priors. Our approach includes two parameter-free strategies: Early-Phase Stabilization~(EPS) and Asymmetric Weight Reversion~(AWR), which stabilize or recover robust shallow configurations without modifying the model architecture or introducing learnable parameters. Extensive experiments demonstrate the efficacy of our framework across various benchmarks and architectures, yielding significant gains in downstream transfer, dynamic adaptation, and diverse computer vision applications.

Figures

Figures reproduced from arXiv: 2608.04442 by the authors.

**Figure 1.** Robustness gap analysis across subnetworks. We compare ResNet-50 with (a) AugMix [23], (b) Mixup [71], and (c) the TVR-V2 training recipe [60]. While all three methods significantly reduce the robustness gap in deeper subnetworks (e.g., #40 and #49), their performance in shallow subnetworks (e.g., #1 and #10) remains nearly identical to the baseline, failing to deliver meaningful gains. layers, with shallow layers r… view at source ↗

**Figure 2.** Weight directional stability evolution. (a) and (b) employ different LR schedulers. The shaded area highlights the initial unstable exploration phase. As demonstrated in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Evolution of representation-level robustness across network depths and training phases. (a-c) CKA similarity between clean and corrupted features at severity levels 3, 4, and 5; each curve tracks a subnetwork’s (Stem to Stage-3) stability during training. (d-f) InfoNCE-based MI estimates for Stem, Stage-1, and Stage-4 across four corruption types. Results reveal that shallow layers achieve peak structural similarit… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Loss surfaces under perturbations, where shallow subnetworks are frozen at different training phases (early, intermediate, and late). Results suggest that earlyphase shallow features possess an intrinsic property that guides the model toward flatter and more robust mi…

**Figure 5.** Figure 5: Sensitivity analysis of (a) EPS intervention timing τeps, (b) AWR hyperparameters tearly, (c) τawr, and (d) ηs. （a）（b）（c）（d） [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

**Figure 6.** Figure 6: Analysis of optimization sensitivity and representation stability. (a-b) Evolution of Local Sharpness (∆Lmax) and Gradient Flatness (∥∇L∥) for Baseline, EPS, and AWR during training. (c-d) Representation geometry stability across five corruption severity levels, meas…

**Figure 7.** Figure 7: t-SNE visualizations of Baseline, EPS, and AWR (from left to right) under (a-c) Gaussian noise and (d-f) Pixelate. model exhibits remarkable robustness within [10−6 , 10−3 ], with a sharp decline only at 10−1 due to optimization instability. Overall, these results conf…

**Figure 8.** Figure 8: Robustness analysis under patch shuffling and frequency domain perturbations. (a) Top-1 Accuracy under patch shuffling, grouped by patch sizes 2×2 and 4×4 (from left to right). (b) Frequency-domain error distribution for Baseline, EPS, and AWR (from left to right). t-S…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 39 canonical work pages

[1]

In: International conference on learning representations (2018)

Achille, A., Rovere, M., Soatto, S.: Critical learning periods in deep networks. In: International conference on learning representations (2018)

work page 2018
[2]

arXiv preprint arXiv:1610.01644 (2016)

Alain, G., Bengio, Y.: Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 (2016)

arXiv 2016
[3]

Bair, A., Yin, H., Shen, M., Molchanov, P., Alvarez, J.M.: Adaptive sharpness- awarepruningforrobustsparsenetworks.In:TheTwelfthInternationalConference on Learning Representations (2024)

work page 2024
[4]

In: Proceedings of the IEEE/CVF international conference on computer vision workshops

Cao, Y., Xu, J., Lin, S., Wei, F., Hu, H.: Gcnet: Non-local networks meet squeeze- excitation networks and beyond. In: Proceedings of the IEEE/CVF international conference on computer vision workshops. pp. 0–0 (2019)

work page 2019
[5]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Chen, D., Wang, D., Darrell, T., Ebrahimi, S.: Contrastive test-time adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 295–305 (2022)

2022
[6]

arXiv preprint arXiv:1906.07155 (2019)

Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., et al.: Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 (2019)

arXiv 2019
[7]

In: Proceedings of the European conference on computer vision (ECCV)

Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV). pp. 801–818 (2018)

2018
[8]

In: Findings of the Association for Computational Linguistics: ACL 2024

Chimoto, E.A., Gala, J., Ahia, O., Kreutzer, J., Bassett, B.A., Hooker, S.: Critical learning periods: Leveraging early training dynamics for efficient data pruning. In: Findings of the Association for Computational Linguistics: ACL 2024. pp. 9407– 9426 (2024)

work page 2024

Show all 72 references

[9]

Yang et al

Contributors, M.: MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.https://github.com/open-mmlab/mmsegmentation(2020) 16 J. Yang et al

2020
[10]

arXiv preprint arXiv:1805.09501 (2018)

Cubuk,E.D.,Zoph,B.,Mane,D.,Vasudevan,V.,Le,Q.V.:Autoaugment:Learning augmentation policies from data. arXiv preprint arXiv:1805.09501 (2018)

2018 arXiv
[11]

Advances in Neural Information Processing Systems33, 13073–13087 (2020)

Dapello, J., Marques, T., Schrimpf, M., Geiger, F., Cox, D., DiCarlo, J.J.: Simulat- ing a primary visual cortex at the front of cnns improves robustness to image per- turbations. Advances in Neural Information Processing Systems33, 13073–13087 (2020)

2020
[12]

In: 2009 IEEE conference on computer vision and pattern recognition

Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)

2009
[13]

Ad- vances in neural information processing systems34, 664–676 (2021)

Diffenderfer, J., Bartoldson, B., Chaganti, S., Zhang, J., Kailkhura, B.: A winning hand: Compressing deep networks can improve out-of-distribution robustness. Ad- vances in neural information processing systems34, 664–676 (2021)

2021
[14]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Döbler, M., Marsden, R.A., Yang, B.: Robust mean teacher for continual and gradual test-time adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7704–7714 (2023)

2023
[15]

arXiv preprint arXiv:2010.01412 (2020)

Foret, P., Kleiner, A., Mobahi, H., Neyshabur, B.: Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412 (2020)

2020 arXiv
[16]

In: International Conference on Learning Representations

Frankle, J., Carbin, M.: The lottery ticket hypothesis: Finding sparse, trainable neural networks. In: International Conference on Learning Representations
[17]

In: International Conference on Learning Representations

Frankle, J., Schwab, D.J., Morcos, A.S.: The early phase of neural network training. In: International Conference on Learning Representations
[18]

Gavrikov, P., Keuper, J.: Can biases in imagenet models explain generalization? In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 22184–22194 (2024)

2024
[19]

In: International conference on learning representations (2018)

Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., Brendel, W.: Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In: International conference on learning representations (2018)

2018
[20]

In: European Conference on Computer Vision

Guo, Y., Stutz, D., Schiele, B.: Improving robustness by enhancing weak subnets. In: European Conference on Computer Vision. pp. 320–338. Springer (2022)

2022
[21]

He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)

2016
[22]

arXiv preprint arXiv:1903.12261 (2019)

Hendrycks,D.,Dietterich,T.:Benchmarkingneuralnetworkrobustnesstocommon corruptions and perturbations. arXiv preprint arXiv:1903.12261 (2019)

2019 arXiv
[23]

In: International Conference on Learning Representations

Hendrycks, D., Mu, N., Cubuk, E.D., Zoph, B., Gilmer, J., Lakshminarayanan, B.: Augmix: A simple data processing method to improve robustness and uncertainty. In: International Conference on Learning Representations
[24]

Advances in neural information processing systems 33, 19000–19015 (2020)

Hermann, K., Chen, T., Kornblith, S.: The origins and prevalence of texture bias in convolutional neural networks. Advances in neural information processing systems 33, 19000–19015 (2020)

2020
[25]

Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., Madry, A.: Adversarial examplesarenotbugs,theyarefeatures.Advancesinneuralinformationprocessing systems32(2019)

2019
[26]

In: 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018

Izmailov, P., Wilson, A., Podoprikhin, D., Vetrov, D., Garipov, T.: Averaging weights leads to wider optima and better generalization. In: 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018. pp. 876–885 (2018)

2018
[27]

0-yolov5 sota realtime instance segmentation

Jocher, G., Chaurasia, A., Stoken, A., Borovec, J., Kwon, Y., Michael, K., Fang, J., Yifu, Z., Wong, C., Montes, D., et al.: ultralytics/yolov5: v7. 0-yolov5 sota realtime instance segmentation. Zenodo (2022) Robustness Emerges Early in Training Dynamics, but Is Not Preserved 17

2022
[28]

International journal of computer vision129(2), 462–483 (2021)

Kamann, C., Rother, C.: Benchmarking the robustness of semantic segmentation models with respect to common corruptions. International journal of computer vision129(2), 462–483 (2021)

2021
[29]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Kar, O.F., Yeo, T., Atanov, A., Zamir, A.: 3d common corruptions and data aug- mentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18963–18974 (2022)

2022
[30]

In: International conference on machine learning

Kornblith, S., Norouzi, M., Lee, H., Hinton, G.: Similarity of neural network rep- resentations revisited. In: International conference on machine learning. pp. 3519–
[31]

In: International Conference on Learning Representations (2018)

Li, C., Farkhoor, H., Liu, R., Yosinski, J.: Measuring the intrinsic dimension of objective landscapes. In: International Conference on Learning Representations (2018)

2018
[32]

Advances in neural information processing systems31(2018)

Li, H., Xu, Z., Taylor, G., Studer, C., Goldstein, T.: Visualizing the loss landscape of neural nets. Advances in neural information processing systems31(2018)

2018
[33]

Advances in Neural Information Processing Systems35, 12934–12949 (2022)

Li, Y., Yuan, G., Wen, Y., Hu, J., Evangelidis, G., Tulyakov, S., Wang, Y., Ren, J.: Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems35, 12934–12949 (2022)

2022
[34]

International Journal of Computer Vision133(2), 567–589 (2025)

Liu, C., Dong, Y., Xiang, W., Yang, X., Su, H., Zhu, J., Chen, Y., He, Y., Xue, H., Zheng, S.: A comprehensive study on robustness of image classification models: Benchmarking and rethinking. International Journal of Computer Vision133(2), 567–589 (2025)

2025
[35]

Journal of machine learning research9(11) (2008)

Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research9(11) (2008)

2008
[36]

In: International Conference on Learning Representations (2018)

Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: International Conference on Learning Representations (2018)

2018
[37]

Advances in Neural Information Processing Systems35, 7520–7533 (2022)

Mao, X., Chen, Y., Duan, R., Zhu, Y., Qi, G., Li, X., Zhang, R., Xue, H., et al.: Enhance the visual representation via discrete adversarial training. Advances in Neural Information Processing Systems35, 7520–7533 (2022)

2022
[38]

Mayilvahanan, P., Wiedemer, T., Rusak, E., Bethge, M., Brendel, W.: Does clip’s generalization performance mainly stem from high train-test similarity? (2024)

2024
[39]

arXiv preprint arXiv:2110.02178 (2021)

Mehta, S., Rastegari, M.: Mobilevit: light-weight, general-purpose, and mobile- friendly vision transformer. arXiv preprint arXiv:2110.02178 (2021)

2021 arXiv
[40]

arXiv preprint arXiv:1907.07484 (2019)

Michaelis, C., Mitzkus, B., Geirhos, R., Rusak, E., Bringmann, O., Ecker, A.S., Bethge, M., Brendel, W.: Benchmarking robustness in object detection: Au- tonomous driving when winter is coming. arXiv preprint arXiv:1907.07484 (2019)

2019 arXiv
[41]

Advances in Neural Information Processing Systems34, 3571–3583 (2021)

Mintun,E.,Kirillov,A.,Xie,S.:Oninteractionbetweenaugmentationsandcorrup- tions in natural corruption robustness. Advances in Neural Information Processing Systems34, 3571–3583 (2021)

2021
[42]

Modas, A., Rade, R., Ortiz-Jiménez, G., Moosavi-Dezfooli, S.M., Frossard, P.: Prime:Afewprimitivescanboostrobustnesstocommoncorruptions.In:European Conference on Computer Vision. pp. 623–640. Springer (2022)

2022
[43]

Müller,R.,Kornblith,S.,Hinton,G.E.:Whendoeslabelsmoothinghelp?Advances in neural information processing systems32(2019)

2019
[44]

arXiv preprint arXiv:2302.12400 (2023)

Niu, S., Wu, J., Zhang, Y., Wen, Z., Chen, Y., Zhao, P., Tan, M.: Towards sta- ble test-time adaptation in dynamic wild world. arXiv preprint arXiv:2302.12400 (2023)

2023 arXiv
[45]

arXiv preprint arXiv:1807.03748 (2018) 18 J

Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748 (2018) 18 J. Yang et al

2018 arXiv
[46]

In: European Conference on Computer Vision

Pérez, J.C., Alfarra, M., Jeanneret, G., Bibi, A., Thabet, A., Ghanem, B., Ar- beláez, P.: Gabor layers enhance network robustness. In: European Conference on Computer Vision. pp. 450–466. Springer (2020)

2020
[47]

Advances in Neural Information Processing Systems35, 16276–16289 (2022)

Qin, Y., Zhang, C., Chen, T., Lakshminarayanan, B., Beutel, A., Wang, X.: Un- derstanding and improving robustness of vision transformers through patch-based negative augmentation. Advances in Neural Information Processing Systems35, 16276–16289 (2022)

2022
[48]

In: International conference on machine learning

Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., Ben- gio, Y., Courville, A.: On the spectral bias of neural networks. In: International conference on machine learning. pp. 5301–5310. PMLR (2019)

2019
[49]

Recht, B., Roelofs, R., Schmidt, L., Shankar, V.: Do imagenet classifiers generalize to imagenet? In: International conference on machine learning. pp. 5389–5400. PMLR (2019)

2019
[50]

IEEE transactions on pattern analysis and machine intelligence39(6), 1137–1149 (2016)

Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object de- tection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence39(6), 1137–1149 (2016)

2016
[51]

In: International conference on machine learning

Rice, L., Wong, E., Kolter, Z.: Overfitting in adversarially robust deep learning. In: International conference on machine learning. pp. 8093–8104. PMLR (2020)

2020
[52]

In: 2007 15th European signal processing conference

Roy, O., Vetterli, M.: The effective rank: A measure of effective dimensionality. In: 2007 15th European signal processing conference. pp. 606–610. IEEE (2007)

2007
[53]

In: Proceedings of the IEEE/CVF International Conference on Computer Vision

Saikia,T.,Schmid,C.,Brox,T.:Improvingrobustnessagainstcommoncorruptions with frequency biased models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10211–10220 (2021)

2021
[54]

In: Proceedings of the IEEE/CVF international conference on computer vision

Sakaridis, C., Dai, D., Van Gool, L.: Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 10765–10775 (2021)

2021
[55]

In: Proceedings of the IEEE conference on computer vision and pattern recognition

Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: In- verted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4510–4520 (2018)

2018
[56]

Santurkar, S., Tsipras, D., Ilyas, A., Madry, A.: How does batch normalization help optimization? Advances in neural information processing systems31(2018)

2018
[57]

The journal of machine learning research15(1), 1929–1958 (2014)

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research15(1), 1929–1958 (2014)

2014
[58]

Advances in Neural Information Processing Systems37, 35492–35516 (2024)

Trinh, Q.T., Heinonen, M., Acerbi, L., Kaski, S.: Improving robustness to corrup- tions with multiplicative weight perturbations. Advances in Neural Information Processing Systems37, 35492–35516 (2024)

2024
[59]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Vaish, P., Wang, S., Strisciuglio, N.: Fourier-basis functions to bridge augmentation gap: Rethinking frequency augmentation in image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17763–17772 (2024)

2024
[60]

PyTorch (2021)

Vryniotis, V.: How to train state-of-the-art models using torchvision’s latest prim- itives. PyTorch (2021)

2021
[61]

Wang, D., Shelhamer, E., Liu, S., Olshausen, B., Darrell, T.: Tent: Fully test- timeadaptationbyentropyminimization.In:InternationalConferenceonLearning Representations (2021)

2021
[62]

In: NeurIPS 2021 Workshop on ImageNet: Past, Present, and Future Robustness Emerges Early in Training Dynamics, but Is Not Preserved 19

Wightman, R., Touvron, H., Jegou, H.: Resnet strikes back: An improved training procedure in timm. In: NeurIPS 2021 Workshop on ImageNet: Past, Present, and Future Robustness Emerges Early in Training Dynamics, but Is Not Preserved 19

2021
[63]

In: The Thirteenth Interna- tional Conference on Learning Representations (2025)

Wu, B., Xiao, Q., Wang, S., Strisciuglio, N., Pechenizkiy, M., van Keulen, M., Mocanu, D.C., Mocanu, E.: Dynamic sparse training versus dense training: The unexpected winner in image corruption robustness. In: The Thirteenth Interna- tional Conference on Learning Representatio...

2025
[64]

Advances in Neural Information Processing Systems32(2019)

Yin, D., Gontijo Lopes, R., Shlens, J., Cubuk, E.D., Gilmer, J.: A fourier per- spective on model robustness in computer vision. Advances in Neural Information Processing Systems32(2019)

2019
[65]

In: International Conference on Machine Learning

Yu, C., Han, B., Shen, L., Yu, J., Gong, C., Gong, M., Liu, T.: Understanding robust overfitting of adversarial training and beyond. In: International Conference on Machine Learning. pp. 25595–25610. PMLR (2022)

2022
[66]

Yu,W.,Wang,X.:Mambaout:Dowereallyneedmambaforvision?In:Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 4484–4496 (2025)

2025
[67]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition

Yuan, L., Xie, B., Li, S.: Robust test-time adaptation in dynamic scenarios. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition. pp. 15922–15932 (2023)

2023
[68]

IEEE transactions on neural networks and learning systems30(9), 2805–2824 (2019)

Yuan, X., He, P., Zhu, Q., Li, X.: Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems30(9), 2805–2824 (2019)

2019
[69]

In: Proceedings of the IEEE/CVF international conference on computer vision

Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6023–6032 (2019)

2019
[70]

arXiv preprint arXiv:1605.07146 (2016)

Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (2016)

2016 arXiv
[71]

In: International Conference on Learning Representations (2018)

Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. In: International Conference on Learning Representations (2018)

2018
[72]

In: European conference on computer vision

Zhao, B., Yu, S., Ma, W., Yu, M., Mei, S., Wang, A., He, J., Yuille, A., Kortylewski, A.: Ood-cv: A benchmark for robustness to out-of-distribution shifts of individual nuisances in natural images. In: European conference on computer vision. pp. 163–

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

In: International conference on learning representations (2018)

Achille, A., Rovere, M., Soatto, S.: Critical learning periods in deep networks. In: International conference on learning representations (2018)

work page 2018

[2] [2]

arXiv preprint arXiv:1610.01644 (2016)

Alain, G., Bengio, Y.: Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 (2016)

arXiv 2016

[3] [3]

Bair, A., Yin, H., Shen, M., Molchanov, P., Alvarez, J.M.: Adaptive sharpness- awarepruningforrobustsparsenetworks.In:TheTwelfthInternationalConference on Learning Representations (2024)

work page 2024

[4] [4]

In: Proceedings of the IEEE/CVF international conference on computer vision workshops

Cao, Y., Xu, J., Lin, S., Wei, F., Hu, H.: Gcnet: Non-local networks meet squeeze- excitation networks and beyond. In: Proceedings of the IEEE/CVF international conference on computer vision workshops. pp. 0–0 (2019)

work page 2019

[5] [5]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Chen, D., Wang, D., Darrell, T., Ebrahimi, S.: Contrastive test-time adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 295–305 (2022)

2022

[6] [6]

arXiv preprint arXiv:1906.07155 (2019)

Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., et al.: Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 (2019)

arXiv 2019

[7] [7]

In: Proceedings of the European conference on computer vision (ECCV)

Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV). pp. 801–818 (2018)

2018

[8] [8]

In: Findings of the Association for Computational Linguistics: ACL 2024

Chimoto, E.A., Gala, J., Ahia, O., Kreutzer, J., Bassett, B.A., Hooker, S.: Critical learning periods: Leveraging early training dynamics for efficient data pruning. In: Findings of the Association for Computational Linguistics: ACL 2024. pp. 9407– 9426 (2024)

work page 2024