Pith. sign in

REVIEW 4 major objections 5 minor 73 references

The paper claims that loss spikes in normalized networks are caused by weight decay shrinking scale-invariant layer norms below a critical boundary, not just by learning-rate effects.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:42 UTC pith:HNTJ5QL3

load-bearing objection A clean Hessian-scaling theorem and a practically useful layer-wise diagnostic, but the paper's central spike boundary is not actually connected to the loss-spike condition. the 4 major comments →

arxiv 2607.21005 v1 pith:HNTJ5QL3 submitted 2026-07-23 cs.LG cs.NE

Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay

Xiaolong Li , Zhangchen Zhou , Zhi-Qin John Xu This is my paper
classification cs.LG cs.NE
keywords loss spikesweight-norm criticalityweight decayscale invariancenormalizationHessian sharpnesstraining instabilityedge of stability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that loss spikes during neural network training can be driven by an overlooked interaction between normalization and weight decay. Normalization makes certain weights scale-invariant—changing their size does not change the network output—so weight decay can shrink those norms while the loss stays functionally similar. The paper shows this shrinkage amplifies local loss curvature at least as the inverse square of the norm, and once a scale-invariant layer's norm crosses a computable boundary, training becomes unstable and loss spikes appear. The boundary is layer-wise, so the mechanism predicts not only whether a spike will occur but which layer is responsible. The paper validates this on MNIST, a controlled regression, a Transformer pretraining run, and ResNet-50, and shows that removing weight decay from the relevant scale-invariant layers suppresses the spikes.

Core claim

The paper's central claim is a mechanism it calls weight-norm criticality: for any scale-invariant parameter block u, so that L(αu,v)=L(u,v) for all α>0, the Hessian restricted to u satisfies H_uu(αu,v)=α^{-2}H_uu(u,v). Since the full Hessian's largest eigenvalue is at least that of any principal subblock, λmax(H(αu,v)) ≥ α^{-2}λmax(H_uu(u,v)). Weight decay shrinks ||u|| over training, so the loss landscape around a scale-invariant layer sharpens at least like ||u||^{-2}. Putting this scaling into the standard stability condition ηλmax(H)≤2 yields a layer-wise critical norm c* = sqrt(ηρ/2) with ρ=||u||²λmax(H_uu(u,v)), and a spike boundary with ρ replaced by the gradient-direction Rayleigh q

What carries the argument

The central object is the homogeneity relation of the Hessian under scale invariance: differentiating L(αu,v)=L(u,v) twice shows the uu-block of the Hessian scales as α^{-2}. The paper combines that relation with the Cauchy interlacing theorem, which says the top eigenvalue of a symmetric matrix is at least the top eigenvalue of any principal subblock, to get λmax(H(αu,v)) ≥ α^{-2}λmax(H_uu(u,v)). This converts the usual stability condition ηλmax≤2 into a weight-norm threshold c* = sqrt(ηρ/2), where ρ=||u||²λmax(H_uu(u,v)) is the intrinsic curvature. The spike boundary uses the same scaling with the gradient-direction Rayleigh quotient in place of the top eigenvalue.

Load-bearing premise

The load-bearing premise is that a scale-invariant layer's weight direction stays fixed as weight decay shrinks its norm, so the intrinsic curvature ρ that defines the critical boundary c*=sqrt(ηρ/2) remains constant; in real training the direction can drift, and the paper does not show the boundary survives that drift.

What would settle it

Track a scale-invariant layer during training, recording at every step both its norm ||W|| and the intrinsic curvature ρ=||W||²λmax(H_WW). If ρ changes appreciably when W is rescaled along the actual trajectory, the computed c* is not a function of norm alone. The cleanest falsifier: find a sustained interval where ||W|| remains above c* yet a loss spike occurs, or below c* without a spike; either event would show the norm boundary does not govern instability.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Increasing weight decay in a normalized network can destabilize training even when the learning rate is unchanged; spikes arise as scale-invariant norms cross the boundary.
  • The threshold is layer-wise: computing the Hessian restricted to each scale-invariant component gives a separate boundary, so spikes can be attributed to specific layers.
  • Weight decay on scale-invariant layers is necessary for this instability: applying decay only to non-scale-invariant parameters did not increase sharpness or produce spikes in the paper's experiments.
  • The analysis does not require the whole network to be scale-invariant, only individual components, so it applies to practical networks with BatchNorm or LayerNorm.
  • The mechanism explains a practical trade-off: weight decay improves generalization up to a point, beyond which weight-norm criticality makes loss spikes unavoidable.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The norm-only boundary suggests a practical intervention not explored in the paper: active control of scale-invariant layer norms, such as clipping or rescaling after each update, should prevent this class of spikes even at large weight decay.
  • Because ρ is direction-dependent but the paper treats it as a fixed constant, the boundary may become a moving target in real training; tracking ρ online rather than once could sharpen spike predictions.
  • The same Hessian-scaling argument should apply to any scale-invariant component, including weight-normalized linear layers and embedding norms, so the mechanism likely generalizes beyond normalization layers.
  • A testable extension: intentionally rescaling only one scale-invariant layer toward zero after training and observing whether spike onset matches the predicted boundary as ρ is re-measured.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes a new mechanism for training loss spikes in networks with normalization layers. The authors observe that weight decay drives the norms of scale-invariant parameters (e.g., weights feeding into BN/LN) toward zero, while the Hessian curvature in that parameter block grows at least quadratically in the inverse norm. They formalize this as 'weight-norm criticality': Theorem 5.1 gives a lower bound λmax(H(αu,v)) ≥ α^{-2} λmax(H_uu(u,v)), and Propositions 5.2–5.3 translate this into norm boundaries c* and c*_spike below which GD is linearly unstable or produces a one-step loss increase. The paper validates the scaling law on a small FNN and ResNet-50, shows that first-layer norm crossings of c*_spike align with loss spikes on MNIST and a synthetic regression task, and presents Transformer ablations indicating that excluding MLP parameters from weight decay reduces spikes.

Significance. If the mechanism is established, it gives a layer-localizable predictor for training instability and explains why weight decay cannot be made arbitrarily strong in normalized architectures. The paper's main theoretical contribution, Theorem 5.1, is mathematically clean; the controlled scaling experiments in Fig. 5 provide independent empirical support for the α^{-2} curvature lower bound, and the synthetic minimal setup (Appendix A.1.4) is a useful design. The extension from globally scale-invariant models to partial scale invariance is a genuine step beyond prior work. However, the spike-boundary result currently rests on an unproven relationship between the restricted curvature λ_grad(H_uu) and the full gradient-direction curvature in the actual spike condition, and the empirical validation of the boundary is partly circular. The central predictive claim is therefore not yet established, although it is plausible and probably repairable.

major comments (4)
  1. [§5.2, Prop. 5.2, Eq. (5)] The proof uses the equality λmax(H_uu(w,v)) = (‖u‖²/‖w‖²) λmax(H_uu(u,v)). This equality is only valid when w is a positive scalar multiple of the reference vector u (Theorem 5.1 applies to radial rescaling). Scale invariance implies ∇u L is orthogonal to u, so the GD update for u is tangential and the direction can drift; nothing in the paper states, proves, or tests that the direction remains fixed. If ρ is recomputed at the current point, the constant boundary c* in Fig. 7 is not the quantity derived; if ρ is fixed at a reference point, the boundary may be stale. Please state the assumption or provide a drift bound, and report how ρ_grad (and hence c*_spike) was obtained.
  2. [§5.2, Prop. 5.3; Appendix A.4.2–A.4.3] The actual one-step spike condition (A.4.2, Eq. (15)) is η λ_grad(H(w,v)) > 2 with λ_grad(H)=gᵀHg/‖g‖² using the full Hessian and full gradient. Proposition 5.3 instead defines λ_grad(H_uu) with the restricted block and g_u only. The text never proves an inequality such as λ_grad(H) ≥ c·λ_grad(H_uu); the cross block H_uv, the vv block, and the v component of the gradient can all suppress the full Rayleigh quotient. Consequently, the intervals ‖W1‖ < c*_spike shaded in Fig. 7 are not logically tied to the loss-increase criterion. A.4.3 only shows the restricted quotient scales as ‖u‖^{-2}. This missing bridge is central to the spike prediction; please supply it or replace the boundary with a condition directly on λ_grad(H).
  3. [§5.2, Fig. 7 and Appendix A.6] The empirical validation of the spike boundary is partly circular and the filtering protocol is ad hoc. c*_spike is computed from Hessian/gradient measurements on the same training trajectory whose crossings are later counted as confirmations; this is in part a consistency check against the same curvature data. The merging/discarding thresholds (30 and 200 iterations) are not justified, and discarding short excursions can select for the claimed alignment. Please report sensitivity to these thresholds, show unfiltered boundary-crossing events, and compare with a null model (e.g., random/shifted crossing times) to establish predictive value. The independent scaling-law test in Fig. 5 is a genuine strength, but it validates the curvature lower bound, not the spike boundary.
  4. [Abstract and §7 Discussion] The claim that the theoretical boundaries are 'empirically validated ... on Transformer and ResNet-50' is overstated. The quantitative norm-boundary versus loss-spike comparison (Fig. 7) is performed only on the small FNN/MNIST and synthetic regression cases. For ResNet-50 and the Transformer, the paper supplies the scaling-law measurement (Fig. 5b) and a module-wise ablation (Fig. 9), but no layer-wise c*_spike crossing analysis. Please either add such an analysis or qualify the conclusion to refer to the controlled settings.
minor comments (5)
  1. [Appendix A.4.3] The scaling statement 'H_uu(u,v)=α(w)H_uu(u,v)' is a typo with identical symbols on both sides; it should presumably be H_uu(w,v) = (‖u‖²/‖w‖²) H_uu(u,v) under the direction-fixed assumption. As written it is unreadable.
  2. [§3] 'nerual' should be 'neural'.
  3. [§4, Figs. 3–4] PC1 explains more than 99% of the variance in both PCA plots; clarify how the shared PCA basis is constructed and whether the 'shift toward the zero-limit point' conclusion is stable when more principal components are retained.
  4. [§5.2, Fig. 7] Specify whether the Hessian and gradient measurements used to compute c*_spike are taken on the full training set or on a mini-batch. The theoretical boundary is derived for deterministic GD, while Fig. 7a is from mini-batch SGD.
  5. [§6.2] The module-wise top-eigenvector decomposition is informative, but the contribution ratio φ_m concerns only the top eigenvector; it is not a statement about the Hessian block's overall sharpness. Please add a caveat that near-degenerate eigenvalues could make this diagnostic unstable.

Circularity Check

0 steps flagged

No significant circularity; the claimed boundaries follow from the scale-invariance identity and are checked against independent scaling measurements.

full rationale

The core derivation (Theorem 5.1, Propositions 5.2 and 5.3) starts from the positive scale-invariance identity L(αu,v)=L(u,v) and the standard EoS/spike Taylor conditions. The curvature scaling λmax(Huu(αu,v)) = α^{-2}λmax(Huu(u,v)) is a direct consequence of that identity, and the weight-norm boundaries are algebraic re-expressions of the stability/spike thresholds; no parameter is fitted to the observed loss spikes. The empirical scaling-law test in Fig. 5 validates the predicted α^{-2} growth independently of the spike-alignment plots. The main weaknesses are logical gaps rather than circularity: Prop. 5.2 implicitly assumes the current weight direction is a scalar multiple of the reference vector (radial rescaling), and Prop. 5.3 replaces the actual spike condition ηλ_grad(H) > 2 by the restricted quantity ηλ_grad(Huu) without an inequality linking the two. These are unproven bridges in the claimed mechanism, but they do not make the derivation equivalent to its inputs. The self-citations (Bai et al. 2025 for λ_grad and the filtering rationale; Li et al. 2025 for threshold-excursion behavior; Zhang et al. 2025 for the synthetic task) are auxiliary and not load-bearing: the spike condition is re-derived in Appendix A.4.2, and the synthetic task is only a testbed. Hence no circular step can be exhibited with the required specificity.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 1 invented entities

The central derivation rests on scale invariance, Cauchy interlacing, and EoS; the most fragile input is the unstated direction-fixed assumption. No new physical entities are introduced; the boundary is a data-dependent diagnostic.

free parameters (1)
  • Interval-filtering thresholds = 30 iterations (merge), 200 iterations (discard)
    Post-hoc thresholds used in Section 5.2 / Fig. 7 to filter unstable intervals before reporting alignment with loss spikes; no principled justification, and they directly affect the apparent quality of the prediction.
axioms (5)
  • domain assumption Positive scale invariance of the loss: L(αu,v)=L(u,v) for all α>0 (Eq. 1)
    Real BN/LN networks are only approximately scale-invariant; biases before normalization and residual connections break exact invariance, as the paper itself notes for ResNet-50.
  • standard math Cauchy interlacing theorem for symmetric matrices (Lemma A.2)
    Used in the proof of Theorem 5.1 to lower-bound the full Hessian's top eigenvalue by that of the uu block.
  • domain assumption Edge-of-Stability stability criterion: GD stable iff η λmax(H) ≤ 2 (Eq. 3)
    Borrowed from prior EoS literature; used to translate curvature into a weight-norm threshold.
  • domain assumption Second-order Taylor expansion is sufficient to characterize loss spikes (A.4.2)
    The spike condition η λgrad > 2 ignores higher-order terms, stochastic gradients, and multi-step dynamics.
  • ad hoc to paper Direction of the scale-invariant vector remains fixed so ρ is constant
    Proposition 5.2's derivation of a norm-only boundary assumes λmax(Huu(w,v)) = ρ/∥w∥², which requires w and the reference u to be collinear; not proven or tested.
invented entities (1)
  • Weight-norm criticality boundary (c*, c*_spike) independent evidence
    purpose: Explains and predicts loss spikes via a norm threshold derived from curvature measurements
    This is a derived diagnostic criterion, not a physical entity. It has a falsifiable handle: the norm-crossing prediction tested in Figs. 7 and 14, though its derivation and validation have the caveats noted above.

pith-pipeline@v1.3.0-alltime-deepseek · 18886 in / 14105 out tokens · 157931 ms · 2026-08-01T08:42:59.653248+00:00 · methodology

0 comments
read the original abstract

Most explanations of training instability focus on \emph{learning-rate criticality}, typically characterized by the Edge of Stability, beyond which optimization becomes unstable. We argue that, in practical deep neural network training, there is an additional and often overlooked \emph{weight-norm criticality}. This criticality is induced by the interaction between normalization (which introduces scale-invariant components) and weight decay (which persistently shrinks parameter norms). As the weight decay coefficient increases, the norms of scale-invariant weights are progressively driven toward zero. Meanwhile, the sharpness of the loss landscape increases rapidly, destabilizing the optimization dynamics and resulting in abrupt loss spikes. This perspective provides a rationale for why weight penalties can improve generalization yet cannot be made arbitrarily strong: excessive decay drives scale-invariant weight norms past a critical boundary and destabilizes training. Our work provides a new mechanistic understanding of loss spikes through the lens of \emph{weight-norm criticality}. Moreover, \emph{weight-norm criticality} yields testable predictions that we validate empirically in networks with scale-invariant components, providing empirical support for the proposed mechanism.

Figures

Figures reproduced from arXiv: 2607.21005 by Xiaolong Li, Zhangchen Zhou, Zhi-Qin John Xu.

Figure 1
Figure 1. Figure 1: The impact of weight decay 𝜆 on training loss. Larger weight decay leads to more frequent loss spikes. (a) Training loss trajectories of a 16-layer, 16-head 187M Transformer on a 100B-token corpus. (b) ResNet-50 trained on the CIFAR-100 dataset. In training deep neural networks, the loss spike phenomenon—referring to a sudden increase in the training loss at certain steps—is often encountered. The mecha￾ni… view at source ↗
Figure 2
Figure 2. Figure 2: Mechanistic probe on MNIST with optional normalization. Training loss trajectories of fully connected neural networks trained on MNIST with SGD under a sweep of the weight decay coefficient. (a) No normalization. Linear → tanh hidden blocks. (b) BatchNorm (BN). Linear → BN → tanh hidden blocks. (c) LayerNorm (LN). Linear → LN → tanh hidden blocks. the effective learning rate to infinity and resulting in tr… view at source ↗
Figure 4
Figure 4. Figure 4: Three-layer FNN with BN on 𝑦 = 𝑥1 + 2𝑥2 function. The learning rate is fixed to be 𝜂 = 0.03. (a) Training loss trajectories under a sweep of the weight decay coefficient. (b) PCA visualization of training trajectories under varying weight decay. The black squares mark the final parameter states and 𝜆max denotes the largest eigenvalue of the Hessian matrix at the training endpoint. The colored stars represe… view at source ↗
Figure 3
Figure 3. Figure 3: PCA visualization of training trajectories for a five￾layer FNN with BN on MNIST under varying weight decay. The learning rate is fixed to be 𝜂 = 0.003. The black squares mark the final parameter states and 𝜆max denotes the largest eigenvalue of the Hessian matrix at the training endpoint. The colored stars represent the derived parameters obtained by freezing the output layer of the trained models and pus… view at source ↗
Figure 5
Figure 5. Figure 5: Verification of the theoretical lower bound via pa￾rameter scaling. (a) Three-layer FNN with BN on synthetic data. The learning rate is fixed to 𝜂 = 0.03. (b) ResNet-50 trained on the CIFAR-100 dataset. The learning rate is fixed to 𝜂 = 0.002. Scatters are the top eigenvalues of the full Hessian plotted against scaled scale-invariant components. The curves represent the direct scaling of the top eigenvalue… view at source ↗
Figure 6
Figure 6. Figure 6: The evolution of the sharpness. Three-layer FNN with BN on synthetic data. The learning rate and weight decay coefficient are fixed to be 𝜂 = 0.03 and 𝜆 = 0.1. As training progresses, the 𝜆max of the Hessian in the scale-invariant layer (orange) increases rapidly, approaching that of the full parameter space (blue) and becoming dominant, far exceeding the 𝜆max of the non-scale-invariant layers (green). The… view at source ↗
Figure 7
Figure 7. Figure 7: Weight-norm criticality and instability under BN. Training dynamics of FNN with BN. In each subfigure, the top panel shows the training loss trajectory during the later stages of optimization, while the bottom panel plots the weight norm ∥𝑊1 ∥ of the first (scale-invariant) layer (black) together with the stability boundary 𝑐 ∗ 𝑠 𝑝𝑖𝑘𝑒 (red). Red shaded regions indicate intervals where ∥𝑊1 ∥ persistently fa… view at source ↗
Figure 8
Figure 8. Figure 8: Evolution of the module-wise decomposition of the top Hessian eigenvector during training on the 3𝑥 → 𝑥 task, following Zhang et al. (2025); see Appendix A.1.5 for full details. (a) applies weight decay to all modules, while (b) disables weight decay for the MLP modules. Different colors correspond to different modules. 0 2k 4k 6k 8k 10k 12k Iteration 10 1 3 × 10 0 4 × 10 0 6 × 10 0 Loss 2.720 2.548 wd=3 f… view at source ↗
Figure 9
Figure 9. Figure 9: Evolution of training loss for a 187M-parameter Trans￾former, comparing standard weight decay against a configuration where weight decay is excluded from the MLP modules in each layer. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Training loss on the synthetic 3𝑥 → 𝑥 next-token prediction task under varying weight decay. Shown are full-batch training loss trajectories for different weight decay coefficients. Each iteration corresponds to a single full-batch parameter update. All other model and optimization settings are kept fixed. A.2. Weight-Norm Shrinkage in Scale-Invariant Layers We plot the evolution of weight norms across tr… view at source ↗
Figure 11
Figure 11. Figure 11: Weight decay selectively contracts scale-invariant layers. Layer-wise weight-norm trajectories during training at learning rate 𝜂 = 0.03 for different weight decay coefficients. The norms of the scale-invariant layers (fc1, fc2) decay significantly faster than that of the non-scale-invariant layer (fc-out). A.3. Ablation Study: Weight Decay Applied Only to Non-Scale-Invariant Layers Additionally, we condu… view at source ↗
Figure 12
Figure 12. Figure 12: Training loss with weight decay applied only to non-scale-invariant layers. Training loss trajectories under different weight decay coefficients when regularization is applied exclusively to the non-scale-invariant layer (fc-out). Results are shown for two learning rates, 𝜂 = 0.01 (left) and 𝜂 = 0.03 (right). In contrast to the standard setting where weight decay is applied to all layers, the loss curves … view at source ↗
Figure 13
Figure 13. Figure 13: PCA visualization of loss landscapes and training trajectories with non-scale-invariant-only weight decay. Two￾dimensional PCA projections of parameter trajectories and the corresponding loss landscapes for the same settings as in [PITH_FULL_IMAGE:figures/full_fig_p014_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Layer-wise stability boundary analysis. Late-training evolution of the weight norms and predicted stability boundaries for the second, third, and fourth scale-invariant layers in the MNIST experiment (𝜂 = 0.005, 𝜆 = 0.01). The same interval-filtering procedure as in the main text is applied. Compared to the first layer, boundary crossings in deeper layers are less frequent and less persistent, though sust… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

73 extracted references · 2 canonical work pages

  1. [1]

    arXiv preprint arXiv:1706.05350 , year=

    L2 regularization versus batch and weight normalization , author=. arXiv preprint arXiv:1706.05350 , year=

    Pith/arXiv arXiv

  2. [2]

    International Conference on Learning Representations , year=

    Three Mechanisms of Weight Decay Regularization , author=. International Conference on Learning Representations , year=

  3. [3]

    Norm matters: efficient and accurate normalization schemes in deep networks , volume =

    Hoffer, Elad and Banner, Ron and Golan, Itay and Soudry, Daniel , booktitle =. Norm matters: efficient and accurate normalization schemes in deep networks , volume =

  4. [4]

    Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate , volume =

    Li, Zhiyuan and Lyu, Kaifeng and Arora, Sanjeev , booktitle =. Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate , volume =

  5. [5]

    Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction , volume =

    Lyu, Kaifeng and Li, Zhiyuan and Arora, Sanjeev , booktitle =. Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction , volume =

  6. [6]

    Online Normalization for Training Neural Networks , volume =

    Chiley, Vitaliy and Sharapov, Ilya and Kosson, Atli and Koster, Urs and Reece, Ryan and Samaniego de la Fuente, Sofia and Subbiah, Vishal and James, Michael , booktitle =. Online Normalization for Training Neural Networks , volume =

  7. [7]

    Spherical Motion Dynamics: Learning Dynamics of Normalized Neural Network using SGD and Weight Decay , volume =

    Wan, Ruosi and Zhu, Zhanxing and Zhang, Xiangyu and Sun, Jian , booktitle =. Spherical Motion Dynamics: Learning Dynamics of Normalized Neural Network using SGD and Weight Decay , volume =

  8. [8]

    arXiv preprint arXiv:1910.07454 , year=

    An exponential learning rate schedule for deep learning , author=. arXiv preprint arXiv:1910.07454 , year=

    Pith/arXiv arXiv 1910

  9. [9]

    Proceedings of the AAAI Conference on Artificial Intelligence , author=

    Understanding the Disharmony between Weight Normalization Family and Weight Decay , volume=. Proceedings of the AAAI Conference on Artificial Intelligence , author=. 2020 , month=. doi:10.1609/aaai.v34i04.5904 , number=

    work page doi:10.1609/aaai.v34i04.5904 2020

  10. [10]

    On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay , volume =

    Lobacheva, Ekaterina and Kodryan, Maxim and Chirkova, Nadezhda and Malinin, Andrey and Vetrov, Dmitry P , booktitle =. On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay , volume =

  11. [11]

    Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes , volume =

    Kodryan, Maxim and Lobacheva, Ekaterina and Nakhodnov, Maksim and Vetrov, Dmitry P , booktitle =. Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes , volume =

  12. [12]

    Greenwade

    George D. Greenwade. The C omprehensive T ex A rchive N etwork ( CTAN ). TUGBoat. 1993

    1993

  13. [13]

    NeurIPS 2023 Workshop Heavy Tails in Machine Learning , year=

    Adaptive Gradient Methods at the Edge of Stability , author=. NeurIPS 2023 Workshop Heavy Tails in Machine Learning , year=

    2023

  14. [14]

    arXiv preprint arXiv:2002.09572 , year=

    The break-even point on optimization trajectories of deep neural networks , author=. arXiv preprint arXiv:2002.09572 , year=

    Pith/arXiv arXiv 2002

  15. [15]

    arXiv preprint arXiv:1807.05031 , year=

    On the relation between the sharpest directions of DNN loss and the SGD step length , author=. arXiv preprint arXiv:1807.05031 , year=

    Pith/arXiv arXiv

  16. [16]

    arXiv preprint arXiv:2003.02218 , year=

    The large learning rate phase of deep learning: the catapult mechanism , author=. arXiv preprint arXiv:2003.02218 , year=

    Pith/arXiv arXiv 2003

  17. [17]

    arXiv preprint arXiv:1808.02941 , year=

    On the convergence of a class of adam-type algorithms for non-convex optimization , author=. arXiv preprint arXiv:1808.02941 , year=

    Pith/arXiv arXiv

  18. [18]

    Transactions on Machine Learning Research , year=

    On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization , author=. Transactions on Machine Learning Research , year=

  19. [19]

    The 22nd international conference on artificial intelligence and statistics , pages=

    On the convergence of stochastic gradient descent with adaptive stepsizes , author=. The 22nd international conference on artificial intelligence and statistics , pages=. 2019 , organization=

    2019

  20. [20]

    International conference on artificial intelligence and statistics , pages=

    Linear convergence of adaptive stochastic gradient descent , author=. International conference on artificial intelligence and statistics , pages=. 2020 , organization=

    2020

  21. [21]

    arXiv preprint arXiv:2003.02395 , year=

    A simple convergence proof of adam and adagrad , author=. arXiv preprint arXiv:2003.02395 , year=

    Pith/arXiv arXiv 2003

  22. [22]

    Journal of Machine Learning Research , volume=

    A general system of differential equations to model first-order adaptive algorithms , author=. Journal of Machine Learning Research , volume=

  23. [23]

    The Thirty-eighth Annual Conference on Neural Information Processing Systems , year =

    ADOPT: Modified Adam Can Converge with Any _2 with the Optimal Rate , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year =

  24. [24]

    Journal of Machine Learning Research , volume=

    Palm: Scaling language modeling with pathways , author=. Journal of Machine Learning Research , volume=

  25. [25]

    Advances in neural information processing systems , volume=

    Cogview: Mastering text-to-image generation via transformers , author=. Advances in neural information processing systems , volume=

  26. [26]

    2025 , eprint=

    Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs , author=. 2025 , eprint=

    2025

  27. [27]

    Thirty-seventh Conference on Neural Information Processing Systems , year=

    Normalization Layers Are All That Sharpness-Aware Minimization Needs , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=

  28. [28]

    2025 , institution =

    Tuning Adam(W): Default _2 May Be Too Large , author =. 2025 , institution =

    2025

  29. [29]

    International Conference on Learning Representations , year =

    On the Variance of the Adaptive Learning Rate and Beyond , author=. International Conference on Learning Representations , year =

  30. [30]

    Advances in neural information processing systems , volume=

    Adam can converge without any modification on update rules , author=. Advances in neural information processing systems , volume=

  31. [31]

    2025 , url=

    Anchor function: a type of benchmark functions for studying language models , author=. 2025 , url=

    2025

  32. [32]

    2005 , edition =

    Saber Elaydi , title =. 2005 , edition =

    2005

  33. [33]

    arXiv preprint arXiv:2502.02007 , year=

    Reasoning Bias of Next Token Prediction Training , author=. arXiv preprint arXiv:2502.02007 , year=

    Pith/arXiv arXiv

  34. [34]

    Journal of Machine Learning , year =

    Ma, Chao and Kunin, Daniel and Wu, Lei and Ying, Lexing , title =. Journal of Machine Learning , year =. doi:https://doi.org/10.4208/jml.220404 , url =

    work page doi:10.4208/jml.220404

  35. [35]

    International Conference on Learning Representations , year=

    Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability , author=. International Conference on Learning Representations , year=

  36. [36]

    Advances in Neural Information Processing Systems , volume=

    How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective , author=. Advances in Neural Information Processing Systems , volume=

  37. [37]

    arXiv preprint arXiv:1412.6980 , year=

    Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=

    Pith/arXiv arXiv

  38. [38]

    International conference on machine learning , pages=

    Understanding the unstable convergence of gradient descent , author=. International conference on machine learning , pages=. 2022 , organization=

    2022

  39. [39]

    arXiv preprint arXiv:1802.08770 , year=

    A walk with sgd , author=. arXiv preprint arXiv:1802.08770 , year=

    Pith/arXiv arXiv

  40. [40]

    arXiv preprint arXiv:1711.04623 , year=

    Three factors influencing minima in sgd , author=. arXiv preprint arXiv:1711.04623 , year=

    Pith/arXiv arXiv

  41. [41]

    On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length , author=

  42. [42]

    Advances in Neural Information Processing Systems , volume=

    Analyzing sharpness along GD trajectory: Progressive sharpening and edge of stability , author=. Advances in Neural Information Processing Systems , volume=

  43. [43]

    The Eleventh International Conference on Learning Representations , year=

    Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability , author=. The Eleventh International Conference on Learning Representations , year=

  44. [44]

    The Break-Even Point on Optimization Trajectories of Deep Neural Networks , author=

  45. [45]

    International Conference on Learning Representations , year=

    The Break-Even Point on Optimization Trajectories of Deep Neural Networks , author=. International Conference on Learning Representations , year=

  46. [46]

    International Conference on Machine Learning , pages=

    Understanding gradient descent on the edge of stability in deep learning , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  47. [47]

    Mathematical and scientific machine learning , pages=

    A qualitative study of the dynamic behavior for adaptive gradient algorithms , author=. Mathematical and scientific machine learning , pages=. 2022 , organization=

    2022

  48. [48]

    Journal of Computational Mathematics , year=

    Loss spike in training neural networks , author=. Journal of Computational Mathematics , year=

  49. [49]

    arXiv preprint arXiv:2410.05192 , year=

    Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective , author=. arXiv preprint arXiv:2410.05192 , year=

    Pith/arXiv arXiv

  50. [50]

    International Conference on Learning Representations , year=

    On the Convergence of Adam and Beyond , author=. International Conference on Learning Representations , year=

  51. [51]

    Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages=

    A sufficient condition for convergences of adam and rmsprop , author=. Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages=

  52. [52]

    arXiv preprint arXiv:2304.09871 , year=

    A theory on adam instability in large-scale machine learning , author=. arXiv preprint arXiv:2304.09871 , year=

    Pith/arXiv arXiv

  53. [53]

    arXiv preprint arXiv:2502.15499 , year=

    Scale-distribution decoupling: Enabling stable and effective training of large language models , author=. arXiv preprint arXiv:2502.15499 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2312.16903 , year=

    Spike no more: Stabilizing the pre-training of large language models , author=. arXiv preprint arXiv:2312.16903 , year=

    Pith/arXiv arXiv

  55. [55]

    International Conference on Machine Learning , pages=

    Stabilizing transformer training by preventing attention entropy collapse , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  56. [56]

    On the Relation Between the Sharpest Directions of

    Stanislaw Jastrzebski and Zachary Kenton and Nicolas Ballas and Asja Fischer and Yoshua Bengio and Amost Storkey , booktitle=. On the Relation Between the Sharpest Directions of. 2019 , url=

    2019

  57. [57]

    International Conference on Learning Representations , year=

    On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization , author=. International Conference on Learning Representations , year=

  58. [58]

    Transactions on Machine Learning Research , issn=

    On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization , author=. Transactions on Machine Learning Research , issn=. 2024 , url=

    2024

  59. [59]

    Transactions on Machine Learning Research , issn=

    A Simple Convergence Proof of Adam and Adagrad , author=. Transactions on Machine Learning Research , issn=. 2022 , url=

    2022

  60. [60]

    2021 , url=

    Naichen Shi and Dawei Li and Mingyi Hong and Ruoyu Sun , booktitle=. 2021 , url=

    2021

  61. [61]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    Pith/arXiv arXiv

  62. [62]

    arXiv e-prints , pages=

    The llama 3 herd of models , author=. arXiv e-prints , pages=

  63. [63]

    arXiv preprint arXiv:2505.21829 , year=

    In Search of Adam's Secret Sauce , author=. arXiv preprint arXiv:2505.21829 , year=

    arXiv

  64. [64]

    The Thirteenth International Conference on Learning Representations , year=

    Understanding Optimization in Deep Learning with Central Flows , author=. The Thirteenth International Conference on Learning Representations , year=

  65. [65]

    arXiv preprint arXiv:2506.04805 , year=

    Adaptive Preconditioners Trigger Loss Spikes in Adam , author=. arXiv preprint arXiv:2506.04805 , year=

    Pith/arXiv arXiv

  66. [66]

    International conference on machine learning , pages=

    Batch normalization: Accelerating deep network training by reducing internal covariate shift , author=. International conference on machine learning , pages=. 2015 , organization=

    2015

  67. [67]

    arXiv preprint arXiv:1607.06450 , year=

    Layer normalization , author=. arXiv preprint arXiv:1607.06450 , year=

    Pith/arXiv arXiv

  68. [68]

    arXiv preprint math/0502408 , year=

    A very short proof of Cauchy's interlace theorem for eigenvalues of Hermitian matrices , author=. arXiv preprint math/0502408 , year=

    Pith/arXiv arXiv

  69. [69]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Deep residual learning for image recognition , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  70. [70]

    Alex Krizhevsky , title=

  71. [71]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  72. [72]

    ATT Labs [Online]

    MNIST handwritten digit database , author=. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , volume=

  73. [73]

    arXiv preprint arXiv:1711.05101 , year=

    Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

    Pith/arXiv arXiv