Pith. sign in

REVIEW 4 major objections 7 minor 38 references

New Evidence of the Two-Phase Learning Dynamics of Neural Networks

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Neural network training passes through an inflection point: before it, tiny parameter perturbations cause large functional divergence (the Chaos Effect), and after it, the network's kernel evolution is confined to a narrow cone (the Cone…

desk verdict A plausible interval-wise reframing of a known two-phase picture, with two named effects that need tighter experimental evidence before they carry much weight. read the letter →

arxiv 2505.13900 v1 pith:U7SXCBA4 submitted 2025-05-20 cs.LG

Zhanpeng Zhou , Yongyi Yang , Mahito Sugiyama , Junchi Yan This is my paper · ORCID
classification cs.LG
keywords two-phaselearningdynamicschaoseffectconeempiricalneuraltangentkernelphasetransitionintrainingperturbationsensitivitylossbarrierslinearmodeconnectivity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that deep network training is not one continuous process but two qualitatively different phases separated by an inflection point. Using an interval-wise analysis that compares network states across time windows rather than at isolated checkpoints, it reports two phenomena: the Chaos Effect, in which a tiny parameter perturbation injected early in training produces large loss barriers and functional disagreement later, and the Cone Effect, in which the empirical neural tangent kernel keeps changing after the inflection point but only within a narrow angular cone in function space. If correct, training has a critical period after which the model enters a constrained refinement regime that is still nonlinear and that linear (lazy) training cannot reproduce. The paper is an empirical study on VGG-16 and ResNet-20 on CIFAR-10, and it offers this two-phase picture as a structural description of how exploration gives way to refinement.

What carries the argument

The machinery is an interval-wise analysis framework: instead of measuring properties at single checkpoints, the paper compares pairs of time points along the trajectory using four pairwise matrices: parameter dissimilarity $C$ (cosine distance between flattened parameter vectors), kernel distance $S$ (cosine distance between empirical neural tangent kernel matrices $H(\theta)$), loss barriers $B$ (maximized test-loss increase along the linear interpolation between two models), and the disagreement rate $D$ (fraction of test inputs classified differently). The eNTK matrix, with entries $H(\theta)_{i,j}$ equal to the inner product of the gradients of the network outputs on inputs $i$ and $j$, is the central object for the Cone Effect, because the claim that 'the kernel continues to change but is trapped in a tight angular region' is exactly a statement about the trajectory of $H(\theta)$ in matrix space. The perturbation experiment for the Chaos Effect rests on comparing two runs that are designed to be identical apart from the injected $\epsilon$.

What would settle it

Train two networks with identical initialization but deliberately different random seeds for mini-batch order and augmentation while keeping everything else fixed, and measure the loss barrier and disagreement rate at the same $t_0$ and $t_1$ as in the paper; if the pre-inflection divergence is similar in magnitude to the synchronized-perturbation case, the Chaos Effect cannot be attributed to perturbation sensitivity. Conversely, if the same experiment uses a perturbation far larger than $10^{-7}$ after the inflection point and divergence remains tiny, the claimed stability of the second phase would be supported.

Watch

Extended reading notes

Core claim

The central discovery is a two-phase structure in learning dynamics, demonstrated by two interval-wise measurements. First, when two identically initialized networks are trained with the same stochastic gradient noise and one receives a parameter perturbation of norm $10^{-7}$ at time $t_0$, the divergence measured later at $t_1$ depends sharply on whether $t_0$ lies before or after an inflection point (around iteration 2500 for VGG-16 and roughly iterations 100–500 for ResNet-20): perturbations before this point lead to substantial loss barriers and high disagreement rates, while perturbations after it leave the trajectories close. Second, the kernel distance between eNTK matrices at different times shows that in the early phase the kernel evolves chaotically, while in the second phase the kernel trajectory is confined to a narrow cone around a reference kernel: the distance to the reference grows then stabilizes, and the distance between adjacent iterates is bounded by the same value for different step sizes $\mathrm{d}t$. The paper also shows that switching from standard training to linearized (lazy) training at later times yields better test performance, indicating that the confined second phase still contributes nonlinear refinement beyond the lazy regime.

Load-bearing premise

The chaos-effect result assumes that the two training runs are identical except for the injected perturbation, meaning the same mini-batch order, data augmentation, and stochastic gradient noise, yet the paper gives no mechanism or seed that guarantees this synchronization; if the noise is not actually identical, the observed divergence could come from independent SGD noise rather than sensitivity to the perturbation.

Editorial extensions

If this is right

  • Training has a critical period: interventions such as data corruption, learning-rate changes, or weight perturbations early in training can have outsized and possibly irreversible effects, while the same interventions later are comparatively benign.
  • The second phase is not a lazy regime: linearized training with a frozen kernel cannot reproduce the final model's performance, so analyses that treat late training as linear miss a real part of the learning.
  • The inflection point is a reproducible landmark: the same time point appears in the parameter dissimilarity, loss barrier, disagreement rate, and eNTK evolution, giving a concrete coordinate for where the regime change happens.
  • The two-phase pattern holds across architectures: both VGG-16 and ResNet-20 on CIFAR-10 show the same qualitative transition, suggesting the phenomenon is not an artifact of one model family.
  • The eNTK continues to evolve after the transition but only along a constrained angular direction, so tracking the kernel's angular position rather than its magnitude can indicate when the model has entered the refinement phase.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: the plateau value of the kernel distance in the second phase may correlate with final generalization, so measuring this 'cone width' across datasets and hyperparameters could turn the cone effect into a predictive diagnostic.
  • The inflection point may coincide with the progressive-sharpening or edge-of-stability transition studied in other work; the paper does not test this alignment, but if it holds, the chaos-cone picture would unify with sharpness-based accounts of training.
  • Varying the perturbation norm $\epsilon$ across orders of magnitude before the inflection point would reveal whether the early regime is truly chaotic (exponential divergence) or merely high-gain but linear, a distinction the paper does not address.
  • Running the perturbation study with explicitly independent SGD noise would isolate whether the effect reflects sensitivity to initial conditions or merely noise-induced divergence, directly testing the synchronization assumption.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes an 'interval-wise' analysis of neural network training dynamics, comparing network states at two different times rather than at isolated checkpoints. It reports two empirical phenomena on VGG-16 and ResNet-20 for CIFAR-10: the 'Chaos Effect,' in which a tiny parameter perturbation injected early in training leads to large later divergence in parameters, loss barriers, and test disagreement, while the same perturbation injected later has little effect; and the 'Cone Effect,' in which the empirical Neural Tangent Kernel (eNTK) continues to evolve after an early inflection point but remains confined to a narrow angular region. The authors interpret these observations as evidence of a two-phase learning process: an early chaotic, highly sensitive phase followed by a stable but still nonlinear refinement phase. They also report a switching experiment intended to show that the second phase provides generalization advantages over a purely lazy/linearized regime. The paper is explicitly empirical and defers theory to future work.

Significance. If the observations are reproducible and robust, the interval-wise perspective is a useful addition to the literature on training dynamics, complementing existing point-wise analyses of sharpness, Hessian spectra, and linear mode connectivity. The use of several independent observables (parameter dissimilarity, loss barrier, disagreement rate, kernel distance) is a genuine strength, as is the explicit statement of the two-phase conjecture. However, the paper is entirely empirical and provides no code, no seeds, and no error bars, which is a serious reproducibility gap for a study whose central claims are quantitative (inflection points, plateau levels). The reported effects are descriptive rather than predictive; no parameter-free derivation or falsifiable quantitative prediction is offered. With the requested controls and repeated-seed evidence, the paper could make a solid empirical contribution.

major comments (4)
  1. [Section 4, Experimental Design and Figure 2] The claim that the two runs use 'the same stochastic gradient noise' is not operationalized. The manuscript does not state how synchronization is achieved (e.g., identical random seed, fixed minibatch order, fixed augmentation sequence), and no zero-perturbation control is reported. If the two runs use different data orders or augmentation randomness, the divergence, loss barriers, and disagreement rates in Figure 3 could arise from independent SGD noise even when epsilon = 0, which would sever the link between the perturbation and the observed divergence. Please specify the synchronization mechanism, provide seeds or code, and report an epsilon = 0 baseline for the chaos-effect experiment.
  2. [Section 3, Main Experimental Setup, and Figures 3-6] All results appear to come from a single training run per architecture. No seed repetitions, confidence intervals, or error bars are reported, so the claimed inflection points (e.g., '2500 iteration' for VGG-16 and '100-500 iteration' for ResNet-20) and the cone-effect plateaus cannot be distinguished from run-to-run variability. Please repeat the experiments with multiple seeds and report means and ranges, or clearly state that the results are single-run and weaken the corresponding claims.
  3. [Section 5, 'The Non-linear Advantages of the Cone Effect', Figure 6] The switching experiment theta_std->lin(T; t) confounds the switching time t with the amount of standard training: later switching means more standard SGD steps and fewer linearized steps. The observed increase of test accuracy with t may simply reflect longer standard training rather than any benefit of the cone effect. A proper control is needed, such as comparing against standard training for t iterations only, or matching the loss or accuracy at the switching point, or using a fixed total budget with different splits. As presented, the claim that the cone effect provides advantages over the lazy regime is not established.
  4. [Sections 1 and 4, 'The Chaos Effect'] The term 'chaotic' is used as a dynamical-systems claim, but the evidence is limited to divergence after one fixed perturbation norm (||epsilon|| = 10^-7). Sensitive dependence on initial conditions is not quantified: there is no analysis of how divergence scales with epsilon, no divergence-rate or doubling-time measurement, and no saturation analysis. Since the paper itself labels the transition a 'conjecture,' please either add such a scaling analysis or consistently phrase the finding as 'sensitivity to perturbations' rather than 'chaotic dynamics,' so that the abstract's 'chaotic to stable' claim is precisely supported.
minor comments (7)
  1. [Equation (1)] The denominator should be ||vec(theta_i)||_2 * ||vec(theta_j)||_2; as written, the formula uses ||vec(theta_j)||_2 twice.
  2. [Equation (4)] The denominator should be ||H(theta_i)||_F * ||H(theta_j)||_F; as written, the subscript on the first norm is malformed.
  3. [Section 3, Basic Notations] The notation f: D x R^p -> R with scalar output is inconsistent with a c-class classification problem and with the disagreement rate in Equation (6), which compares predicted labels. Please clarify whether f returns logits, probabilities, or argmax predictions, or introduce a separate prediction function.
  4. [Figure 3 and Section 3] The axes of Figure 3 are said to be in iterations, but the checkpoint interval k used to sample {theta_t} is not specified, nor is the total number of checkpoints. Please state these values.
  5. [Section 5, Key Takeaways] The sentence 'the neural networks the neural network continues to evolve' contains a duplicated phrase; please correct.
  6. [Throughout] There are minor wording issues, including 'the inflection point severs as a hallmark' (should be 'serves'), and inconsistent use of 'non-chaotic' versus 'nonchaotic.' Please copyedit.
  7. [Reproducibility] No code or data availability statement is provided. For an empirical paper, a statement on whether code and training configurations will be released is important.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the two-phase claims rest on direct empirical measurements, not on fitted inputs or self-citation chains.

full rationale

This paper is an empirical study, not a derivation. The quantities used—parameter dissimilarity (Eq. 1), kernel distance (Eq. 4), loss barriers (Eq. 5), and disagreement rate (Eq. 6)—are defined independently of the conclusions and are measured directly from training runs. No parameter is fitted to a subset of data and then renamed as a prediction; the reported inflection point is read off the measured curves and is then corroborated by several independent observables (loss barrier, disagreement rate, eNTK evolution). The Chaos Effect and Cone Effect are presented as observed phenomena rather than as consequences of an assumed model. The footnote stating that the work extends a workshop paper is provenance, not load-bearing self-citation, and the related-work references that include the authors' earlier papers are contextual rather than foundational to the present claims. The paper itself acknowledges its limitations, including the lack of theoretical analysis and restriction to image classification. The skeptical concern about synchronized SGD noise is an experimental-validity issue, not a form of circular reasoning, because it does not make a conclusion equal to an input by construction. Accordingly, no circular step is identified.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central narrative rests on two hand-chosen quantities (inflection point and perturbation size) and on assumptions about synchronization, NTK representativeness, and generalization beyond the tested setup. No new physical entities or formal constructs are introduced beyond the named effects.

free parameters (2)
  • inflection point t* = VGG-16: about 2500 iterations; ResNet-20: about 100-500 iterations
    Identified empirically from parameter dissimilarity and used as the boundary between chaotic and non-chaotic phases; not predicted from theory.
  • perturbation norm epsilon = 1e-7
    Chosen by hand as 'tiny'; the claim that the perturbation is imperceptibly small depends on parameter scale and loss-landscape curvature, which are not reported.
assumptions (4)
  • domain assumption The two runs in the perturbation experiment use exactly the same stochastic gradient noise, including mini-batch order and data augmentation randomness.
    Stated in Section 4 and Figure 2 but no mechanism or seed is given; the Chaos Effect measurement is only meaningful under this assumption.
  • domain assumption The empirical NTK matrix H(theta_t) captures the relevant function-space trajectory of the network.
    Used in Section 3 Equations (2)-(3) to link function evolution to the kernel; assumes the NTK is a faithful proxy for functional change.
  • ad hoc to paper Sensitivity to small perturbations is evidence of chaotic dynamics.
    The paper labels the early phase 'chaotic' from divergence after a small perturbation; this is an interpretive leap because sensitivity alone does not establish chaos in the dynamical-systems sense.
  • domain assumption Observations on CIFAR-10 with VGG-16 and ResNet-20 generalize to deep learning at large.
    The authors state in Limitations that only image classification was tested; the central two-phase claim is extrapolated beyond this setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of New Evidence of the Two-Phase Learning Dynamics of Neural Networks." pith.science (2026). https://pith.science/paper/U7SXCBA4

@misc{pith2026250513900,
  author       = {Pith},
  title        = {Pith review of: New Evidence of the Two-Phase Learning Dynamics of Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U7SXCBA4}},
  note         = {Machine review of arXiv:2505.13900}
}
read the original abstract

Understanding how deep neural networks learn remains a fundamental challenge in modern machine learning. A growing body of evidence suggests that training dynamics undergo a distinct phase transition, yet our understanding of this transition is still incomplete. In this paper, we introduce an interval-wise perspective that compares network states across a time window, revealing two new phenomena that illuminate the two-phase nature of deep learning. i) \textbf{The Chaos Effect.} By injecting an imperceptibly small parameter perturbation at various stages, we show that the response of the network to the perturbation exhibits a transition from chaotic to stable, suggesting there is an early critical period where the network is highly sensitive to initial conditions; ii) \textbf{The Cone Effect.} Tracking the evolution of the empirical Neural Tangent Kernel (eNTK), we find that after this transition point the model's functional trajectory is confined to a narrow cone-shaped subset: while the kernel continues to change, it gets trapped into a tight angular region. Together, these effects provide a structural, dynamical view of how deep networks transition from sensitive exploration to stable refinement during training.

Figures

Figures reproduced from arXiv: 2505.13900 by the authors.

Figure 1
Figure 1. An illustration of the two-phase training dynamics. The optimization trajectory initially [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. The sensitivity of learning dynamics to tiny perturbations. We apply the perturbation ϵ at the time point t0 and compare resulting models at t1 with different metrics. We set ∥ϵ∥0 = 10−7 . Our results are reported for VGG-16 and ResNet-20 on CIFAR-10. (a) The parameter dissimilarity (C)t0,t1 . (b) The loss barrier (B)t0,t1 . (c) The disagreement rate (D)t0,t1 . Note that the t0 and t1 are presented in iterations, no… view at source ↗
Figure 2
Figure 2. The illustration of the injected perturbation. θ0 denotes the initialization. For both θ and θ ′ , the same stochastic gradient noise are applied during training. To answer this question, in the section, we empirically inves￾tigate the sensitivity of neural network learning dynamics to small perturbations. Surprisingly, we find that an inflection point emerges during the training process. We observe that even a tiny… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The kernel distance between every pair of two points at the optimization trajectory {θt} T t=1. Our results are reported for both VGG-16 and ResNet-20 on CIFAR-10. Note that the i and j are presented in iterations, not epochs. Finding I. Optimization trajectory changes…
Figure 5
Figure 5. Figure 5: Constrained learning dynamics in the second phase. (a) The kernel distance between the current iterate θt and a reference point θτ v.s. training iteration t, where τ is varied. (b) The kernel distance between two adjacent iterates θt and θt+dt vs. training iteration t,…
Figure 6
Figure 6. Figure 6: The non-linear advantage of the cone effect. Test accuracy AccDtest (θstd→lin(T;t)) (left) and Test loss LDtest (θstd→lin(T;t)) vs. the switching iteration t. θstd→lin(T;t) represents the model initially trained with standard method up to iteration t, followed by linea…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 26 canonical work pages

  1. [1]

    Critical learning periods in deep neural networks

    Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep neural networks. arXiv preprint arXiv:1711.08856, 2017

    arXiv 2017

  2. [2]

    Ainsworth, Jonathan Hayase, and Siddhartha S

    Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries. InICLR. OpenReview.net, 2023

    work page 2023

  3. [3]

    A convergence theory for deep learning via over- parameterization

    Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 242–252. PMLR, 09–15 Jun 2019

    work page 2019

  4. [4]

    On exact computation with an infinitely wide neural net

    Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    work page 2019

  5. [5]

    A survey on deep learning applied to medical images: from simple artificial neural networks to generative models.Neural Computing and Applications, 35(3):2291–2323, 2023

    Pedro Celard, Eva Lorenzo Iglesias, José Manuel Sorribes-Fdez, Rubén Romero, A Seara Vieira, and Lourdes Borrajo. A survey on deep learning applied to medical images: from simple artificial neural networks to generative models.Neural Computing and Applications, 35(3):2291–2323, 2023

    work page 2023

  6. [6]

    On lazy training in differentiable programming

    Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. 9

    work page 2019

  7. [7]

    Gradient descent on neural networks typically occurs at the edge of stability.arXiv preprint arXiv:2103.00065, 2021

    Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability.arXiv preprint arXiv:2103.00065, 2021

    arXiv 2021

  8. [8]

    Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. InThe Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=nhKHA59gXz

    work page 2023

Show all 38 references
  1. [9]

    Gradient descent finds global minima of deep neural networks

    Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of M...

    2019

  2. [10]

    Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh

    Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. InInternational Conference on Learning Representations, 2019

    2019

  3. [11]

    Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel.Advances in Neural Informatio...

    2020

  4. [12]

    Linear mode connectivity and the lottery ticket hypothesis

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. InInternational Conference on Machine Learning, pages 3259–3269. PMLR, 2020

    2020

  5. [13]

    Schwab, and Ari S

    Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network training. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum? id=Hkl1iRNFwS

    2020

  6. [14]

    An investigation into neural net optimization via hessian eigenvalue density

    Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. InInternational Conference on Machine Learning, pages 2232–2241. PMLR, 2019

    2019

  7. [15]

    Gradient descent happens in a tiny subspace.arXiv preprint arXiv:1812.04754, 2018

    Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace.arXiv preprint arXiv:1812.04754, 2018

    2018 arXiv

  8. [16]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. doi: 10.1109/CVPR.2016.90

    2016 doi

  9. [17]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume ...

    2018

  10. [18]

    The break-even point on optimization trajectories of deep neural networks.arXiv preprint arXiv:2002.09572, 2020

    Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks.arXiv preprint arXiv:2002.09572, 2020

    2002 arXiv

  11. [19]

    Assessing generalization of SGD via disagreement

    Yiding Jiang, Vaishnavh Nagarajan, Christina Baek, and J Zico Kolter. Assessing generalization of SGD via disagreement. InInternational Conference on Learning Representations, 2022. URL https: //openreview.net/forum?id=WvOGCEAQhxl

    2022

  12. [20]

    Wide neural networks of any depth evolve as linear models under gradient descent

    Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R...

    2019

  13. [21]

    Learning overparameterized neural networks via stochastic gradient descent on structured data

    Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 3...

    2018

  14. [22]

    What happens after SGD reaches zero loss? –a mathematical framework

    Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after SGD reaches zero loss? –a mathematical framework. InInternational Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=siCt4xZn5Ve

    2022

  15. [23]

    Task arithmetic in the tangent space: Improved editing of pre-trained models

    Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URLhttps://openreview.net/forum?id=0A9f2jZDGW. 10

    2023

  16. [24]

    Deep learning-based weather prediction: a survey.Big Data Research, 23:100178, 2021

    Xiaoli Ren, Xiaoyong Li, Kaijun Ren, Junqiang Song, Zichen Xu, Kefeng Deng, and Xiang Wang. Deep learning-based weather prediction: a survey.Big Data Research, 23:100178, 2021

    2021

  17. [25]

    Very deep convolutional networks for large-scale image recogni- tion

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In Yoshua Bengio and Yann LeCun, editors,3rd International Conference on Learning Represen- tations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Procee...

    2015 arXiv

  18. [26]

    The directionality of optimization trajectories in neural networks

    Sidak Pal Singh, Bobby He, Thomas Hofmann, and Bernhard Schölkopf. The directionality of optimization trajectories in neural networks. InThe Thirteenth International Conference on Learning Representations,

  19. [27]

    A survey on statistical theory of deep learning: Approximation, training dynamics, and generative models.Annual Review of Statistics and Its Application, 12, 2024

    Namjoon Suh and Guang Cheng. A survey on statistical theory of deep learning: Approximation, training dynamics, and generative models.Annual Review of Statistics and Its Application, 12, 2024

    2024

  20. [28]

    Deep reinforcement learning for robotics: A survey of real-world successes

    Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martín-Martín, and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 28694–28698, 2025

    2025

  21. [29]

    Analyzing sharpness along gd trajectory: Progressive sharpening and edge of stability.Advances in Neural Information Processing Systems, 35:9983–9994, 2022

    Zixuan Wang, Zhouzi Li, and Jian Li. Analyzing sharpness along gd trajectory: Progressive sharpening and edge of stability.Advances in Neural Information Processing Systems, 35:9983–9994, 2022

    2022

  22. [30]

    Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing in...

    2022

  23. [31]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. InProceedings of the IEEE/CVF Conference on Com...

    2022

  24. [32]

    How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective

    Lei Wu, Chao Ma, and Weinan E. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors,Advances in Neural Information Processing Systems, ...

    2018

  25. [33]

    Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation.arXiv preprint arXiv:1902.04760, 2019

    1902 arXiv

  26. [34]

    Swing-by dynamics in concept learning and compositional generalization

    Yongyi Yang, Core Francisco Park, Ekdeep Singh Lubana, Maya Okawa, Wei Hu, and Hidenori Tanaka. Swing-by dynamics in concept learning and compositional generalization. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  27. [35]

    Going beyond linear mode connectivity: The layerwise linear feature connectivity.Advances in neural information processing systems, 36:60853–60877, 2023

    Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan, and Wei Hu. Going beyond linear mode connectivity: The layerwise linear feature connectivity.Advances in neural information processing systems, 36:60853–60877, 2023

    2023

  28. [36]

    On the emergence of cross-task linearity in pretraining-finetuning paradigm

    Zhanpeng Zhou, Zijun Chen, Yilan Chen, Bo Zhang, and Junchi Yan. On the emergence of cross-task linearity in pretraining-finetuning paradigm. InICML, 2024. URL https://openreview.net/forum? id=qg6AlnpEQH

    2024

  29. [37]

    Gradient descent optimizes over-parameterized deep relu networks.Machine learning, 109:467–492, 2020

    Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over-parameterized deep relu networks.Machine learning, 109:467–492, 2020. 11

    2020

  30. [2025]

    URLhttps://openreview.net/forum?id=JY6P45sFDS

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.