The reviewed record of science sign in
Pith

arxiv: 2605.29152 · v1 · pith:MLZBYQUY · submitted 2026-05-27 · cs.LG · math.OC· stat.ML

Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias

Mohua Das , Pierfrancesco Beneventano , Shibshankar Dey , Gareth H. McKinkey , Tomaso Poggio This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-29 13:18 UTCgrok-4.3pith:MLZBYQUYrecord.jsonopen to challenge →

classification cs.LG math.OCstat.ML
keywords initialization memoryinductive biasneural network trainingSGDforgettingregularizationResNetCIFAR-10
0
0 comments X

The pith

The inductive bias of a trained neural network is its architectural prior filtered by the forgetting dynamics of the training pipeline.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Neural networks start with a prior induced by random initialization, but training changes how much of that survives. Experiments on ResNets for CIFAR-10 reveal that low-learning-rate SGD keeps strong dependence on initialization scale, producing up to 26.5 percentage point swings in test accuracy even when training accuracy exceeds 99.5 percent. Adam optimizers and explicit regularization largely remove this dependence by accelerating forgetting of the initial conditions. The result is that effective inductive bias is not architecture alone but architecture after the training process has filtered the starting point.

Core claim

In controlled experiments, initialization memory—the dependence of the final predictor on random initialization scale—persists under gradient-flow-like dynamics such as low-LR SGD but is erased on timescales set by stochastic effects, norm decay, or adaptive preconditioning; therefore the practical bias equals the architectural prior after filtering by forgetting dynamics, and regularizers improve generalization precisely by erasing initialization memory.

What carries the argument

Initialization memory, the dependence of the validation-selected predictor on the scale of the random initialization; it quantifies how much initial bias survives training.

If this is right

  • Low-learning-rate SGD interpolates yet retains initialization memory, leading to large test accuracy variation.
  • Adam-family methods erase the dependence on initialization scale.
  • Pairing larger learning rates with L2 norm control causes SGD to forget initialization.
  • The time scale of forgetting is governed by the size of explicit or implicit regularization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Initialization scale may need to be tuned differently depending on the optimizer used.
  • This forgetting view could explain why certain training choices improve generalization beyond what architecture alone predicts.
  • Extending training time does not necessarily increase forgetting if the regime preserves memory.

Load-bearing premise

Variation in test accuracy across initialization scales after high training accuracy isolates retained initialization memory rather than other differences in optimization.

What would settle it

An experiment in which the accuracy spread across init scales disappears when other optimization factors are controlled while maintaining the same training accuracy.

Figures

Figures reproduced from arXiv: 2605.29152 by Gareth H. McKinkey, Mohua Das, Pierfrancesco Beneventano, Shibshankar Dey, Tomaso Poggio.

Figure 1
Figure 1. Figure 1: SGD remembers initialization; Adam-family methods forget. ResNet-9 under a shared low-learning-rate training procedure. Each curve shows the mean over n = 10 seeds; shaded bands indicate the 10th − 90th percentile range. SGD interpolates, but its generalization gap grows with σw. Adam, AdamW, and Muon show substantially weaker dependence on σw. The norm panels show radial memory: SGD retains sensitivity to… view at source ↗
Figure 2
Figure 2. Figure 2: Large-batch fixed-epoch regimes forget initialization more slowly. ResNet-9 test accuracy at τbest, averaged over n = 10 seeds. Each panel corresponds to one batch size b ∈ {16, 32, 64, 128, 256}; within each panel, rows are optimizers and columns are initialization scales. Cells marked × did not reach 99.5% mean training accuracy at the best-validation-loss checkpoint τbest. SGD shows a strong left-to-rig… view at source ↗
Figure 3
Figure 3. Figure 3: Interpolation is not forgetting. (a,b) Interpolation epoch τinterp versus σw. SGD’s interpolation time grows sharply with σw, especially at large batch sizes; Adam’s is nearly flat. (c,d) Repair gap ∆repair = ValAccτbest − ValAccτinterp . At large batch (b≥128) Adam-family methods continue to gain validation accuracy after interpolation (τbest ≫τinterp, ∆repair >0). At small batch (b≤64) they reach τbest b… view at source ↗
Figure 4
Figure 4. Figure 4: What helps SGD forget initialization? Test accuracy vs. σw for (a) b=16, and (b) b=128; inset shows the spread (pp) per configuration, sorted ascending. Long training alone leaves the curves nearly unchanged; larger learning rates and especially moderate LR with explicit L2 sharply reduce the spread. Spreads use the 5-point grid σw ∈ {0.1, 0.5, 1.0, 1.8, 2.5} [ [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Depth makes poor forgetting dynamics more damaging; pooling is a partial confound. Test accuracy versus σw for ResNet-9, R9-AvgPool, ResNet-56, and ResNet-110. Switching pooling strategy reduces ResNet-9 performance, but the 9-layer R9 remains more robust than ResNet-56, suggesting that depth and optimization difficulty contribute beyond pooling alone. (|∆| = 0.0 pp) with 94.2% test accuracy ( [PITH_FULL_… view at source ↗
Figure 6
Figure 6. Figure 6: Initialization sensitivity decays on regularization timescales, not epoch count. (a– d) |β(t)| versus epoch for the SGD family (left) and adaptive methods (right) at b=16 (top) and b=128 (bottom). Vanilla SGD (red) and SGD+momentum (orange) stay flat; Adam/AdamW/Muon decay steadily; SGD with η=10−2 , L2=10−2 (green) reaches the adaptive noise floor. (e) Vanilla SGD |β| versus TSGD=(1/b) P k η 2 k across al… view at source ↗
Figure 7
Figure 7. Figure 7: Train accuracy vs. initialization scale σw (same layout as [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Test accuracy vs. initialization scale σw for ResNet-9 under BatchNorm + augmentation (blue), LayerNorm + augmentation (orange), and the BatchNorm baseline without augmentation (grey dashed). Lines: mean over seeds; shaded bands: 10th–90th percentile. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Optimizer trajectories on the scalar loss (ab − 1)2 , initialization (a0, b0) = (1, 6). All four panels use the same two learning rates η ∈ {0.01, 0.04} (red and orange, respectively); only the optimizer changes. The black curve is the manifold of global minima ab = 1; magenta stars mark the minimum-norm solutions (±1, ±1). (a) Gradient descent (20 steps): the small-η trajectory stays near the initializati… view at source ↗
read the original abstract

Randomly initialized neural networks induce a prior over functions, but the predictor used in practice is produced only after training. We ask how much of this initial bias survives the training pipeline. To make the question measurable, we introduce initialization memory: the dependence of the validation-selected predictor on the scale of the random initialization. We perform controlled CIFAR-10 experiments on ResNets where initialization memory already sharply separates training regimes. Low-learning-rate SGD can interpolate while still remembering its initialization: on ResNet-9 with batch size $b=128$, test accuracy varies by $26.5$ percentage points across initialization scales despite $\ge99.5\%$ training accuracy. This is not undertraining: extending the same low-learning-rate regime to $5{,}000$ epochs leaves the spread essentially unchanged. In contrast, Adam-family methods largely erase the dependence. SGD can also be made to forget when larger learning rates are paired with explicit $L_2$ norm control. We interpret these findings in terms of the time scale of forgetting: gradient-flow-like dynamics can preserve initialization memory, whereas stochastic finite-step effects, explicit norm decay, and adaptive preconditioning erase it on scales governed by the size of explicit or implicit regularization. The practical inductive bias of a trained network is therefore not the architectural prior alone, but the architectural prior after being filtered by the forgetting dynamics of the training pipeline; and the same regularizers that improve generalization are precisely those that erase memory of initialization.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper claims that trained deep networks retain a measurable dependence on the scale of their random initialization ('initialization memory'), which survives low-learning-rate SGD but is erased by Adam-family optimizers or explicit L2 regularization. This is demonstrated via controlled CIFAR-10 ResNet experiments showing a 26.5 percentage point spread in test accuracy across initialization scales at >=99.5% training accuracy; the spread persists even after extending training to 5000 epochs. The authors interpret the results as evidence that practical inductive bias is the architectural prior filtered by the forgetting dynamics of the training pipeline, with the same regularizers that aid generalization also erasing initialization memory.

Significance. If the reported accuracy spreads can be isolated to retained initialization memory rather than correlated differences in optimization trajectories, the work offers a useful empirical lens on how training dynamics shape effective priors. The controlled regime comparisons (low-LR SGD vs. Adam vs. L2) and the observation that extended epochs do not close the gap provide concrete, falsifiable distinctions between regimes. The absence of free parameters or fitted models in the core measurements is a strength of the empirical design.

major comments (2)
  1. [Abstract] Abstract and the CIFAR-10 ResNet-9 experiments: the central attribution of the 26.5 pp test-accuracy spread to retained 'initialization memory' (rather than other init-scale-dependent trajectory factors such as effective step-size distribution, basin geometry, or final weight norms) is load-bearing for the forgetting-time interpretation, yet the reported controls (high training accuracy plus extension to 5000 epochs) do not constrain these alternatives. High train accuracy plus fixed epoch count is insufficient to isolate the claimed mechanism.
  2. [Abstract] The manuscript reports quantitative separation under controlled setups but provides no details on the number of independent runs, error bars, or exact hyperparameter sweep ranges for the 26.5 pp spread. Without these, it is impossible to assess whether the observed variation is statistically robust or sensitive to minor implementation choices.
minor comments (1)
  1. Notation for 'initialization memory' is introduced without a formal definition or equation; a precise mathematical statement of the dependence being measured would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful reading and constructive feedback. We respond to each major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract and the CIFAR-10 ResNet-9 experiments: the central attribution of the 26.5 pp test-accuracy spread to retained 'initialization memory' (rather than other init-scale-dependent trajectory factors such as effective step-size distribution, basin geometry, or final weight norms) is load-bearing for the forgetting-time interpretation, yet the reported controls (high training accuracy plus extension to 5000 epochs) do not constrain these alternatives. High train accuracy plus fixed epoch count is insufficient to isolate the claimed mechanism.

    Authors: We define initialization memory operationally as the dependence of the validation-selected predictor on initialization scale under a fixed training procedure. The reported experiments show that this dependence survives low-LR SGD even after ≥99.5% training accuracy is reached and after training is extended to 5000 epochs, while the same dependence is erased under Adam or explicit L2 regularization. These regime contrasts are the primary evidence for the forgetting-time view. We agree that additional measurements (e.g., final weight norms across scales or basin geometry) would further constrain alternative explanations and will add a limitations paragraph discussing this point in the revision. revision: partial

  2. Referee: [Abstract] The manuscript reports quantitative separation under controlled setups but provides no details on the number of independent runs, error bars, or exact hyperparameter sweep ranges for the 26.5 pp spread. Without these, it is impossible to assess whether the observed variation is statistically robust or sensitive to minor implementation choices.

    Authors: We agree that these experimental details are necessary. The 26.5 pp figure is computed from multiple independent runs that vary only the initialization scale (different random seeds), and we will include the exact number of runs, standard-error bars, and the precise hyperparameter ranges used for all reported regimes in the revised manuscript and supplementary material. revision: yes

Circularity Check

0 steps flagged

No circularity; purely empirical measurements with no self-referential derivations

full rationale

The paper reports controlled CIFAR-10 experiments on ResNets that measure test-accuracy spread across initialization scales under fixed training-accuracy thresholds. No equations, fitted parameters, or predictions are defined in terms of the target quantities. No self-citations are invoked as load-bearing uniqueness theorems or ansatzes. The central observations (e.g., 26.5 pp spread under low-LR SGD) are direct empirical outputs, not reductions of any input by construction. The work is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on empirical measurements of accuracy variation rather than theoretical derivations; the only addition is the newly introduced measurable quantity.

axioms (1)
  • domain assumption Standard assumptions underlying SGD and Adam dynamics on non-convex loss surfaces
    Invoked when interpreting why low-LR SGD preserves memory while Adam erases it.
invented entities (1)
  • initialization memory no independent evidence
    purpose: Quantify the retained dependence of the validation-selected predictor on random initialization scale
    Newly defined to make the survival of initial bias measurable; no independent evidence outside the paper's experiments.

pith-pipeline@v0.9.1-grok · 5823 in / 1399 out tokens · 35323 ms · 2026-06-29T13:18:32.567648+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

117 extracted references · 8 canonical work pages · 1 internal anchor

  1. [1]

    Under- standing deep learning (still) requires rethinking generalization.Communications of the ACM, 64(3):107–115, 2021

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Under- standing deep learning (still) requires rethinking generalization.Communications of the ACM, 64(3):107–115, 2021

    2021

  2. [2]

    Strogatz.Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering

    Steven H. Strogatz.Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering. Westview Press, 2 edition, 2015

    2015

  3. [3]

    Hirsch, Stephen Smale, and Robert L

    Morris W. Hirsch, Stephen Smale, and Robert L. Devaney.Differential Equations, Dynamical Systems, and an Introduction to Chaos. Academic Press, 3 edition, 2013

    2013

  4. [4]

    On the explicit role of initialization on the convergence and implicit bias of overparametrized linear networks

    Hancheng Min, Salma Tarmoun, René Vidal, and Enrique Mallada. On the explicit role of initialization on the convergence and implicit bias of overparametrized linear networks. InProceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7760–7768. PMLR, 2021

    2021

  5. [5]

    On the role of initialization on the implicit bias in deep linear networks, 2024

    Oria Gruber and Haim Avron. On the role of initialization on the implicit bias in deep linear networks, 2024

    2024

  6. [6]

    Camargo, and Ard A

    Guillermo Valle-Pérez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes because the parameter–function map is biased towards simple functions. InInternational Conference on Learning Representations, 2019

    2019

  7. [7]

    Chris Mingard, Henry Rees, Guillermo Valle-Pérez, and Ard A. Louis. Deep neural networks have an inbuilt occam’s razor.Nature Communications, 16:220, 2025

    2025

  8. [8]

    Deep-layered machines have a built-in occam’s razor.arXiv preprint arXiv:2603.01217, 2026

    Thomas Fink. Deep-layered machines have a built-in occam’s razor.arXiv preprint arXiv:2603.01217, 2026

    work page arXiv 2026

  9. [9]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. InProceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 ofProceedings of Machine Learning Research, pages 249–256. PMLR, 2010

    2010

  10. [10]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. InProceedings of the IEEE International Conference on Computer Vision, pages 1026–1034, 2015

    2015

  11. [11]

    Exponential expressivity in deep neural networks through transient chaos

    Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. InAdvances in Neural Information Processing Systems, volume 29, 2016

    2016

  12. [12]

    Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein

    Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. InInternational Conference on Learning Representations, 2017

    2017

  13. [13]

    Schoenholz, and Surya Ganguli

    Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: Theory and practice. InAdvances in Neural Information Processing Systems, volume 30, 2017

    2017

  14. [14]

    How to start training: The effect of initialization and architecture

    Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. InAdvances in Neural Information Processing Systems, volume 31, 2018

    2018

  15. [15]

    Which neural net architectures give rise to exploding and vanishing gradients? InAdvances in Neural Information Processing Systems, volume 31, 2018

    Boris Hanin. Which neural net architectures give rise to exploding and vanishing gradients? InAdvances in Neural Information Processing Systems, volume 31, 2018. 10

    2018

  16. [16]

    Schoenholz, and Jeffrey Pennington

    Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks. InProceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 5393–5402. ...

    2018

  17. [17]

    Schoenholz

    Minmin Chen, Jeffrey Pennington, and Samuel S. Schoenholz. Dynamical isometry and a mean field theory of RNNs: Gating enables signal propagation in recurrent neural networks. InProceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 873–882. PMLR, 2018

    2018

  18. [18]

    Greg Yang and Edward J. Hu. Tensor programs IV: Feature learning in infinite-width neural networks. InProceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, pages 11727–11737. PMLR, 2021

    2021

  19. [19]

    Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao

    Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer. InAdvances in Neural Information Processing Systems, volume 34, 2021

    2021

  20. [20]

    Self-consistent dynamical field theory of kernel evo- lution in wide neural networks.Journal of Statistical Mechanics: Theory and Experiment, 2023(11):114009, 2023

    Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evo- lution in wide neural networks.Journal of Statistical Mechanics: Theory and Experiment, 2023(11):114009, 2023

    2023

  21. [21]

    Dynamics of finite width kernel and prediction fluctua- tions in mean field neural networks.Journal of Statistical Mechanics: Theory and Experiment, 2024(10):104021, 2024

    Blake Bordelon and Cengiz Pehlevan. Dynamics of finite width kernel and prediction fluctua- tions in mean field neural networks.Journal of Statistical Mechanics: Theory and Experiment, 2024(10):104021, 2024

    2024

  22. [22]

    Deep linear network training dynamics from random initialization: Data, width, depth, and hyperparameter transfer

    Blake Bordelon and Cengiz Pehlevan. Deep linear network training dynamics from random initialization: Data, width, depth, and hyperparameter transfer. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 4968–4997. PMLR, 2025

    2025

  23. [23]

    Adaptive kernel predictors from feature-learning infinite limits of neural networks

    Clarissa Lauditi, Blake Bordelon, and Cengiz Pehlevan. Adaptive kernel predictors from feature-learning infinite limits of neural networks. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 32617–32648. PMLR, 2025

    2025

  24. [24]

    Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.CoRR, abs/2002.06305, 2020

    work page arXiv 2002

  25. [25]

    Zuidema, and Stella R

    Oskar van der Wal, Pietro Lesci, Max Müller-Eberstein, Naomi Saphra, Hailey Schoelkopf, Willem H. Zuidema, and Stella R. Biderman. PolyPythias: Stability and outliers across fifty language model pre-training runs. InInternational Conference on Learning Representations, 2025

    2025

  26. [26]

    Convergence and divergence of language models under different random seeds

    Finlay Fehlauer, Kyle Mahowald, and Tiago Pimentel. Convergence and divergence of language models under different random seeds. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32982–32991, Suzhou, China,

    2025

  27. [27]

    Association for Computational Linguistics

  28. [28]

    SeedPrints: Fingerprints can even tell which seed your large language model was trained from

    Yao Tong, Haonan Wang, Siquan Li, Kenji Kawaguchi, and Tianyang Hu. SeedPrints: Fingerprints can even tell which seed your large language model was trained from. In International Conference on Learning Representations, 2026. Poster

    2026

  29. [29]

    Transformers are born biased: Structural inductive biases at random initialization and their practical consequences, 2026

    Siquan Li, Yao Tong, Haonan Wang, and Tianyang Hu. Transformers are born biased: Structural inductive biases at random initialization and their practical consequences, 2026

    2026

  30. [30]

    Physics of language models: Part 3.1, knowledge storage and extraction

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 1067–1077. PMLR, 2024. 11

    2024

  31. [31]

    Physics of language models: Part 4.1, architecture design and the magic of canon layers

    Zeyuan Allen-Zhu. Physics of language models: Part 4.1, architecture design and the magic of canon layers. InProceedings of the 39th Conference on Neural Information Processing Systems, NeurIPS ’25, 2025. Full version available athttps://ssrn.com/abstract=5240330

    2025

  32. [33]

    David G. T. Barrett and Benoit Dherin. Implicit gradient regularization. InInternational Conference on Learning Representations, 2021

    2021

  33. [34]

    Smith, Benoit Dherin, David G

    Samuel L. Smith, Benoit Dherin, David G. T. Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. InInternational Conference on Learning Representations, 2021

    2021

  34. [35]

    On the trajectories of sgd without replacement.arXiv preprint arXiv:2312.16143, 2023

    Pierfrancesco Beneventano. On the trajectories of sgd without replacement.arXiv preprint arXiv:2312.16143, 2023

    work page arXiv 2023

  35. [36]

    How neural networks learn the support is an implicit regularization effect of SGD

    Pierfrancesco Beneventano, Andrea Pinto, and Tomaso Poggio. How neural networks learn the support is an implicit regularization effect of SGD. 2024

    2024

  36. [37]

    Griffiths and J

    David F. Griffiths and J. M. Sanz-Serna. On the scope of the method of modified equations. SIAM Journal on Scientific and Statistical Computing, 7(3):994–1008, 1986

    1986

  37. [38]

    Springer, Berlin, Heidelberg, 2 edition, 2006

    Ernst Hairer, Christian Lubich, and Gerhard Wanner.Geometric Numerical Integration: Structure-Preserving Algorithms for Ordinary Differential Equations, volume 31 ofSpringer Series in Computational Mathematics. Springer, Berlin, Heidelberg, 2 edition, 2006

    2006

  38. [39]

    Implicit regularization in heavy- ball momentum accelerated stochastic gradient descent.arXiv preprint arXiv:2302.00849, 2023

    Avrajit Ghosh, He Lyu, Xitong Zhang, and Rongrong Wang. Implicit regularization in heavy- ball momentum accelerated stochastic gradient descent.arXiv preprint arXiv:2302.00849, 2023

    work page arXiv 2023

  39. [40]

    Cattaneo, Jason Matthew Klusowski, and Boris Shigida

    Matias D. Cattaneo, Jason Matthew Klusowski, and Boris Shigida. On the implicit bias of Adam. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 5862–5906. PMLR, 2024

    2024

  40. [41]

    Implicit regularization in deep matrix factorization

    Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. InAdvances in Neural Information Processing Systems, volume 32, 2019

    2019

  41. [42]

    Gradient descent converges linearly to flatter minima than gradient flow in shallow linear networks.arXiv preprint arXiv:2501.09137, 2025

    Pierfrancesco Beneventano and Blake Woodworth. Gradient descent converges linearly to flatter minima than gradient flow in shallow linear networks.arXiv preprint arXiv:2501.09137, 2025

    work page arXiv 2025

  42. [43]

    Chris Mingard, Joar Skalse, Guillermo Valle-Pérez, David Martínez-Rubio, Vladimir Mikulik, and Ard A. Louis. Neural networks are a priori biased towards boolean functions with low entropy, 2020

    2020

  43. [44]

    Vapnik and Alexey Ya

    Vladimir N. Vapnik and Alexey Ya. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities.Theory of Probability and Its Applications, 16(2):264–280, 1971

    1971

  44. [45]

    Vapnik.Statistical Learning Theory

    Vladimir N. Vapnik.Statistical Learning Theory. Wiley, 1998

    1998

  45. [46]

    Bartlett

    Peter L. Bartlett. The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network.IEEE Transactions on Information Theory, 44(2):525–536, 1998

    1998

  46. [47]

    Bartlett and Shahar Mendelson

    Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results.Journal of Machine Learning Research, 3:463–482, 2002

    2002

  47. [48]

    Norm-based capacity control in neural networks

    Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. InProceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pages 1376–1401. PMLR, 2015. 12

    2015

  48. [49]

    Bartlett, Dylan J

    Peter L. Bartlett, Dylan J. Foster, and Matus J. Telgarsky. Spectrally-normalized margin bounds for neural networks. InAdvances in Neural Information Processing Systems, volume 30, 2017

    2017

  49. [50]

    Understand- ing deep learning requires rethinking generalization

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. InInternational Conference on Learning Representations, 2017

    2017

  50. [51]

    Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien

    Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. InProceedings of the 34th International Conference on Machine Learning, volume 70 ofProceedings of Machine Learning ...

    2017

  51. [52]

    Edelman, Fred Zhang, and Boaz Barak

    Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L. Edelman, Fred Zhang, and Boaz Barak. SGD on neural networks learns functions of increasing complexity. InAdvances in Neural Information Processing Systems, volume 32, 2019

    2019

  52. [53]

    Train faster, generalize better: Stability of stochastic gradient descent

    Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. InProceedings of the 33rd International Conference on Machine Learning, volume 48 ofProceedings of Machine Learning Research, pages 1225–1234. PMLR, 2016

    2016

  53. [54]

    Exploring generalization in deep learning

    Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring generalization in deep learning. InAdvances in Neural Information Processing Systems, volume 30, 2017

    2017

  54. [55]

    A PAC- bayesian approach to spectrally-normalized margin bounds for neural networks

    Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. A PAC- bayesian approach to spectrally-normalized margin bounds for neural networks. InInterna- tional Conference on Learning Representations, 2018

    2018

  55. [56]

    Predicting the generalization gap in deep networks with margin distributions

    Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. InInternational Conference on Learning Representations, 2019

    2019

  56. [57]

    Fantas- tic generalization measures and where to find them

    Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantas- tic generalization measures and where to find them. InInternational Conference on Learning Representations, 2020

    2020

  57. [58]

    Flat minima.Neural Computation, 9(1):1–42, 1997

    Sepp Hochreiter and Jürgen Schmidhuber. Flat minima.Neural Computation, 9(1):1–42, 1997

    1997

  58. [59]

    On large-batch training for deep learning: Generalization gap and sharp minima

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. InInternational Conference on Learning Representations, 2017

    2017

  59. [60]

    Sharp minima can generalize for deep nets

    Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. InProceedings of the 34th International Conference on Machine Learning, volume 70 ofProceedings of Machine Learning Research, pages 1019–1028. PMLR, 2017

    2017

  60. [61]

    Sharpness-aware minimization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. InInternational Conference on Learning Representations, 2021

    2021

  61. [62]

    McAllester

    David A. McAllester. PAC-bayesian model averaging. InProceedings of the Twelfth Annual Conference on Computational Learning Theory, pages 164–170, 1999

    1999

  62. [63]

    Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Proceedings of the Thirty-Third Conference on Uncertainty in Artificial Intelligence, 2017

    2017

  63. [64]

    Adams, and Peter Orbanz

    Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P. Adams, and Peter Orbanz. Non- vacuous generalization bounds at the ImageNet scale: A PAC-bayesian compression approach. InInternational Conference on Learning Representations, 2019. 13

    2019

  64. [65]

    Stronger generalization bounds for deep nets via a compression approach

    Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. InInternational Conference on Learning Represen- tations, 2018

    2018

  65. [66]

    Reconciling modern machine- learning practice and the classical bias–variance trade-off.Proceedings of the National Academy of Sciences, 116(32):15849–15854, 2019

    Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias–variance trade-off.Proceedings of the National Academy of Sciences, 116(32):15849–15854, 2019

    2019

  66. [67]

    Deep double descent: Where bigger models and more data hurt

    Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. InInternational Conference on Learning Representations, 2020

    2020

  67. [68]

    Bartlett, Philip M

    Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression.Proceedings of the National Academy of Sciences, 117(48):30063–30070, 2020

    2020

  68. [69]

    Neal.Bayesian Learning for Neural Networks, volume 118 ofLecture Notes in Statistics

    Radford M. Neal.Bayesian Learning for Neural Networks, volume 118 ofLecture Notes in Statistics. Springer, 1996

    1996

  69. [70]

    Christopher K. I. Williams. Computing with infinite networks. InAdvances in Neural Information Processing Systems, volume 9, 1996

    1996

  70. [71]

    Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein

    Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. InInternational Conference on Learning Representations, 2018

    2018

  71. [72]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. InAdvances in Neural Information Processing Systems, volume 31, 2018

    2018

  72. [73]

    Camargo, and Ard A

    Kamaludin Dingle, Chico Q. Camargo, and Ard A. Louis. Input–output maps are strongly biased towards simple outputs.Nature Communications, 9:761, 2018

    2018

  73. [74]

    Random deep neural networks are biased towards simple functions

    Giacomo De Palma, Bobak Kiani, and Seth Lloyd. Random deep neural networks are biased towards simple functions. InAdvances in Neural Information Processing Systems, volume 32, 2019

    2019

  74. [75]

    On the complexity of finite sequences.IEEE Transactions on Information Theory, 22(1):75–81, 1976

    Abraham Lempel and Jacob Ziv. On the complexity of finite sequences.IEEE Transactions on Information Theory, 22(1):75–81, 1976

    1976

  75. [76]

    A universal algorithm for sequential data compression.IEEE Transactions on Information Theory, 23(3):337–343, 1977

    Jacob Ziv and Abraham Lempel. A universal algorithm for sequential data compression.IEEE Transactions on Information Theory, 23(3):337–343, 1977

    1977

  76. [77]

    Ming Li and Paul M. B. Vitányi.An Introduction to Kolmogorov Complexity and Its Applica- tions. Springer, 3 edition, 2008

    2008

  77. [78]

    Simplicity bias in transformers and their ability to learn sparse Boolean functions

    Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom. Simplicity bias in transformers and their ability to learn sparse Boolean functions. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5767–5791, Toronto, Canada, 2023. Association for Computational Linguistics

    2023

  78. [79]

    Michael Hahn and Mark Rofin. Why are sensitive functions hard for transformers? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14973–15008, Bangkok, Thailand, 2024. Association for Computational Linguistics

    2024

  79. [80]

    Transformers learn low sensitivity functions: Investigations and implications

    Bhavya Vasudeva, Deqing Fu, Tianyi Zhou, Elliott Kau, Youqi Huang, and Vatsal Sharan. Transformers learn low sensitivity functions: Investigations and implications. InInternational Conference on Learning Representations, 2025

    2025

  80. [81]

    Hamprecht, Yoshua Bengio, and Aaron Courville

    Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. InProceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 5301–5310. PMLR, 2019. 14

    2019

Showing first 80 references.