Pith. sign in

REVIEW 2 major objections 4 minor 2 cited by

Optimizers Qualitatively Alter Solutions And We Should Leverage This

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The choice of optimizer in deep learning determines qualitative properties of the learned solution—sparsity, representation structure, and resistance to forgetting—not just convergence speed.

desk verdict A well-framed position paper whose most concrete proposal—the preconditioner equivalence—is asserted rather than proven or tested. read the letter →

arxiv 2507.12224 v1 pith:BLQBDXAF submitted 2025-07-16 cs.LG

Razvan Pascanu , Clare Lyle , Ionut-Vlad Modoranu , Naima Elosegui Borras , Dan Alistarh , Petar Velickovic , Sarath Chandar , Soham De
show 1 more author
James Martens
This is my paper
classification cs.LG MSC 68T07
keywords optimizerinductivebiaseffectiveexpressivitypreconditioningsparsity-inducingoptimizationcatastrophicforgettingsecond-orderoptimizerscontinuallearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deep neural network losses are non-convex, so a local, gradient-based optimizer cannot be guaranteed to reach a global minimum; which minimum it reaches depends on the path it takes. This paper argues that the optimizer therefore does more than set the speed of training: it selects qualitative features of the final solution, including the sparsity of weights, the effective dimensionality of representations, and how much interference occurs between tasks learned sequentially. The authors contend that optimizer design should be treated as an explicit mechanism for encoding inductive biases, comparable in importance to architecture and data, and that arguments about what a model class can express are incomplete without considering what its learning rule can reach. Two illustrative cases are developed: non-diagonal preconditioning (Shampoo) produces lower-rank representations and less catastrophic forgetting than Adam or SGD, and a sparsity-inducing reparameterization from the literature is reframed as a simple preconditioner.

What carries the argument

The load-bearing object is the preconditioner $P$ in the update rule, a matrix that rescales and rotates the gradient before it is applied. The paper's argument is that this matrix is not merely an acceleration device: by changing which directions are favoured at each step, it changes the basin of attraction the training process enters and therefore the qualitative character of the solution. Two specific mechanisms carry the examples. A non-diagonal preconditioner contains off-diagonal entries that encode how moving one weight affects the gradient of another, which the paper argues removes wasteful over-shooting and keeps learned representations in a lower-dimensional subspace. For sparsity, the mechanism is a saddle at zero: the reparameterization $\theta = \phi|\phi|^{\alpha-1}$ multiplies gradients by $|\phi|^{\alpha-1}$, making small parameters expensive to move, and the paper claims the same step-by-step dynamics are produced by the preconditioner $P = \operatorname{diag}(|\theta|^\beta)$ on the original parameters, with $\alpha$ folded into the learning rate.

What would settle it

Run a matched head-to-head: from the same initialization, train the same architecture on the same data with (a) the Power-propagation reparameterization and (b) plain SGD with preconditioner $P = \operatorname{diag}(|\theta|^\beta)$, with the learning-rate scaling specified in the paper. Record per-step parameter updates and the final weight-sparsity distribution; if the trajectories or the sparsity of the solutions differ materially, the claimed equivalence that underpins the sparsity example is false. For the continual-learning claim, a falsifying observation would be a non-diagonal preconditioner that consistently increases, rather than decreases, cross-task interference in a benchmark where Shampoo's lower-forgetting effect is currently the evidence.

Watch

Extended reading notes

Core claim

The paper's central claim is that the learning algorithm, and specifically its preconditioner, is a genuine source of inductive bias: it determines not only how fast a model converges but what kind of solution it converges to, and hence the effective expressivity of the model class. In the non-convex loss landscapes of deep networks, different optimizers follow different paths from the same initialization and can land in minima with different qualitative properties. The paper advances this thesis through two concrete illustrations. First, a non-diagonal preconditioner such as Shampoo is argued to produce more localized, lower-rank representations and less interference across sequentially learned tasks than diagonal methods like Adam, with supporting experiments on permuted and class-incremental MNIST. Second, the Power-propagation reparameterization for sparsity is recast as an equivalent preconditioner $P = \operatorname{diag}(|\theta|^\beta)$, showing that a solution property like sparsity can be engineered directly through optimizer choice, even at the cost of convergence speed.

Load-bearing premise

The concrete sparsity proposal depends on the claim that a change to the optimizer's step sizes can exactly reproduce the learning dynamics of a particular change to how the network's parameters enter the computation, and this exact equivalence is asserted without a proof or a direct experiment.

Editorial extensions

If this is right

  • Expressivity analyses that consider only architecture and data are incomplete; the learning algorithm determines which functions are reachable from initialization, and this should be part of model-selection reasoning.
  • Non-diagonal second-order preconditioners become a candidate tool for continual learning, since they are argued to reduce interference and forgetting while also compressing the effective dimensionality of representations.
  • Optimizers can be deliberately built to sacrifice convergence speed in exchange for solution properties such as sparsity, exactly as a preconditioner can bias learning toward sparse solutions.
  • For large pretrained models whose architecture is fixed, fine-tuning with a carefully chosen optimizer offers a practical route to injecting an inductive bias that architecture changes cannot provide.
  • Comparing two optimizers solely by final accuracy is misleading: they may achieve similar error while producing solutions with different sparsity, representation rank, and robustness to task interference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's central equivalence is to train identical models with Power-propagation reparameterization and with preconditioned SGD using $P = \operatorname{diag}(|\theta|^\beta)$, tracking both the parameter trajectories and the sparsity of the final weights; if the dynamics diverge, the concrete sparsity proposal needs a weaker justification.
  • If optimizer choice changes effective expressivity, negative results about what a model class can learn—such as debates over whether recurrent networks are Turing complete under gradient training—should be re-examined through the lens of reachability rather than pure representability.
  • The paper's position implies that benchmark comparisons of optimizers should report solution-level diagnostics (effective rank, sparsity, cross-task interference) alongside accuracy and wall-clock time, because optimizers that tie on accuracy can still differ on these properties.
  • A stronger but untested extension of the argument is that optimizer-induced biases and architectural inductive biases are partially interchangeable, so architecture search and optimizer search could be traded against each other in the design of a learning system.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This position paper argues that under non-convex objectives the optimization algorithm is not merely a speed-of-convergence device: it selects among minima with qualitatively different properties, acting as an inductive bias and shaping the effective expressivity of the model class. The authors support this with existing literature (flat minima, implicit regularization, rich/lazy regimes) and with two examples. The first (Section 3.1) compares Shampoo and Adam/AdamW on permuted and class-incremental MNIST with small MLPs, reporting less catastrophic forgetting and lower effective rank for Shampoo. The second (Section 3.2) reframes Power-propagation as a sparsity-inducing preconditioner P=diag(|θ|^β), claiming identical learning dynamics to training the reparameterized model φ^α. Section 4 discusses limitations and counterarguments (reparameterization duality, hand-engineering of biases). The paper's recommendation is to design optimizers with explicit solution properties in mind rather than optimizing convergence speed only.

Significance. If the thesis holds, it elevates optimizer design to the same status as architecture selection and gives a concrete mechanism for inducing sparsity, low-rank representations, or low forgetting without changing architecture. The paper is a useful and honest synthesis: it cites a broad literature, openly labels the reparameterization–optimizer relationship as informal, and frames the examples as thought experiments rather than methodological contributions. Its main present weakness is that the load-bearing sparsity proposal rests on an unproved equivalence and an untested preconditioner; as written, the only concrete evidence for that mechanism comes from prior work on Power-propagation. The forgetting experiments are small-scale and lack uncertainty information. Nevertheless, the central conceptual claim is well-supported by prior literature and by the authors' experiments in a directional way, and the paper's call to study optimizer-induced biases is timely.

major comments (2)
  1. [Section 3.2, paragraph after the gradient equation] The assertion that optimizing θ=φ^α with Power-propagation gives 'identical learning dynamics' to preconditioned SGD with P=diag(|θ|^β) is not derived and is only true in an infinitesimal-step sense. For the reparameterized model, a finite step in φ yields θ_{t+1}=(φ_t−η φ_t^{α−1}g_θ)^α, while the preconditioned update on θ is θ_t−η|θ_t|^βg_θ; these agree only to first order in η (and only if β=2(α−1)/α, a relation the paper never states). The difference is O(η²) and is non-negligible near the very zero saddles that are supposed to drive sparsity. In addition, the proposed preconditioner is never implemented or tested. This is load-bearing because Section 3.2 is the paper's only concrete proposal for a new sparsity-inducing optimizer. Please either provide a rigorous derivation (including the β–α relation and a statement of the discrete regime in which the equivalence holds) or add a direct empirical comparison of P=diag(|θ|^β) against Power-propagation.
  2. [Section 3.1, Figures 3 and 4] The empirical support for the claim that non-diagonal preconditioners reduce forgetting and yield lower-rank representations is currently too thin to carry the example. The experiments use one architecture family (small MLPs) and one dataset family (MNIST variants); Figure 3 (right) shows a single covariance spectrum with no error bars, and Figure 4 does not report the number of seeds or variance. The Adam-vs-Shampoo comparison also changes the preconditioning scheme and the adaptive step-size dynamics simultaneously, so the observed difference in forgetting could in principle be due to different effective learning-rate schedules rather than to the non-diagonal structure of the preconditioner. Please add seed counts and error bars, and ideally a diagonal-preconditioner control matched in convergence speed, before presenting this as an illustration of the thesis.
minor comments (4)
  1. [Section 3.2] The notation switches between the simplified φ^α and the actual absolute-value reparameterization θ=φ|φ|^{α−1}; please state explicitly that the 'identical dynamics' claim is made for the simplified form and discuss whether the same calculation carries over for the absolute-value version.
  2. [Figure 3] The right panel lacks axis labels and a legend, and 'effective rank' is mentioned in the text but never defined; please define the quantity and show the truncation threshold used.
  3. [Abstract and Introduction] The paper would benefit from a short list of solution properties it considers 'qualitative' (e.g., sparsity, effective rank, forgetting, flatness), since the current examples imply but do not delimit the set.
  4. [Throughout] There are minor copyediting issues: 'Politechnique Montreal' should be 'Polytechnique Montréal', 'V elickovic' in the author list has a stray space, the denominator in the Section 3.2 gradient expression should be ∂(φ^α) rather than ∂ϕ^α, and the double 'the' in Section 4 should be corrected.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central position is supported by independent literature and new experiments, and the Power-propagation recasting is transparently a reformulation rather than a self-derived prediction.

full rationale

The paper is a position/opinion piece. Its central thesis, that optimizers encode inductive biases and affect the qualitative properties of solutions, is supported by a broad literature review (flat minima, implicit regularization, SAM, NTK/rich regimes) and by new experiments in this paper: the Shampoo-vs-Adam comparisons on permuted and class-incremental MNIST are performed here and show differences in representation rank and forgetting. These experiments are not derived from the thesis, so the main claim is not circular. The sparsity example in Section 3.2 explicitly recasts Power-propagation, a published method with overlapping authorship, as a preconditioner. This is a self-citation, but the cited work contains its own published experiments (weight-magnitude distributions and sparsification benchmarks) that are externally checkable and not fitted to the present paper's conclusions. The paper also transparently labels the example as a 'thought experiment' and footnotes that the reparameterization/optimizer 'duality' is informal and requires considerably more work, so the asserted equivalence is not hidden. The key gap is that the claimed equivalence between Power-propagation and preconditioned SGD with P=diag(|θ|^β) is stated without a full derivation and the proposed preconditioner is never tested; folding 'α' into the learning rate is at best a first-order approximation. That is an unverified correctness/empirical gap, not a circular reduction: the preconditioner is not fitted to data, no equation is defined in terms of the target conclusion, and no fitted parameter is renamed as a prediction. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The paper's own limitations section acknowledges the main counterarguments, further weakening any appearance of a closed self-supporting loop.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper's broad thesis is a conceptual claim resting on standard non-convex optimization assumptions; its two concrete examples rest on one asserted equivalence and an informal manifold/subspace narrative. No new physical or mathematical entities are introduced.

free parameters (1)
  • beta = None specified (beta > 0)
    Introduced in Section 3.2 as the exponent of the proposed preconditioner P = diag(|theta|^beta). No value or tuning recipe is given in this paper; it is a design knob inherited from the Power-propagation idea.
assumptions (4)
  • domain assumption Non-convex loss landscapes contain multiple minima with qualitatively different properties, and the optimizer's trajectory selects among them.
    This is the baseline for the whole paper, illustrated in Figure 1. It is standard in deep learning but not proven for the specific cases discussed.
  • domain assumption Wasteful over-shooting movement in parameter space perturbs representations in a way that expands their dimensionality, and lower-dimensional representations reduce interference and forgetting.
    Section 3.1, Figures 2-4. This is an informal narrative connecting optimizer geometry to representation geometry; no formal argument is supplied.
  • ad hoc to paper Step-wise equivalence between the phi^alpha reparameterization and preconditioner P = diag(|theta|^beta), with alpha folded into the learning rate.
    Section 3.2, footnote 3. The equivalence is asserted rather than derived, and it is load-bearing for the paper's sparsity example.
  • domain assumption The optimizer limits the set of reachable functions and therefore defines the effective expressivity of a model class.
    Section 4 and Figure 6. This frames the paper's argument that expressivity arguments ignoring optimization are incomplete.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimizers Qualitatively Alter Solutions And We Should Leverage This." pith.science (2026). https://pith.science/paper/BLQBDXAF

@misc{pith2026250712224,
  author       = {Pith},
  title        = {Pith review of: Optimizers Qualitatively Alter Solutions And We Should Leverage This},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BLQBDXAF}},
  note         = {Machine review of arXiv:2507.12224}
}
read the original abstract

Due to the nonlinear nature of Deep Neural Networks (DNNs), one can not guarantee convergence to a unique global minimum of the loss when using optimizers relying only on local information, such as SGD. Indeed, this was a primary source of skepticism regarding the feasibility of DNNs in the early days of the field. The past decades of progress in deep learning have revealed this skepticism to be misplaced, and a large body of empirical evidence shows that sufficiently large DNNs following standard training protocols exhibit well-behaved optimization dynamics that converge to performant solutions. This success has biased the community to use convex optimization as a mental model for learning, leading to a focus on training efficiency, either in terms of required iteration, FLOPs or wall-clock time, when improving optimizers. We argue that, while this perspective has proven extremely fruitful, another perspective specific to DNNs has received considerably less attention: the optimizer not only influences the rate of convergence, but also the qualitative properties of the learned solutions. Restated, the optimizer can and will encode inductive biases and change the effective expressivity of a given class of models. Furthermore, we believe the optimizer can be an effective way of encoding desiderata in the learning process. We contend that the community should aim at understanding the biases of already existing methods, as well as aim to build new optimizers with the explicit intent of inducing certain properties of the solution, rather than solely judging them based on their convergence rates. We hope our arguments will inspire research to improve our understanding of how the learning process can impact the type of solution we converge to, and lead to a greater recognition of optimizers design as a critical lever that complements the roles of architecture and data in shaping model outcomes.

Figures

Figures reproduced from arXiv: 2507.12224 by the authors.

Figure 1
Figure 1. Compared to the convex case, where optimizers can converge to a global minima, in the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Diagram depicting the intuition of why second order method lead to more localized [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Left: catastrophic forgetting in a 2-layer MLP trained on class-incremental MNIST, where the network trains on each pair of classes sequentially. All networks exhibit worse performance on earlier class pairs, but the decline in performance is much sharper for Adam than for Shampoo. This effect is not mitigated by reducing the learning rate on Adam. Right: visualization of the alignment between features of different … view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: Left: Test accuracy averaged over 3 variants of permuted MNIST learned sequentially with different optimizers (for different sizes of MLPs); Right: Spectrum of the covariance of the representa￾tion for the 100 units MLP. Note how the number of significant eigen￾values …
Figure 5
Figure 5. Figure 5: Saddle point created by using ϕ α, for α = 3. Note that we chose an odd power to use the simplified for￾mula ϕ α which is more intuitive to reason about. The shape of the loss with respect to a specific parameter ϕi around 0 will roughly look like the depiction in [PI…
Figure 6
Figure 6. Figure 6: Diagram depicting our main position. While the choice of architecture and model size [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Learned replacement non-linearities show transformers are rarely optimal for algorithmic tasks, with benefits that are task-specific, while language/code gains are smaller and more transferable.

  2. Cross-Model Semantics in Representation Learning

    cs.LG 2025-08 reject novelty 3.0 of 10

    The paper restates existing alignment metrics and claims, with no numerical evidence, that structured architectures show more stable cross-model representation geometry.

Reference graph

Works this paper leans on

81 extracted references · 51 canonical work pages · cited by 2 Pith papers

  1. [1]

    Advani and Andrew M

    Madhu S. Advani and Andrew M. Saxe. High-dimensional dynamics of generalization error in neural networks, 2017. URL https://arxiv.org/abs/1710.03667

    arXiv 2017

  2. [2]

    Selfless Sequential Learning

    Rahaf Aljundi, Marcus Rohrbach, and Tinne Tuytelaars. Selfless sequential learning, 2019. URL https://arxiv.org/abs/1806.05421

    work page Pith review arXiv 2019

  3. [3]

    Learning and generalization in overparame- terized neural networks, going beyond two layers

    Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparame- terized neural networks, going beyond two layers. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.ne...

    2019

  4. [4]

    Natural gradient works efficiently in learning

    Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural Comput., 10(2): 251–276, February 1998. ISSN 0899-7667. doi: 10.1162/089976698300017746. URL https://doi.org/10.1162/089976698300017746

    doi:10.1162/089976698300017746 1998

  5. [5]

    When does preconditioning help or hurt generalization? 2020

    Shun-ichi Amari, Jimmy Ba, Roger Grosse, Xuechen Li, Atsushi Nitanda, Taiji Suzuki, Denny Wu, and Ji Xu. When does preconditioning help or hurt generalization? 2020

    work page 2020

  6. [6]

    Implicit regularization in deep matrix factorization, 2019

    Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization, 2019. URL https://arxiv.org/abs/1905.13655

    arXiv 2019

  7. [7]

    David G. T. Barrett and Benoit Dherin. Implicit gradient regularization, 2022. URL https: //arxiv.org/abs/2009.11162

    arXiv 2022

  8. [8]

    Reconciling modern machine- learning practice and the classical bias–variance trade-off.Proceedings of the National Academy of Sciences, 116(32):15849–15854, July 2019

    Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias–variance trade-off.Proceedings of the National Academy of Sciences, 116(32):15849–15854, July 2019. ISSN 1091-6490. doi: 10.1073/pnas.1903070116. URL http://dx.doi.org/10.1073/pnas.1903070116. 10

    doi:10.1073/pnas.1903070116 2019

Show all 81 references
  1. [9]

    Learning deep architectures for ai

    Yoshua Bengio. Learning deep architectures for ai. Foundations and Trends in Machine Learn- ing, 2(1):1–127, 2009. URL http://dblp.uni-trier.de/db/journals/ftml/ftml2. html#Bengio09

    2009

  2. [10]

    Scaling Learning Algorithms towards AI

    Yoshua Bengio and Yann Lecun. Scaling Learning Algorithms towards AI. MIT Press, 2007

    2007

  3. [11]

    Greedy layer-wise training of deep networks

    Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Proceedings of the 20th International Conference on Neural Information Processing Systems, NIPS’06, page 153–160, Cambridge, MA, USA, 2006. MIT Press

    2006

  4. [12]

    Old optimizer, new norm: An anthology, 2024

    Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology, 2024. URL https://arxiv.org/abs/2409.20325

    2024 arXiv

  5. [13]

    The tradeoffs of large scale learning

    Léon Bottou and Olivier Bousquet. The tradeoffs of large scale learning. In J. Platt, D. Koller, Y . Singer, and S. Roweis, editors,Advances in Neural Information Processing Systems, vol- ume 20. Curran Associates, Inc., 2007. URL https://proceedings.neurips.cc/paper_ files/pa...

    2007

  6. [14]

    Large scale online learning

    Léon Bottou and Yann Cun. Large scale online learning. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems , volume 16. MIT Press, 2003. URL https://proceedings.neurips.cc/paper_files/paper/2003/ file/9fb7b048c96d44a0337f049e0a61f...

    2003

  7. [15]

    Chayes, Levent Sagun, and Riccardo Zecchina

    Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer T. Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. CoRR, abs/1611.01838, 2016. URL http://arxiv.org/abs/ 1611.01838

    2016 arXiv

  8. [16]

    On lazy training in differentiable program- ming

    Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable program- ming. In Advances in Neural Information Processing Systems, volume 32, pages 2937–2947, 2019

    2019

  9. [17]

    Open problem: The landscape of the loss surfaces of multilayer networks

    Anna Choromanska, Yann LeCun, and Gérard Ben Arous. Open problem: The landscape of the loss surfaces of multilayer networks. In Peter Grünwald, Elad Hazan, and Satyen Kale, editors, Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Lear...

    2015

  10. [18]

    Turing completeness of bounded-precision recurrent neural networks

    Stephen Chung and Hava Siegelmann. Turing completeness of bounded-precision recurrent neural networks. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 28431–28441. Cur...

    2021

  11. [19]

    Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio

    Yann N. Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non- convex optimization. InProceedings of the 28th International Conference on Neural Information Processi...

    2014

  12. [20]

    A continual learning survey: Defying forgetting in classification tasks

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7): 3366–33...

    2022

  13. [21]

    Sharp minima can generalize for deep nets, 2017

    Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets, 2017. URL https://arxiv.org/abs/1703.04933

    2017 arXiv

  14. [22]

    A theoretical analysis of catastrophic forgetting through the ntk overlap matrix

    Thang Doan, Mehdi Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS), volume

  15. [23]

    Continual backprop: Stochastic gradient descent with persistent randomness

    Shibhansh Dohare, A Rupam Mahmood, and Richard S Sutton. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325, 2021

    2021 arXiv

  16. [24]

    Asynchronous algorithmic alignment with cocycles, 2024

    Andrew Dudzik, Tamara von Glehn, Razvan Pascanu, and Petar Veliˇckovi´c. Asynchronous algorithmic alignment with cocycles, 2024. URL https://arxiv.org/abs/2306.15632

    2024 arXiv

  17. [25]

    Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 ...

    2010

  18. [26]

    Model-agnostic meta-learning for fast adap- tation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017

    2017

  19. [27]

    Sharpness-aware min- imization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=6Tm1mposlrM

    2021

  20. [28]

    Stanislav Fort, Andrew Brock, Razvan Pascanu, Soham De, and Samuel L. Smith. Drawing multiple augmentation samples per image during training efficiently decreases test error, 2022. URL https://arxiv.org/abs/2105.13343

    2022 arXiv

  21. [29]

    qualitatively characterizing neural network optimization prob- lems

    Jonathan Frankle. Revisiting "qualitatively characterizing neural network optimization prob- lems", 2020. URL https://arxiv.org/abs/2012.06898

    2020 arXiv

  22. [30]

    Catastrophic forgetting in connectionist networks

    Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135, 1999

    1999

  23. [31]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors,Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings ...

  24. [32]

    Goodfellow, Oriol Vinyals, and Andrew Saxe

    Ian J. Goodfellow, Oriol Vinyals, and Andrew Saxe. Qualitatively characterizing neural network optimization problems. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Tr...

    2015 arXiv

  25. [33]

    Griffiths

    Thomas L. Griffiths. Understanding human intelligence through human limitations, 2020. URL https://arxiv.org/abs/2009.14050

    2020 arXiv

  26. [34]

    Shampoo: Preconditioned stochastic tensor optimization, 2018

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization, 2018. URL https://arxiv.org/abs/1802.09568

    2018 arXiv

  27. [35]

    Embracing change: Con- tinual learning in deep neural networks

    Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Con- tinual learning in deep neural networks. Trends in Cognitive Sciences , 24(12):1028–1040, 2020

    2020

  28. [36]

    Hinton, Simon Osindero, and Yee-Whye Teh

    Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural Comput., 18(7):1527–1554, July 2006. ISSN 0899-7667. doi: 10.1162/neco.2006.18.7.1527. URL https://doi.org/10.1162/neco.2006.18.7.1527

    2006 doi

  29. [37]

    Flat minima

    Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Comput., 9(1):1–42, January

  30. [38]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems , volume 31, pages 8571–8580, 2018. 12

    2018

  31. [39]

    On large-batch training for deep learning: Generalization gap and sharp minima

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017. URL https://openreview. net/fo...

    2017

  32. [40]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Scien...

    2017

  33. [41]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25 , pages 1097–1105. Curran As...

    2012

  34. [42]

    Continual learning as computationally constrained reinforcement learning,

    Saurabh Kumar, Henrik Marklund, Ashish Rao, Yifan Zhu, Hong Jun Jeon, Yueyang Liu, and Benjamin Van Roy. Continual learning as computationally constrained reinforcement learning,

  35. [43]

    Maintaining plasticity in continual learning via regenerative regularization

    Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learning via regenerative regularization. 2024

    2024

  36. [44]

    Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks

    Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume...

    2021

  37. [45]

    Directions of curvature as an explanation for loss of plasticity

    Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C Machado. Directions of curvature as an explanation for loss of plasticity. arXiv preprint arXiv:2312.00246, 2023

    2023 arXiv

  38. [46]

    Visualizing the loss landscape of neural nets

    Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018

    2018

  39. [47]

    Lifelong machine learning: a paradigm for continuous learning

    Bing Liu. Lifelong machine learning: a paradigm for continuous learning. Front. Comput. Sci., 11(3):359–361, June 2017. ISSN 2095-2228. doi: 10.1007/s11704-016-6903-6. URL https://doi.org/10.1007/s11704-016-6903-6

    2017 doi

  40. [48]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  41. [49]

    Understanding and preventing capacity loss in reinforcement learning

    Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2021

    2021

  42. [50]

    Normalization and effective learning rates in reinforcement learning, 2024

    Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado van Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning, 2024. URL https://arxiv.org/abs/2407.01800

    2024 arXiv

  43. [51]

    Deep learning via hessian-free optimization

    James Martens. Deep learning via hessian-free optimization. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, page 735–742, Madison, WI, USA, 2010. Omnipress. ISBN 9781605589077

    2010

  44. [52]

    Optimizing neural networks with kronecker-factored approximate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Ma- chine Learning Research, pa...

    2015

  45. [53]

    A logical calculus of ideas immanent in nervous activity

    Warren McCulloch and Walter Pitts. A logical calculus of ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5:127–147, 1943. 13

    1943

  46. [54]

    Minsky and S

    M. Minsky and S. Papert. Perceptrons. MIT Press, Cambridge, MA, 1969

    1969

  47. [55]

    Deep double descent: Where bigger models and more data hurt

    Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. CoRR, abs/1912.02292, 2019. URL http://arxiv.org/abs/1912.02292

    1912 arXiv

  48. [56]

    Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens

    Arvind Neelakantan, Luke Vilnis, Quoc V . Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks. ArXiv, abs/1511.06807, 2015. URL https://api.semanticscholar.org/CorpusID:826188

    2015 arXiv

  49. [57]

    Nerem, Samantha Chen, Sanjoy Dasgupta, and Yusu Wang

    Robert R. Nerem, Samantha Chen, Sanjoy Dasgupta, and Yusu Wang. Graph neural networks extrapolate out-of-distribution for shortest paths, 2025. URLhttps://arxiv.org/abs/2503. 19173

    2025

  50. [58]

    The role of over-parametrization in generalization of neural networks

    Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-parametrization in generalization of neural networks. InInternational Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=BygfghAcYX

    2019

  51. [59]

    The primacy bias in deep reinforcement learning

    Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, pages 16828–16847. PMLR, 2022

    2022

  52. [60]

    Jorge Nocedal and Stephen J. Wright. Numerical optimization. Springer Series in Operations Research and Financial Engineering, pages 1–664, 2006. ISSN 1431-8598

    2006

  53. [61]

    Parisi, Ronald Kemker, Jose L

    German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54 – 71, 2019. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2019.01.012. URL http: //www.sciencedirec...

    2019 doi

  54. [62]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on International Con- ference on Machine Learning - Volume 28 , ICML’13, page III–1310–III–1318. JMLR.org, 2013

    2013

  55. [63]

    Attention is turing-complete

    Jorge Perez, Pablo Barcelo, and Javier Marinkovic. Attention is turing-complete. Journal of Machine Learning Research, 22(75):1–35, 2021. URL http://jmlr.org/papers/v22/ 20-302.html

    2021

  56. [64]

    Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.Advances in Neural Information Processing Systems, 36:71095–71134, 2023

    Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.Advances in Neural Information Processing System...

    2023

  57. [65]

    Sparse feature learning for deep belief networks

    Marc’ Aurelio Ranzato, Y-Lan Boureau, and Yann LeCun. Sparse feature learning for deep belief networks. In Proceedings of the 21st International Conference on Neural Information Processing Systems, NIPS’07, page 1185–1192, Red Hook, NY , USA, 2007. Curran Associates Inc. ISBN ...

    2007

  58. [66]

    Catastrophic forgetting, rehearsal and pseudorehearsal

    Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7 (2):123–146, 1995

    1995

  59. [67]

    Learning representations by back-propagating errors

    David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6088):533–536, 1986

    1986

  60. [68]

    Jayakumar, Razvan Pascanu, Peter E

    Jonathan Schwarz, Siddhant M. Jayakumar, Razvan Pascanu, Peter E. Latham, and Yee Whye Teh. Powerpropagation: A sparsity inducing weight reparameterisation, 2021. URL https: //arxiv.org/abs/2110.00296

    2021 arXiv

  61. [69]

    Siegelmann and Eduardo D

    Hava T. Siegelmann and Eduardo D. Sontag. On the computational power of neural nets. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory , COLT ’92, page 440–449, New York, NY , USA, 1992. Association for Computing Machinery. ISBN 089791497X. doi: 10.1...

    1992

  62. [70]

    Mas- tering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016

    2016

  63. [71]

    Woodfisher: Efficient second-order approximation for neural network compression, 2020

    Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression, 2020. URL https://arxiv.org/abs/2004.14340

    2020 arXiv

  64. [72]

    Smith, Benoit Dherin, David G

    Samuel L. Smith, Benoit Dherin, David G. T. Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent, 2021. URL https://arxiv.org/abs/2101. 12176

    2021

  65. [73]

    The dormant neuron phenomenon in deep reinforcement learning

    Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, pages 32145–32168. PMLR, 2023

    2023

  66. [74]

    Mahdi Soltanolkotabi, Adel Javanmard, and Jason D. Lee. Theoretical insights into the opti- mization landscape of over-parameterized shallow neural networks. IEEE Trans. Inf. Theor., 65(2):742–769, February 2019. ISSN 0018-9448. doi: 10.1109/TIT.2018.2854560. URL https://doi.o...

    2019

  67. [75]

    Practical issues in temporal difference learning

    Gerald Tesauro. Practical issues in temporal difference learning. Mach. Learn., 8(3–4):257–277, May 1992. ISSN 0885-6125. doi: 10.1007/BF00992697. URL https://doi.org/10.1007/ BF00992697

    1992 doi

  68. [76]

    softmax is not enough (for sharp out-of-distribution), 2024

    Petar Veliˇckovi´c, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. softmax is not enough (for sharp out-of-distribution), 2024. URL https://arxiv.org/abs/2410. 01104

    2024

  69. [77]

    Tiffany J Vlaar and Jonathan Frankle. What can linear interpolation of neural network loss landscapes tell us? In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine L...

    2022

  70. [78]

    Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro

    Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learni...

    2020

  71. [79]

    Understanding deep learning requires rethinking generalization

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Sy8gdB9xx. 15

    2017

  72. [1997]

    doi: 10.1162/neco.1997.9.1.1

    ISSN 0899-7667. doi: 10.1162/neco.1997.9.1.1. URL https://doi.org/10.1162/ neco.1997.9.1.1

    1997 doi

  73. [2023]

    URL https://arxiv.org/abs/2307.04345

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.