Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Fisher-based optimizer that approximates curvature with a diagonal block-Kronecker product claims to outperform Adam, K-FAC, and Shampoo at the same wall-clock training cost.

desk verdict The new diagonal block-Kronecker Fisher optimizer is plausible, but the central SOTA claim rests on a weak ImageNet baseline and an uneven wall-clock protocol; the empirical core warrants revision, not the headline. read the letter →

arxiv 2504.20096 v1 pith:6SHJGYHI submitted 2025-04-26 cs.LG math.OC

Damien Martins Gomes This is my paper · ORCID
classification cs.LGmath.OC MSC 68T0790C2665K05
keywords Fisherinformationmatrixsecond-orderoptimizationKronecker-factoredapproximationnaturalgradientdescentAdamadaptiveoptimizersdeeplearningtrainingconvergenceandgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AdaFisher says second-order optimization for deep networks can be made practical by approximating the Fisher information matrix with a diagonal block-Kronecker product. The optimizer replaces the squared-gradient second moment of Adam with this Fisher-based preconditioner, so each update step rescales the gradient by local curvature information at a computational cost close to first-order methods. On image classification (CIFAR-10/100, Tiny ImageNet, ImageNet-1k) and language modeling (WikiText-2, PTB), it reports higher accuracy or lower perplexity than Adam, AdaHessian, K-FAC, and Shampoo under equal wall-clock time. If the claims hold, curvature-aware training no longer requires the expensive per-iteration overhead that has kept second-order methods out of large-scale practice.

What carries the argument

The load-bearing object is the diagonal block-Kronecker approximation of the empirical Fisher information matrix, denoted $\tilde F_D$ in the paper. For each layer $i$, the Fisher block is factored as $H_{i-1}\otimes S_i$ in the K-FAC style, then the factors are replaced by their diagonals, min-max normalized, and combined as $\tilde F^i_D = H'_{D,i-1}\otimes S'_{D,i}+\lambda I$. The diagonal dominance of the Kronecker factors is what licenses this truncation; the EMA over batches and the inclusion of BatchNorm and LayerNorm factors in Proposition 3.3.1 are what make the approximation stable enough to use as a preconditioner throughout training.

What would settle it

Measure the spectral norm or energy fraction of the off-diagonal part of $H$ and $S$ across layers for a transformer or a wide CNN; if some layer shows off-diagonal energy comparable to the diagonal and a version of AdaFisher that restores off-diagonal blocks in that layer visibly outperforms the fully diagonal version, the diagonal-dominance premise fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the Fisher information matrix of a deep network does not need to be formed or inverted in full: its layer-wise Kronecker factors are empirically diagonal-dominant, so the essential curvature information can be captured by taking only the diagonal of each factor. AdaFisher builds on K-FAC's factorization $\hat{F}_i = H_{i-1} \otimes S_i$, where $H_{i-1}$ is the activation covariance and $S_i$ the sensitivity covariance, then keeps $\tilde{F}_D^i = H'_{D,i-1} \otimes S'_{D,i} + \lambda I$ with min-max normalized diagonal factors and Tikhonov damping. This preconditioner replaces the second moment $v^{(t)}$ in the Adam-style update $\theta^{(t+1)}=\theta^{(t)}-\alpha(\tilde F_D^{(t)})^{-1} m^{(t)}$, and an exponential moving average over batches keeps the curvature estimate stable. The paper reports that this yields faster convergence and better final accuracy than first-order and other second-order baselines, and that it scales to ImageNet-1k and transformer language models at near-Adam epoch times.

Load-bearing premise

The method rests on the assumption that the two per-layer matrices whose Kronecker product approximates the Fisher information are diagonally dominant at every layer and throughout training, so discarding their off-diagonal entries does not remove useful curvature information.

Editorial extensions

If this is right

  • AdaFisher and its weight-decay variant AdaFisherW should reach higher validation accuracy than Adam when both are run for the same wall-clock time, making second-order curvature information available without a per-iteration slowdown.
  • Distributed AdaFisher averages the Kronecker factors across GPUs before updating, so larger batch sizes (512, 1024) on ImageNet-1k continue to improve validation accuracy rather than degrading it.
  • The method should reduce hyperparameter sensitivity: the same settings work across multiple learning rates, batch sizes, and scheduler choices, as reported for ResNet-50 on CIFAR-10/100.
  • In language modeling, AdaFisherW should improve test perplexity on WikiText-2 and PTB compared with AdamW and AdaHessian, while K-FAC and Shampoo fail to scale or converge in the paper's setup.
  • The diagonal-concentration observation implies that for CNNs similar to ResNet-18, most useful curvature information in the Fisher factors lives on the diagonal, which is why the diagonalized preconditioner does not lose the benefit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to measure diagonal dominance of $H$ and $S$ in transformer blocks and very deep networks; if off-diagonal energy there is large, AdaFisher's approximation could be combined with block sizes larger than 1 to retain the advantage.
  • The reported tendency to land in flatter minima suggests AdaFisher could be stacked with stochastic weight averaging or sharpness-aware objectives to push generalization further, though the paper does not test this.
  • The removal of the square root from the adaptive update, which the paper argues is possible because the Fisher preconditioner already carries curvature information, implies that other Adam-family optimizers might adopt a similar normalization and drop the batch-size-dependent rescaling that plain Adam without the square root needs.
  • If the diagonal Fisher approximation is robust across architectures, the same trick could be applied to Hessian-based natural-gradient variants or to second-order fine-tuning of large language models, cutting memory and compute in regimes where full K-FAC is infeasible.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The thesis introduces AdaFisher, an adaptive second-order optimizer that replaces Adam's diagonal second-moment estimate with a diagonal block-Kronecker approximation of the empirical Fisher information matrix (EFIM). The central claim is that this preconditioner captures more curvature than Adam, yields faster convergence and better generalization, and does so at computational cost close to first-order methods. The manuscript supports this claim with a broad empirical study across CIFAR-10/100, Tiny ImageNet, ImageNet-1k, transfer learning, and language modeling, together with ablations on learning-rate schedulers, EMA of Kronecker factors, normalization layers, memory usage, and epoch times. A convergence analysis for convex and non-convex settings is also provided. The paper includes a public code repository and a series of visualizations of optimizer trajectories and Fisher spectra.

Significance. If the central claim held, AdaFisher would be a genuinely useful practical second-order optimizer: the algorithm is simple, the code is public, and the experimental body is broader than a typical conference paper, covering multiple architectures, datasets, and careful hyperparameter tuning. The diagonal-dominance observation for Kronecker factors is a useful empirical contribution, and the ablation/stability analysis is commendably thorough. However, the headline 'outperforms state-of-the-art' is not yet established: the flagship ImageNet comparison rests on an unusually weak in-house Adam baseline and a single run, the wall-clock comparison protocol gives baselines unequal epoch budgets, and the equal-epoch comparison shows Shampoo at least matching AdaFisher on one benchmark while AdaFisher is slower per epoch than Adam. The convergence proof also contains a gap that would need to be closed or explicitly scoped. With careful re-benchmarking and a corrected theory section, this could be a solid contribution.

major comments (4)
  1. [Section 5.2.1, Table 5.9] The ImageNet-1k comparison does not support the claim that AdaFisher outperforms state-of-the-art optimizers. The in-house Adam baseline reaches only 67.78 top-1 accuracy, while the same table lists standard SGD at 76.40 and AdamW at 76.34 from the literature; AdaFisher's margins over those literature baselines are only 0.55 to 0.61 points, and the experiment is a single run with no error bars. This suggests the reported 9-point gap over the in-house Adam is an artifact of baseline configuration rather than an AdaFisher advantage. The authors should tune a proper AdamW baseline, report multiple seeds or at least standard deviations, and clearly state which settings differ from the literature baselines.
  2. [Section 5.1.1, Tables 5.1 and 5.11] The wall-clock-time (WCT) protocol gives baselines unequal epoch budgets: in Table 5.1, Shampoo is trained for 36 epochs, K-FAC for 107, and AdaFisher for 200 on CIFAR/Tiny ImageNet. Consequently, the accuracies in Tables 5.7 and 5.8 conflate per-step progress with total wall-clock time. The equal-epoch comparison in Table 5.11 shows Shampoo marginally ahead of AdaFisher on ResNet-18/CIFAR-100 (77.35 vs. 77.28) and AdaFisher being about 14% slower per epoch than Adam (26.58 vs. 23.33 minutes), which undermines the claim of preserving first-order computational efficiency. The authors should present per-epoch learning curves alongside wall-clock results and be explicit about the speed-accuracy trade-off in each comparison.
  3. [Section 3.2 and Proposition 3.3.2] The diagonal-dominance assumption is load-bearing for the entire preconditioner design, but the evidence is limited to the 37th convolutional layer and 41st linear layer of ResNet-18 on CIFAR-10 at two training steps. Proposition 3.3.2 then applies the diagonal approximation globally to all layers, architectures, and datasets. If off-diagonal correlations in the Kronecker factors are significant in transformers, embeddings, or normalization layers, AdaFisher discards curvature information that K-FAC would retain, and the claimed advantage could disappear. The authors should test the diagonal-concentration hypothesis on representative layers of several architectures (including a transformer and a BatchNorm-only network) across datasets, and report the fraction of energy off the diagonal as a quantitative diagnostic.
  4. [Section 4.3, Proposition 4.3.1] The stated convergence bound is the standard gradient-descent bound, but the update is preconditioned by (F_D^(t))^{-1}; the proposition does not include a Lipschitz condition or boundedness assumption involving the preconditioner, so the inequality J(theta(k)) - J(theta*) <= ||theta(0)-theta*||^2 / (2 alpha k) does not follow from the statement as written. Proposition 4.3.2 similarly lists bounded preconditioner norm, bounded gradients, unbiased noise, and a monotone ratio condition as assumptions without demonstrating that AdaFisher's actual preconditioner satisfies them. The proofs are deferred to an appendix, but in the submitted text the appendix content is not available for verification. The convergence section should either state the exact assumptions under which the preconditioned update is proven to converge, or clearly label the result as a heuristic bound.
minor comments (4)
  1. [Section 5.1.1, ImageNet preprocessing] The text says 'random resized cropping to 224×244' but the standard ImageNet training size is 224×224; please correct this typo.
  2. [Section 6.1.2, Figure 6.2] The text refers to 'Panel (D)' in the discussion of the FIM approximation error, but the figure caption and the surrounding description list only panels (A), (B), and (C); please align the panel references.
  3. [Table 5.7 footnote] The footnote states that Adam and AdaFisher were used for CNNs while AdamW and AdaFisherW were used for ViTs; this conflation of optimizer variant with architecture family should be made more explicit in the main text so that readers do not interpret the table as a single-optimizer comparison.
  4. [Section 4.1] The update rule is written as theta(t+1) = theta(t) - alpha m(t)/v(t), but the elementwise nature of the division is not defined; please add a sentence clarifying that the operation is applied coordinate-wise or define the vectorized notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's derivation chain is self-contained and its predictions are validated against external benchmarks.

full rationale

The paper's central derivation is the diagonal block-Kronecker approximation of the Fisher Information Matrix. That approximation is motivated by an empirical study of Kronecker-factor diagonal concentration (Gershgorin discs, eigenvalue perturbation, FFT, SNR, and matrix visualizations for selected ResNet-18 layers), not by the target claim that AdaFisher outperforms other optimizers. The approximation is then used to define the preconditioner in Algorithm 1, and its performance is measured against Adam, SGD, K-FAC, Shampoo, AdaHessian, and other baselines on external datasets. No fitted parameter is relabeled as a prediction: the hyperparameters (learning rate, damping lambda, decay gamma) are tuned per optimizer and then fixed; the reported accuracy and convergence results come from the training runs themselves. The only self-citation, 'AdaFisher (GOMES, Zhang, Belilovsky, Wolf, & Hosseini, 2025),' appears in the literature-review bridge and is not load-bearing; the thesis contains the full method, experiments, and ablations. The convergence analysis (Propositions 4.3.1 and 4.3.2) follows standard Adam-type proof templates and states its assumptions explicitly; this is a rigor/scope issue rather than circularity. Although the diagonal-dominance evidence is limited to two layers of one architecture and later generalized, that is an inductive leap, not a circular reduction. Therefore, no circular step can be exhibited from the paper's own equations or citations.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an empirical diagonal-dominance assumption, the standard K-FAC independence assumption, and several hand-chosen hyperparameters such as lambda and gamma. No new physical or mathematical entities are introduced. The convergence rate is imported from Adam-type analysis under unverified conditions.

free parameters (4)
  • Tikhonov damping lambda = 0.001
    Used in the diagonal EFIM definition in Eq. (20) and set to 0.001 in Algorithm 1. Chosen by hand from a damping range inspired by K-FAC, not derived.
  • KF EMA decay gamma = 0.8
    Exponential moving average decay for Kronecker factors in Eq. (19). Tuned over {0.1, 0.2, ..., 0.9, 0.99} in Section 5.1.1, with 0.8 selected as optimal.
  • First-moment decay beta = 0.9
    Inherited from Adam in Algorithm 1 and treated as a default hyperparameter rather than derived from the Fisher approximation.
  • Learning rate alpha = per dataset and optimizer
    Tuned by grid search for every optimizer and dataset (Tables 5.2, 5.3, 5.6). Critical for the empirical comparison, though standard for optimizer benchmarks.
assumptions (4)
  • ad hoc to paper Kronecker factors H_i and S_i are well approximated by their diagonal entries for all layers and training stages.
    Section 3.2 demonstrates diagonal dominance on two layers of one ResNet-18 on CIFAR-10, and Proposition 3.3.2 promotes this to a global approximation used for all architectures and datasets.
  • domain assumption Activations and pre-activation derivatives are mutually independent, and gradients of different layers are uncorrelated.
    This is the standard K-FAC approximation, inherited from Grosse and Martens (2016) and used in Section 3.1 to justify the block-diagonal Kronecker form of the FIM.
  • domain assumption The Fisher information matrix approximates the Hessian of the negative log-likelihood, especially near local minima.
    Used in Sections 2.1.3 and 4.1 to interpret AdaFisher as a curvature-aware optimizer and to argue it finds flat minima.
  • ad hoc to paper The convergence assumptions in Proposition 4.3.2 hold for AdaFisher: bounded preconditioner norm, bounded gradients, unbiased noise, and a monotone ratio condition.
    These conditions are stated so that AdaFisher can inherit the Adam-type O(log T / sqrt T) rate, but they are not verified for the diagonal block-Kronecker preconditioner.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis." pith.science (2026). https://pith.science/paper/6SHJGYHI

@misc{pith2026250420096,
  author       = {Pith},
  title        = {Pith review of: Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6SHJGYHI}},
  note         = {Machine review of arXiv:2504.20096}
}
read the original abstract

First-order optimization methods remain the standard for training deep neural networks (DNNs). Optimizers like Adam incorporate limited curvature information by preconditioning the stochastic gradient with a diagonal matrix. Despite the widespread adoption of first-order methods, second-order optimization algorithms often exhibit superior convergence compared to methods like Adam and SGD. However, their practicality in training DNNs is still limited by a significantly higher per-iteration computational cost compared to first-order methods. In this thesis, we present AdaFisher, a novel adaptive second-order optimizer that leverages a diagonal block-Kronecker approximation of the Fisher information matrix to adaptively precondition gradients. AdaFisher aims to bridge the gap between the improved convergence and generalization of second-order methods and the computational efficiency needed for training DNNs. Despite the traditionally slower speed of second-order optimizers, AdaFisher is effective for tasks such as image classification and language modeling, exhibiting remarkable stability and robustness during hyperparameter tuning. We demonstrate that AdaFisher outperforms state-of-the-art optimizers in both accuracy and convergence speed. The code is available from https://github.com/AtlasAnalyticsLab/AdaFisher.

Figures

Figures reproduced from arXiv: 2504.20096 by the authors.

Figure 1
Figure 1. Visualizing optimization trajectories for various optimizers overlaid a loss [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 3
Figure 3. Gershgorin discs and eigenvalue perturbation analysis for matrices [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 3
Figure 3. Visualization of KFs [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Weight trajectories within different loss landscapes (evaluated using four [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Training loss and validation error of ResNet-50 on ImageNet-1k. AdaFisher [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 5
Figure 5. Figure 5: Training Loss and Test Perplexity of Small GPT-1 Model on WikiText-2 and [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 1.1
Figure 1.1. Figure 1.1: Visualizing optimization trajectories for various optimizers overlaid a loss landscape. [PITH_FULL_IMAGE:figures/full_fig_p019_1_1.png]
Figure 2.1
Figure 2.1. Figure 2.1: A 2D slice of a loss landscape. This is a 3D rendering of a 2D slice of the loss landscape [PITH_FULL_IMAGE:figures/full_fig_p032_2_1.png]
Figure 2.2
Figure 2.2. Figure 2.2: Three types of stationary points in non-convex optimization landscapes: local minima, [PITH_FULL_IMAGE:figures/full_fig_p036_2_2.png]
Figure 2.3
Figure 2.3. Figure 2.3: Illustration of EFIM computation using K-FAC for a given layer [PITH_FULL_IMAGE:figures/full_fig_p058_2_3.png]
Figure 3.1
Figure 3.1. Figure 3.1: Gershgorin discs and eigenvalue perturbations for the [PITH_FULL_IMAGE:figures/full_fig_p069_3_1.png]
Figure 3.2
Figure 3.2. Figure 3.2: Gershgorin discs and eigenvalue perturbation analysis for matrices [PITH_FULL_IMAGE:figures/full_fig_p069_3_2.png]
Figure 3.3
Figure 3.3. Figure 3.3: showcases the FFT magnitude plots for both noise-free and noisy conditions at training steps 5200 (middle of training) and 9800 (end of training). Even under Gaussian perturbations, the primary diagonal structure of the KFs remains conspicuous in the frequency domain…
Figure 3.4
Figure 3.4. Figure 3.4: Visualization of KFs H and S in convolutional (A) and linear (B) layers of ResNet-18 at different iteration steps (5200 and 9800). The first two plots in (A) depict factor H at the two training stages, while the next two plots illustrate factor S. Analogously, (B) sh…
Figure 4.1
Figure 4.1. Figure 4.1: Comparison of FIM Diagonal Histograms during ResNet18 Training on CIFAR10 with [PITH_FULL_IMAGE:figures/full_fig_p079_4_1.png]
Figure 4.2
Figure 4.2. Figure 4.2: Weight trajectories within different loss landscapes (evaluated using four seeds) of a toy [PITH_FULL_IMAGE:figures/full_fig_p084_4_2.png]
Figure 5.1
Figure 5.1. Figure 5.1: WCT training loss and testing error curves of several optimizers on Tiny ImageNet [PITH_FULL_IMAGE:figures/full_fig_p093_5_1.png]
Figure 5.2
Figure 5.2. Figure 5.2: Training loss and validation error of ResNet-50 on ImageNet-1k. AdaFisher consistently achieves lower test error as com￾pared to its counterparts [PITH_FULL_IMAGE:figures/full_fig_p094_5_2.png]
Figure 5.3
Figure 5.3. Figure 5.3: Performance of distributed AdaFisher using ResNet50 on ImageNet-1k with different [PITH_FULL_IMAGE:figures/full_fig_p094_5_3.png]
Figure 5.4
Figure 5.4. Figure 5.4: WCT training loss, test error, for ResNet-18 on CIFAR100 and MobileNet-V3 on CI [PITH_FULL_IMAGE:figures/full_fig_p095_5_4.png]
Figure 5.5
Figure 5.5. Figure 5.5: Performance comparison of AdaFisher and other well-finetuned optimizers at their best [PITH_FULL_IMAGE:figures/full_fig_p097_5_5.png]
Figure 5.6
Figure 5.6. Figure 5.6: Training Loss and Test Perplexity of Small GPT-1 Model on WikiText-2 and PTB [PITH_FULL_IMAGE:figures/full_fig_p098_5_6.png]
Figure 6.1
Figure 6.1. Figure 6.1: Performance comparison of AdaFisher using the ResNet50 on the CIFAR10 with a [PITH_FULL_IMAGE:figures/full_fig_p102_6_1.png]
Figure 6.2
Figure 6.2. Figure 6.2: AdaFisher Component Analysis. (A) Comparison of MAE between the true FIM [PITH_FULL_IMAGE:figures/full_fig_p102_6_2.png]
Figure 6.3
Figure 6.3. Figure 6.3: Performance comparison of AdaFisher and other optimizers using the ResNet50 net [PITH_FULL_IMAGE:figures/full_fig_p104_6_3.png]
Figure 6.4
Figure 6.4. Figure 6.4: (A) Performance comparison of AdaFisher and other optimizers across various batch [PITH_FULL_IMAGE:figures/full_fig_p106_6_4.png]
Figure 6.5
Figure 6.5. Figure 6.5: Epoch times for various networks on CIFAR10 (A) and CIFAR100 (B) using Adam, [PITH_FULL_IMAGE:figures/full_fig_p107_6_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DHO$_2$: Accelerating Distributed Hybrid Order Optimization via Model Parallelism and ADMM

    cs.LG 2025-05 conditional novelty 6.0 of 10

    DHO2 distributes FOSI's Hessian eigenvector computation with model parallelism and an ADMM-like update, claiming lower per-GPU memory and faster time-to-solution in distributed DNN training.

Reference graph

Works this paper leans on

26 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Al-Dujaili, A., & O’Reilly, U.-M. (2020). Sign bits are all you need for black-box attacks. In International conference on learning representations. Allen-Zhu, Z., Li, Y ., & Liang, Y . (2019). Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in neural information processing systems,

    work page 2020

  2. [2]

    A., & Johnson, C

    Horn, R. A., & Johnson, C. R. (2012). Matrix analysis (2nd ed.). Cambridge University Press. Howard, A., Sandler, M., Chu, G., Chen, L.-C., Chen, B., Tan, M., . . . others (2019). Searching for mobilenetv3. In Proceedings of the ieee/cvf international conference on computer vision(pp. 1314–1324). Hu, Z., & Huang, H. (2023). Optimization and bayes: a trade...

    work page arXiv 2012

  3. [3]

    (2001, 08)

    Lecun, Y . (2001, 08). A theoretical framework for back-propagation. LeCun, Y ., Bottou, L., Orr, G., & M¨uller, K. (2012). Efficient backprop. In Neural networks (pp. 9–48). Springer Verlag. (Copyright: Copyright 2021 Elsevier B.V ., All rights reserved.) doi: 10.1007/978-3-642-35289-8 3 Lei Ba, J., Kiros, J. R., & Hinton, G. E. (2016). Layer normalizati...

    doi:10.1007/978-3-642-35289-8 2012

  4. [7]

    B., & LeCun, Y

    Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., & LeCun, Y . (2015). The loss surfaces of multilayer networks. In Artificial intelligence and statistics (pp. 192–204). Dauphin, Y ., De Vries, H., & Bengio, Y . (2015). Equilibrated adaptive learning rates for non-convex optimization. Advances in neural information processing systems,

    work page 2015

  5. [10]

    Foret, P., Kleiner, A., Mobahi, H., & Neyshabur, B. (2021). Sharpness-aware minimization for efficiently improving generalization. InInternational conference on learning representations. Retrieved from https://openreview.net/forum?id=6Tm1mposlrM Fort, S., Dziugaite, G. K., Paul, M., Kharaghani, S., Roy, D. M., & Ganguli, S. (2020). Deep learning versus ke...

    2021

  6. [12]

    Ghadimi, S., & Lan, G. (2013). Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23(4), 2341–2368. Goldfarb, D., Ren, Y ., & Bahamou, A. (2020). Practical quasi-newton methods for training deep neural networks. Advances in Neural Information Processing Systems, 33, 2386–2396. Golovin, D., Karro, ...

    arXiv 2013

  7. [13]

    He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the ieee conference on computer vision and pattern recognition (pp. 770– 778). Heo, B., Chun, S., Oh, S. J., Han, D., Yun, S., Kim, G., . . . Ha, J.-W. (2021). Adamp: Slowing down the slowdown for momentum optimizers on scale-invariant weights. In...

    work page 2016

  8. [15]

    Jastrzebski, S., Szymczak, M., Fort, S., Arpit, D., Tabor, J., Cho*, K., & Geras*, K. (2020). The break-even point on optimization trajectories of deep neural networks. In International conference on learning representations. Retrieved from https://openreview.net/ forum?id=r1g87C4KwB Jiang, Z., Gu, J., Zhu, H., & Pan, D. (2024). Pre-rmsnorm and pre-crmsno...

    work page 2020

Show all 26 references
  1. [16]

    M., & Jordan, M

    Jin, C., Ge, R., Netrapalli, P., Kakade, S. M., & Jordan, M. I. (2017). How to escape saddle points efficiently. In International conference on machine learning (pp. 1724–1732). Kalra, D. S., & Barkeshli, M. (2023). Phase diagram of early training dynamics in deep neural netwo...

    2017 arXiv

  2. [18]

    578–594)

    (pp. 578–594). Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Pham, H., . . . others (2024). Symbolic discovery of optimization algorithms. Advances in neural information processing systems ,

    2024

  3. [20]

    Liu, Z., Lin, Y ., Cao, Y ., Hu, H., Wei, Y ., Zhang, Z., . . . Guo, B. (2021). Swin transformer: Hierar- chical vision transformer using shifted windows. In Proceedings of the ieee/cvf international conference on computer vision (pp. 10012–10022). Loshchilov, I., & Hutter, F....

    2021 arXiv

  4. [21]

    Patro, S., & Sahu, K. K. (2015). Normalization: A preprocessing stage. arXiv preprint arXiv:1503.06462. Polyak, B. T. (1964). Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 4(5), 1–17. 125 Radford, A.,...

    2015 arXiv

  5. [22]

    J., Kale, S., & Kumar, S

    Reddi, S. J., Kale, S., & Kumar, S. (2018). On the convergence of adam and beyond. InInternational conference on learning representations. Retrieved from https://openreview.net/ forum?id=ryQu7f-RZ Rotem, N., Fix, J., Abdulrasool, S., Catron, G., Deng, S., Dzhabarov, R., . . . ...

    2018 arXiv

  6. [23]

    S., Gunasekar, S., & Srebro, N

    Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., & Srebro, N. (2018). The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70), 1–57. Sun, H., Shen, L., Zhong, Q., Ding, L., Chen, S., Sun, J., . . . Tao, D. (2024). Adasam: Boos...

    2018

  7. [24]

    Xie, Z., Sato, I., & Sugiyama, M. (2021). A diffusion theory for deep learning dynamics: Stochas- tic gradient descent exponentially favors flat minima. In International conference on learn- ing representations. Retrieved from https://openreview.net/forum?id=wXgk iCiYGo Xie, Z...

    2021

  8. [25]

    122 Kunstner, F., Hennig, P., & Balles, L. (2019). Limitations of the empirical fisher approx- imation for natural gradient descent. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alch ´e-Buc, E. Fox, & R. Garnett (Eds.), Advances in neural infor- mation processing syste...

    2019

  9. [26]

    Zhang, L., Shi, S., & Li, B. (2023). Eva: Practical second-order optimization with kronecker- vectorized approximation. In The eleventh international conference on learning representa- tions. Retrieved from https://openreview.net/forum?id= Mic8V96Voy Zhang, Y ., Yao, Y ., Jia,...

    2023

  10. [27]

    Dedieu, J.-P. (2015). Newton-raphson method. In B. Engquist (Ed.), Encyclopedia of applied and computational mathematics (pp. 1023–1028). Berlin, Heidelberg: Springer Berlin Hei- delberg. Retrieved from https://doi.org/10.1007/978-3-540-70529-1 374 doi: 10.1007/978-3-540-70529...

    2015

  11. [28]

    N., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y

    117 Dauphin, Y. N., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y . (2014). Identi- fying and attacking the saddle point problem in high-dimensional non-convex optimization. Advances in neural information processing systems,

    2014

  12. [29]

    E., & Makhzani, A

    Lin, W., Dangel, F., Eschenhagen, R., Bae, J., Turner, R. E., & Makhzani, A. (2024). Can we remove the square-root in adaptive gradient methods? a second-order perspective. InForty-first inter- national conference on machine learning. Retrieved from https://openreview.net/ for...

    2024

  13. [30]

    Benzing, F. (2022). Gradient descent on neurons and its link to approximate second-order opti- mization. In International conference on machine learning (pp. 1817–1853). Bollapragada, R., & Wild, S. M. (2023). Adaptive sampling quasi-newton methods for zeroth-order stochastic ...

    2022 arXiv

  14. [31]

    (2021, February)

    119 George, T. (2021, February). NNGeometry: Easy and Fast Fisher Information Matrices and Neural Tangent Kernels in PyTorch. Zenodo. Retrieved from https://doi.org/10.5281/ zenodo.4532597 doi: 10.5281/zenodo.4532597 George, T., Laurent, C., Bouthillier, X., Ballas, N., & Vinc...

    2018 doi

  15. [32]

    Amari, S., & Nagaoka, H. (2000). Methods of information geometry.. Retrieved from https:// api.semanticscholar.org/CorpusID:116976027 Amid, E., Anil, R., & Warmuth, M. (2022). Locoprop: Enhancing backprop via local loss optimiza- tion. In International conference on artificial...

    2000

  16. [33]

    Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., & Tian, Y . (2024). Galore: Memory- efficient LLM training by gradient low-rank projection. In Forty-first international confer- ence on machine learning. Retrieved from https://openreview.net/forum?id= hYHsrKDiX7 Zhao, ...

    2024

  17. [36]

    Chen, X., Liu, S., Sun, R., & Hong, M. (2019a). On the convergence of a class of adam-type algo- rithms for non-convex optimization. InInternational conference on learning representations. Chen, X., Liu, S., Sun, R., & Hong, M. (2019b). On the convergence of a class of adam-ty...

    2019

  18. [450]

    D., Dhariwal, P.,

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., . . . others (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877–1901. Cha, J., Chun, S., Lee, K., Cho, H.-C., Park, S., Lee, Y ., & Park, S. (2021)...

    2020

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.