Pith. sign in

REVIEW 3 major objections 4 minor 44 references

MAC: An Efficient Gradient Preconditioning using Mean Activation Approximated Curvature

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MAC replaces KFAC's Fisher factors with a mean-activation rank-1 approximation, getting second-order convergence at near-SGD cost.

desk verdict A practically useful optimizer paper with a solid empirical case and a genuinely new attention-layer derivation, but the convergence proof has a real gap that should be fixed before the theory is cited. read the letter →

arxiv 2506.08464 v2 pith:MWG7QGU5 submitted 2025-06-10 cs.LG

Hyunseok Seung , Jaewoo Lee , Hyunsuk Ko This is my paper
classification cs.LG
keywords second-orderoptimizationKFACKronecker-factoredapproximationrank-1meanactivationpreconditioningattentionlayersFisherinformationmatrix
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MAC is an optimizer that aims to deliver the convergence benefits of KFAC-style second-order preconditioning at roughly first-order cost. It replaces the two Kronecker factors of each layer's Fisher information matrix with cheap stand-ins: the activation factor becomes the outer product of the mean activation vector, and the pre-activation gradient factor becomes the identity matrix. For transformer attention layers, the paper derives a Kronecker-factored Fisher approximation that keeps attention scores, weighting the value-projection curvature by the mean attention vector. If the method works as claimed, second-order training becomes practical on modern architectures, with memory near SGD and end-to-end time close to it while preserving or improving accuracy. The paper also proves linear convergence to a global minimum for an overparameterized two-layer ReLU network under two explicit conditions.

What carries the argument

The load-bearing object is the rank-1 mean-activation approximation $A \approx \mathbb{E}[a]\mathbb{E}[a]^\top$ for the activation Kronecker factor, together with $P \approx I$ for fully-connected and convolutional layers. This turns preconditioning into a rank-1 damped update, so the Sherman-Morrison formula gives the inverse in closed form: per-layer preconditioner cost drops from $O(d^3)$ time and $O(d^2)$ memory to $O(d^2)$ time and $O(d)$ memory. In attention layers the same rank-1 idea is applied to the attention-score matrix $T \approx \bar{t}\bar{t}^\top$, which reweights the value-projection curvature by $\mathbb{E}[X^\top \bar{t}]$ and thus keeps attention scores inside the preconditioner. The Davis-Kahan theorem is used to justify that the top eigenvector of $A$ points along the mean activation, which is what licenses the rank-1 collapse.

What would settle it

Measure the ratio $\|\Sigma_a\|_F / \|\mathbb{E}[a]\|^2$ layer by layer in a trained network; if a layer with a large ratio still shows MAC beating SGD, the rank-1 premise is not the operative explanation. A cleaner test is to train a transformer on data with sharply peaked attention distributions and compare MAC against a variant that keeps a low-rank attention covariance instead of the rank-1 $\bar{t}\bar{t}^\top$.

Watch

Extended reading notes

Core claim

The paper's central claim is that the layer-wise Fisher information matrix used by KFAC can be approximated as $$$F^{{(l)}}$_{\mathrm{MAC}} = (\mathbb{E}[a]\mathbb{E}[a]^\top + \rho I_{m_{l-1}}) \otimes I_{m_l}$$ with $\mathbb{E}[a]$ the mean activation vector, without losing KFAC's optimization benefits. The justification is empirical: the activation Kronecker factor has a dominant top eigenvalue whose eigenvector aligns with the mean activation, while the pre-activation gradient factor's eigenvalues are uniformly small and can be dropped. For attention layers, the paper derives $F_{\mathrm{MAC},q} = I_d \otimes \mathbb{E}[x]\mathbb{E}[x]^\top$, $F_{\mathrm{MAC},k} = I_d \otimes \mathbb{E}[x]\mathbb{E}[x]^\top$, and $F_{\mathrm{MAC},v} = I_d \otimes \mathbb{E}[X^\top \bar{t}]\mathbb{E}[X^\top \bar{t}]^\top$, where $\bar{t}$ is the mean attention vector, and claims this is the first Kronecker factorization of an attention-layer Fisher matrix that explicitly incorporates attention scores. Across CIFAR and ImageNet benchmarks, the paper reports that MAC matches or exceeds KFAC and its variants in accuracy while cutting end-to-end training time by up to 55.4% and using memory comparable to SGD.

Load-bearing premise

The method assumes that within each layer, activations stay close to their average, so a single average vector captures the dominant curvature direction; if activations vary widely, MAC no longer approximates the curvature KFAC relies on.

Editorial extensions

If this is right

  • KFAC-style preconditioning becomes scalable to large models: per-layer cost drops from cubic to quadratic time and from matrix to vector memory.
  • Transformers become a realistic target for second-order optimizers; the paper reports up to 3.6% higher ImageNet top-1 accuracy on DeiT-Small than KFAC, where KFAC itself frequently diverges.
  • Because the $P$ factor is dropped, MAC sidesteps the numerically unstable inversions that the paper says crash KFAC variants during large-scale training.
  • The convergence proof gives MAC a linear rate to a global minimum governed by $\|\mathbb{E}[a]\|^2$ rather than the full condition number of $X^\top X$, which the paper argues is comparable to KFAC's rate.
  • On the tested benchmarks MAC runs within 2-21% of SGD's wall-clock time and 0-5% of SGD's memory, so the practical trade-off between convergence quality and compute largely disappears.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: If the mean-activation premise transfers across architectures, it would suggest that KFAC's $P$ factor is largely redundant for non-negative, low-variance activations, extending the FOOF observation to a broader setting than the paper tests.
  • Inference: For natural-language transformers, where attention distributions are sharper and more variable, the rank-1 attention-score assumption is the most fragile piece; a testable extension is to replace $\bar{t}\bar{t}^\top$ with a small-rank attention covariance and compare sensitivity.
  • Inference: The decoupled-damping update can be read as SGD plus a correction that projects gradients onto the mean-activation direction, which suggests trying MAC-style preconditioning inside Adam-style momentum updates, though the paper does not explore that combination.
  • Inference: A direct stress test is to train with activations that have large variance, for example removing batch normalization or using much larger learning rates; if MAC's advantage over SGD vanishes there, the rank-1 approximation is the operative mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes MAC, a KFAC-style preconditioner that replaces the activation Kronecker factor A with a rank-1 outer product of the mean activation plus damping, and replaces the pre-activation gradient factor P with the identity. For transformer attention layers, the authors derive analogous approximations that reweight the value-projection preconditioner by the mean attention vector, thereby explicitly incorporating attention scores. The method is evaluated on CIFAR-10/100 and ImageNet-1k across CNNs and vision transformers, reporting competitive or better accuracy than KFAC, FOOF, Eva, and LNGD with wall-clock time and memory close to SGD. The paper also states a global-convergence theorem for a two-layer ReLU network trained with MAC.

Significance. If the empirical claims hold, MAC is a practically valuable simplification of KFAC: it replaces matrix-valued Kronecker factors with vectors, enables closed-form inversion via Sherman-Morrison, and is plausibly the first Kronecker-factored preconditioner for attention layers that explicitly uses attention scores. The experimental evaluation is extensive, covers several architectures and dataset scales, and the authors provide code. The main weakness is theoretical: the proof of Theorem 5.5 contains an invalid step concerning the output-layer weights q_r, so the stated global-convergence guarantee is not established as written. This is a load-bearing issue for the paper's theoretical contribution, although the algorithmic and empirical parts remain credible and potentially fixable.

major comments (3)
  1. [Appendix B.3 and Theorem 5.5] The proof of Theorem 5.5 is invalid at the step where the Jacobian Gram matrix SS^T is replaced by ΓΓ^T. With the stated initialization q_r ~ Unif(-1,1), the entries of S are (1/sqrt(m)) q_r φ'(z_i[r]), so (SS^T)_{ij} = (1/m) Σ_r q_r^2 φ'(z_i[r]) φ'(z_j[r]), which is not equal to (1/m) Σ_r φ'(z_i[r]) φ'(z_j[r]) = (ΓΓ^T)_{ij}. The manuscript claims that q_r ∈ (-1,1) makes q_r 'disappear' in SS^T, but q_r^2 is not identically 1. The subsequent eigenvalue lower bound λ_min(X(·)X^T ⊙ ΓΓ^T) ≥ λ_min(X^T X) λ_Γ / (2(ρ + ||x̄||^2)) and the width condition m = Ω(...) both depend on this substitution. Thus Theorem 5.5's linear convergence to a global minimum is not proven for the stated initialization. The standard fix is to initialize q_r ∈ {±1} (Rademacher), or to add a concentration argument for the random matrix (1/m) Σ_r q_r^2 φ'(Xw_r)φ'(Xw_r)^T. As written, a central theoretical claim is unsupported.
  2. [Table 3 and Section 6.3] The ImageNet results in Table 3 report single accuracy numbers with no variance, number of seeds, or run-level statistics, while the CIFAR tables report standard deviations over five runs. The central empirical claim that MAC achieves the highest average accuracy and is ranked first on ImageNet is therefore not supported to the same standard. Several entries are marked with '✗' (training failure) for KFAC, FOOF, and Eva on vision transformers, but the failure criterion is not defined. Please add repeated runs with standard deviations or error bars, and specify what constitutes a training failure and how many runs failed.
  3. [Section 4.4 and Algorithm 1] The adaptive damping formula ρ = (trace(E[aa^T]) - ||E[a]||^2) / m_{l-1} is introduced as dynamically adjusting ρ, but Algorithm 1 does not show when or how ρ is updated, and it is unclear whether the bias-corrected EMA statistic â^{(l)}_k or the raw EMA is used in this trace-matching computation. Since ρ appears both in the Sherman-Morrison inverse and in the 'decoupled' update in Eq. (7), the pseudocode should match the equation exactly and state the update schedule for ρ.
minor comments (4)
  1. [Section 6.1 / Appendix C] Hyperparameters for MAC, FOOF, and LNGD are grid-searched, while KFAC and Eva use recommended values from the literature; this asymmetry should be acknowledged as a potential source of bias in favor of the proposed method.
  2. [Section 4.3] There is a typo in the final sentence of Section 4.3: 'precoditioning' should be 'preconditioning'.
  3. [Related Work / Introduction] The paper cites Sophia [23] in the introduction but does not discuss it in the related work section; given that Sophia is also a second-order-style optimizer for transformers, a brief comparison or justification for its exclusion would help position the contribution.
  4. [Tables 8 and 9 / Appendix D] The appendix introduces SMAC with its own experimental tables, but the main text does not reference SMAC or explain its status relative to MAC; adding a sentence in the main text would avoid reader confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MAC's approximations are empirical design choices validated by external benchmarks, not equations that presuppose the claimed outcomes.

full rationale

The paper's central construction, F(l)_MAC = (E[a]E[a]^T + ρI) ⊗ I, is proposed as an approximation to the KFAC Fisher block, justified by eigenanalysis of the activation and pre-activation Kronecker factors and by empirical measurements of covariance norms and eigenvector alignment. These are testable empirical hypotheses, not definitions that build the target conclusion into the input. The adaptive damping rule sets ρ to match the trace of the true covariance by construction, but the paper does not present this trace-matching as a prediction; the accuracy, speed, and memory claims are evaluated against held-out test accuracy and wall-clock time, so no fitted parameter is renamed as a predicted outcome. The convergence proof for the two-layer ReLU network uses F_MAC as defined and standard NTK-style lemmas; the questionable step in which q_r is said to disappear from S S^T is a mathematical error in the proof rather than a circular reduction, since the theorem's statement does not assume the desired convergence rate. The paper contains no load-bearing self-citations: the cited prior work on KFAC, FOOF, and Eva is external, and the claimed novelty of applying Kronecker-factorized preconditioning to attention layers is supported by a separate derivation from the attention backpropagation equations. Therefore, no equation-level identity between inputs and outputs was found, and the derivation chain is not circular.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

No new physical or mathematical entities are introduced; the MAC preconditioner is an approximation of the existing KFAC curvature. The central approximations rest on empirical premises about activation and attention spectra, plus standard NTK assumptions for the convergence theorem. The main free parameters are the damping, the EMA coefficient, and the update frequencies, all of which affect the reported speed and accuracy.

free parameters (5)
  • Damping ρ = 1.0 (CIFAR); 1.0/3.0 (ImageNet ResNets/ViTs); grid-searched in experiments
    Controls the diagonal regularization in the rank-1 preconditioner and the effective scale of the update. The paper also introduces a data-dependent trace-matching formula for ρ.
  • EMA coefficient β2 = 0.95
    Smooths the mean activation and attention statistics; affects the quality of the curvature estimate.
  • Curvature update frequency τcov = 5
    Sets how often the maintained statistics are refreshed; trades computation against approximation accuracy.
  • Inverse update frequency τinv = 50 (5 for ViTs)
    Controls how often the Sherman-Morrison preconditioner is recomputed; central to the reported wall-clock speedups.
  • Learning rate η = 0.1 (CIFAR); 0.5 (ImageNet); grid-searched for MAC
    Standard optimizer hyperparameter, but the paper grid-searches it for MAC and two baselines, which affects the fairness of comparisons.
assumptions (6)
  • domain assumption Inputs have unit norm, no two inputs are parallel, and labels are bounded (Assumption 5.1).
    Used in the 2-layer ReLU convergence proof; standard in NTK analyses but not guaranteed for real datasets.
  • standard math The limiting Gram matrix Σ∞ is positive definite with minimum eigenvalue λΓ (Condition 5.3).
    Standard NTK condition required for the linear-rate argument in Theorem 5.5.
  • domain assumption Stable Jacobian condition ||J - J(θ0)||_2 ≤ Cρ/(2σmax(X)) (Condition 5.4).
    The paper argues this holds for sufficiently wide networks, but it is stated as a condition in the theorem.
  • ad hoc to paper The centered activation covariance Σ_a is small in Frobenius norm relative to the mean outer product E[a]E[a]^T (Proposition 4.1 and Figure 2).
    This is the key empirical premise justifying the rank-1 approximation of A; it is demonstrated on a few architectures rather than proven generally.
  • ad hoc to paper The attention score matrix T is approximately rank-1 and its top eigenvector aligns with the mean attention vector t̄ (Figure 3).
    Used to replace vec(T)vec(T)^T by a rank-1 outer product in the value FIM derivation.
  • ad hoc to paper The pre-activation gradient covariance can be replaced by the identity in attention FIMs (Eqs. 4 to 6).
    Based on the empirical eigenspectrum observation that P has small and relatively uniform eigenvalues; not derived from first principles.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MAC: An Efficient Gradient Preconditioning using Mean Activation Approximated Curvature." pith.science (2026). https://pith.science/paper/MWG7QGU5

@misc{pith2026250608464,
  author       = {Pith},
  title        = {Pith review of: MAC: An Efficient Gradient Preconditioning using Mean Activation Approximated Curvature},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MWG7QGU5}},
  note         = {Machine review of arXiv:2506.08464}
}
read the original abstract

Second-order optimization methods for training neural networks, such as KFAC, exhibit superior convergence by utilizing curvature information of loss landscape. However, it comes at the expense of high computational burden. In this work, we analyze the two components that constitute the layer-wise Fisher information matrix (FIM) used in KFAC: the Kronecker factors related to activations and pre-activation gradients. Based on empirical observations on their eigenspectra, we propose efficient approximations for them, resulting in a computationally efficient optimization method called MAC. To the best of our knowledge, MAC is the first algorithm to apply the Kronecker factorization to the FIM of attention layers used in transformers and explicitly integrate attention scores into the preconditioning. We also study the convergence property of MAC on nonlinear neural networks and provide two conditions under which it converges to global minima. Our extensive evaluations on various network architectures and datasets show that the proposed method outperforms KFAC and other state-of-the-art methods in terms of accuracy, end-to-end training time, and memory usage.

Figures

Figures reproduced from arXiv: 2506.08464 by the authors.

Figure 1
Figure 1. Top-50 eigenspectra of FIM, activation KF, and pre-activation gradient KF in KFAC were analyzed [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. (Left) Cosine similarity between the top eigenvector of A and the mean activations per layer. (Right) Comparison of centered covariance norms with squared norms of mean activation using the CIFAR-100 dataset. Proposition 4.1. Let X be an m × n matrix with column-wise mean vector x¯ ∈ R n. Define a perturbation matrix E such that X = 1mx¯ ⊺ + E, where 1m is an m-dimensional column vector of ones. For some small ϵ > 0… view at source ↗
Figure 3
Figure 3. Trained DeiT-Tiny on Tiny ImageNet. (Left, Center) Eigenspectra of attention scores T from two distinct blocks as representative cases. (Right) Cosine similarity between the top eigenvector of T and the mean attention per block. During backpropagation, we have ∂L ∂Wq = ∂L ∂R ∂R ∂Q ∂Q ∂Wq = X ⊺ ∂L ∂R K = X ⊺∆R K , ∂L ∂Wk = ∂L ∂R ∂R ∂K ∂K ∂Wk = X ⊺ ∆ ⊺ R Q , ∂L ∂Wv = ∂L ∂H ∂H ∂V ∂V ∂Wv = X ⊺ T ⊺ ∆H , where ∆R = ∂L/∂R … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Comparison of the convergence factor between KFAC and [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison of train loss and test accuracy over wall-clock time on CIFAR-100 dataset. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Hyperparameter sensitivity analysis of MAC compared to KFAC and Eva on CIFAR-10 using ResNet￾32, trained for 100 epochs. Subplots show test accuracy across variations in (a) learning rate, (b) EMA coefficient, (c) damping, and (d) batch size. 0 50 100 150 200 250 300 W…
Figure 7
Figure 7. Figure 7: Wall-clock time comparison of different inverse update frequencies (1, 10, 50, and 100 steps) during [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Heatmap visualizations of the pre-activation gradient KF for convolutional and linear layers in LeNet-5 [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 38 canonical work pages

  1. [1]

    Natural gradient works efficiently in learning.Neural Computation, 1998

    Shun-ichi Amari. Natural gradient works efficiently in learning.Neural Computation, 1998

    work page 1998

  2. [2]

    Distributed second-order optimization using kronecker-factored approximations

    Jimmy Ba, Roger Baker Grosse, and James Martens. Distributed second-order optimization using kronecker-factored approximations. InProceedings of the International Conference on Learning Representations, 2016

    work page 2016

  3. [3]

    Gradient descent on neurons and its link to approximate second-order optimization

    Frederik Benzing. Gradient descent on neurons and its link to approximate second-order optimization. InProceedings of the International Conference on Machine Learning, 2022

    work page 2022

  4. [4]

    Multi- grain: a unified image embedding for classes and instances.ArXiv, 2019

    Maxim Berman, Hervé Jégou, Andrea Vedaldi, Iasonas Kokkinos, and Matthijs Douze. Multi- grain: a unified image embedding for classes and instances.ArXiv, 2019

    work page 2019

  5. [5]

    Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V . Le. Randaugment: Practical automated data augmentation with a reduced search space.IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019

    work page 2019

  6. [6]

    Li, and Li Fei-Fei

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database.IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

    work page 2009

  7. [7]

    Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout.ArXiv, 2017

    work page 2017

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InProceedings of the International Conference on Learning Representations, 2021

    2021

Show all 44 references
  1. [9]

    Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh

    Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. InProceedings of the International Conference on Learning Representations, 2019

    2019

  2. [10]

    M-FAC: Efficient matrix-free approximations of second-order information

    Elias Frantar, Eldar Kurtic, and Dan Alistarh. M-FAC: Efficient matrix-free approximations of second-order information. In A. Beygelzimer, Y . Dauphin, P. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, 2021

    2021

  3. [11]

    Practical quasi-newton methods for training deep neural networks

    Donald Goldfarb, Yi Ren, and Achraf Bahamou. Practical quasi-newton methods for training deep neural networks. InAdvances in Neural Information Processing Systems, 2020

    2020

  4. [12]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. InProceedings of the International Conference on Machine Learning, 2018

    2018

  5. [13]

    Deep residual learning for im- age recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016

    2016

  6. [14]

    Augment your batch: better training with larger batches.ArXiv, 2019

    Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: better training with larger batches.ArXiv, 2019

    2019

  7. [15]

    Weinberger

    Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. IEEE Conference on Computer Vision and Pattern Recognition, 2016

    2016

  8. [16]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InProceedings of the International Conference on Learning Representations, 2015

    2015

  9. [17]

    Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition.ESAIM: Proceedings and Surveys, 2023

    Abdoulaye Koroko, Ani Anciaux-Sedrakian, Ibtihel Ben Gharbia, Valérie Garès, Mounir Haddou, and Quang-Huy Tran. Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition.ESAIM: Proceedings and Surveys, 2023. 13

    2023

  10. [18]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009

    2009

  11. [19]

    Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015

    2015

  12. [20]

    Lecun, L

    Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner. Gradient-based learning applied to document recognition.Proceedings of the IEEE, 1998

    1998

  13. [21]

    Turner, and Alireza Makhzani

    Wu Lin, Felix Dangel, Runa Eschenhagen, Kirill Neklyudov, Agustinus Kristiadi, Richard E. Turner, and Alireza Makhzani. Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC for Large Neural Nets. October 2023

    2023

  14. [22]

    Simplifying momentum-based positive-definite submanifold optimization with applications to deep learning

    Wu Lin, Valentin Duruisseaux, Melvin Leok, Frank Nielsen, Mohammad Emtiyaz Khan, and Mark Schmidt. Simplifying momentum-based positive-definite submanifold optimization with applications to deep learning. InProceedings of the International Conference on Machine Learning, 2023

    2023

  15. [23]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training

    Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. InProceedings of the International Conference on Learning Representations, 2024

    2024

  16. [24]

    A layer-wise natural gradient optimizer for training deep neural networks

    Xiaolei Liu, Shaoshuai Li, Kaixin Gao, and Binfeng Wang. A layer-wise natural gradient optimizer for training deep neural networks. InProceedings of the Conference on Neural Information Processing Systems, 2024

    2024

  17. [25]

    Swin transformer: Hierarchical vision transformer using shifted windows.IEEE/CVF International Conference on Computer Vision (ICCV), 2021

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows.IEEE/CVF International Conference on Computer Vision (ICCV), 2021

    2021

  18. [26]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. Proceedings of the International Conference on Learning Representations, 2016

    2016

  19. [27]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InProceedings of the International Conference on Learning Representations, 2019

    2019

  20. [28]

    Optimizing neural networks with kronecker-factored ap- proximate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored ap- proximate curvature. InProceedings of the International Conference on Machine Learning, 2015

    2015

  21. [29]

    Müller and Frank Hutter

    Samuel G. Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmentation.IEEE/CVF International Conference on Computer Vision (ICCV), pages 754– 762, 2021

    2021

  22. [30]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  23. [31]

    J. Schur. Bemerkungen zur theorie der beschränkten bilinearformen mit unendlich vielen veränderlichen.Journal für die reine und angewandte Mathematik (Crelles Journal), 1911:1 – 28

    1911

  24. [32]

    Rethinking the inception architecture for computer vision.IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826, 2015

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision.IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826, 2015

    2015

  25. [33]

    Skfac: Training neural networks with faster kronecker-factored approximate curvature.IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13474–13482, 2021

    Zedong Tang, Fenlong Jiang, Maoguo Gong, Hao Li, Yue Wu, Fan Yu, Zidong Wang, and Min Wang. Skfac: Training neural networks with faster kronecker-factored approximate curvature.IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13474–13482, 2021. 14

    2021

  26. [34]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv’e J’egou. Training data-efficient image transformers & distillation through attention. In Proceedings of the International Conference on Machine Learning, 2020

    2020

  27. [35]

    Xiaoxia Wu, Simon Shaolei Du, and Rachel A. Ward. Global convergence of adaptive gradient methods for an over-parameterized neural network.ArXiv, 2019

    2019

  28. [36]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017

    Han Xiao, Kashif Rasul, and Roland V ollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017

    2017

  29. [37]

    Zhewei Yao, Amir Gholami, Sheng Shen, Kurt Keutzer, and Michael W. Mahoney. Adahes- sian: An adaptive second order optimizer for machine learning. InProceedings of the AAAI Conference on Artificial Intelligence, 2020

    2020

  30. [38]

    Samworth

    Yi Yu, Tengyao Wang, and Richard J. Samworth. A useful variant of the davis–kahan theorem for statisticians.Biometrika, 102:315–323, 2014

    2014

  31. [39]

    Cutmix: Regularization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Young Joon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. IEEE/CVF International Conference on Computer Vision (ICCV), 2019

    2019

  32. [40]

    Wide residual networks.British Machine Vision Conference, 2016

    Sergey Zagoruyko and Nikos Komodakis. Wide residual networks.British Machine Vision Conference, 2016

    2016

  33. [41]

    Fast convergence of natural gradient descent for overparameterized neural networks, 2019

    Guodong Zhang, James Martens, and Roger Grosse. Fast convergence of natural gradient descent for overparameterized neural networks, 2019

    2019

  34. [42]

    Dauphin, and David Lopez-Paz

    Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. InProceedings of the International Conference on Learning Representations, 2018

    2018

  35. [43]

    Eva: Practical second-order optimization with kronecker- vectorized approximation

    Lin Zhang, Shaohuai Shi, and Bo Li. Eva: Practical second-order optimization with kronecker- vectorized approximation. InProceedings of the International Conference on Learning Repre- sentations, 2023

    2023

  36. [44]

    Random erasing data augmentation.AAAI Conference on Artificial Intelligence, 2017

    Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation.AAAI Conference on Artificial Intelligence, 2017. 15 Appendix A Proof of Proposition 4.1 Proof. Let X be an m×n matrix and ¯x∈R n be the mean vector of X. Define a perturbation mat...

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.