Pith. sign in

REVIEW 4 major objections 5 minor 41 references

NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NysAct preconditions gradients with a low-rank Nyström approximation of the activation covariance matrix, matching second-order optimizers at near-SGD speed.

desk verdict Promising fast Nyström preconditioner with sound empirical scaling, but the convergence theorem analyzes a different update than the pseudocode and the abstract's accuracy claim is contradicted by the paper's own tables. read the letter →

arxiv 2506.08360 v1 pith:CCEILMG7 submitted 2025-06-10 cs.LG

Hyunseok Seung , Jaewoo Lee , Hyunsuk Ko This is my paper · ORCID
classification cs.LG
keywords gradientpreconditioningNyströmapproximationactivationcovariancesecond-orderoptimizationdeeplearningoptimizerconvergenceanalysisimageclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a cheap randomized sketch of the activation covariance matrix is enough to precondition gradients in deep neural networks, giving the accuracy benefits of curvature-based preconditioning without storing or inverting full matrices. The proposed optimizer, NysAct, keeps an exponential moving average of activation covariances but stores only a rank-$r$ Nyström sketch per layer, dropping memory from $O(d^2)$ to $O(dr)$ and inversion cost from $O(d^3)$ to roughly $O(d r^2)$. On CIFAR image classification the authors report test accuracy that matches or beats KFAC, Eva, and SGD while coming close to the stronger FOOF baseline, at 1.2 to 1.4 times SGD wall-clock time instead of the 1.7 to 2.5 times typical of other preconditioners; on ImageNet it reaches fixed accuracy thresholds with less wall-clock time than SGD and the second-order baselines. They also prove an $O(1/T)$ gradient-norm convergence rate for non-convex smooth losses, with the Nyström error entering the bound linearly. If the claim holds, expensive curvature-based preconditioning becomes tractable for large models at almost first-order cost.

What carries the argument

The load-bearing object is the eigenvalue-shifted Nyström approximation of a positive-semidefinite covariance matrix: given a sketch $S$ made of $r$ randomly chosen columns (or $r$ Gaussian combinations) of the activation covariance $A$, NysAct forms $A S (S^T A S)^+ S^T A$ and computes its inverse stably using damping $\rho$, a Cholesky factor $R$ of $S^T (\tilde C_{\text{damped}}) S$, and an SVD of $\tilde C_{\text{damped}} R^{-1}$, with a shift by $|\lambda_{\min}| + \rho$ added and then removed to keep the linear algebra positive definite. This reduces per-layer preconditioning from $O(d_{\text{out}}^3 + d_{\text{in}}^3)$ time and $O(d_{\text{out}}^2 + d_{\text{in}}^2)$ memory for KFAC, or $O(d_{\text{in}}^3)$ and $O(d_{\text{in}}^2)$ for FOOF, to $O(r^3 + d_{\text{in}} r^2)$ time and $O(d_{\text{in}} r)$ memory, with $r = 10$ on CIFAR and $r = 50$ or $20$ on ImageNet. The mechanism carries the argument because every expensive matrix in the preconditioner is replaced by a small sketch, while the convergence theorem tracks the approximation error $\epsilon$ through the eigenvalues of the preconditioned update.

What would settle it

On a small network, maintain the exact exponential moving average $C_k$ of activation covariances alongside NysAct's sketch, and at each update compare the Nyström approximation of $C_k$ with the sketch-averaged preconditioner from Algorithm 1. A Frobenius-norm gap that grows with training steps rather than shrinking as the rank $r$ increases, or a clear test-accuracy gap favoring the exact-EMA version, would show the sketch-EMA update is not a valid stand-in.

Watch

Extended reading notes

Core claim

The paper's central claim is that the exponential moving average of activation covariances is a sufficient preconditioner for gradient descent, and that this matrix can be replaced by its eigenvalue-shifted Nyström approximation without losing accuracy. Concretely, NysAct maintains $\tilde C_k = \beta_2 \tilde C_{k-1} + (1-\beta_2) A_k S_k$ for each layer, a rank-$r$ sketch of the covariance EMA, and builds a stable damped inverse from the Cholesky and singular value decompositions of the small $r \times r$ sketch. The claimed payoff is that a method with $O(dr)$ memory and near-SGD wall-clock time matches or exceeds KFAC, Eva, and SGD on CIFAR classification, trails the full-covariance FOOF baseline by only a few tenths of a percent, and reaches ImageNet accuracy thresholds faster in wall-clock time than SGD and the second-order methods, while Theorem 5.5 gives a gradient-norm convergence rate of $O(1/T)$ under Assumptions 5.1 through 5.4.

Load-bearing premise

The method's correctness depends on treating the exponentially averaged random column samples of the covariance as though they were random column samples of the exponentially averaged covariance, even though the sampled columns are redrawn at every update; if that averaging shortcut fails, both the preconditioner and the proof lose their footing.

Editorial extensions

If this is right

  • A rank as small as 10 on CIFAR captures enough of the activation covariance structure that NysAct stays within a few tenths of a percent of exact FOOF, so the Nyström approximation, not the full matrix, is doing the work.
  • On ImageNet, NysAct-S reaches the 72% top-1 threshold for ResNet-50 in 8.21 hours versus 8.48 for SGD and 10.10 for KFAC, showing the speedup persists at 27M-parameter scale.
  • KFAC and FOOF fail to complete 200-epoch DeiT-S training due to numerical issues in preconditioner inversion, while NysAct-S completes it, suggesting the damped shifted Nyström inverse is more robust.
  • Theorem 5.5 makes the Nyström approximation error enter the convergence rate linearly, so reducing $\epsilon$, for example by raising the rank, should speed convergence as well as improve the preconditioner.
  • Subcolumn sampling is the safer choice at ImageNet scale; Gaussian sketching performs comparably on small datasets but degrades on large-scale inputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same sketch-EMA preconditioner should transfer to embedding and attention-projection layers of transformers, where the analogous covariance matrices are also too large to invert exactly; the DeiT-S experiment is a first hint but does not establish this.
  • Editorial inference: because the convergence bound depends on the Nyström error $\epsilon$, an adaptive-rank schedule that increases $r$ when preconditioned gradient norms stall is a natural untested extension.
  • Editorial inference: the sketch-EMA update is the part most worth stress-testing; comparing NysAct against an exact-EMA preconditioner on a small network would cleanly separate approximation error from the averaging-invalidity issue.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces NYSACT, a preconditioned gradient method that replaces the exact activation covariance matrix in FOOF/KFAC-style updates with an eigenvalue-shifted Nyström approximation maintained as an exponential moving average of sketched activations. Two sketching variants are proposed (Gaussian, NYSACT-G; column sampling, NYSACT-S). The paper claims a convergence rate of O(1/T) for non-convex smooth stochastic optimization (Theorem 5.5), reports complexity advantages over KFAC/FOOF/Eva, and presents image-classification experiments on CIFAR and ImageNet. The central theoretical and empirical claims are not supported by the manuscript as written.

Significance. If it worked as claimed, NYSACT would be a practically useful scalable preconditioner: the complexity analysis in Table 1 and the wall-clock/memory results in Tables 3 and 6 are attractive, and the empirical study is wide (multiple architectures, datasets, epochs, ablations over rank, damping, and update frequency). The code is made public. However, the core algorithmic object maintained in Algorithm 1 is not a valid Nyström sketch of the EMA covariance, so the convergence theorem analyzes a different update from the one implemented; the proof of that theorem also contains a sign error. The abstract's accuracy claim is contradicted by the paper's own tables. These are load-bearing issues, so the contribution as presented is not established.

major comments (4)
  1. [Algorithm 1, Eq. (4)] The maintained array eC_k = β2 eC_{k-1} + (1-β2) A_k S_k is not a Nyström sketch of the EMA covariance C_k, because S_k is freshly drawn at every covariance update (lines 3-7) and sketching does not commute with the EMA. For Gaussian sketching with zero-mean entries independent of the activations, the conditional expectation of eC_k over the sketching randomness is zero, so the object carries no covariance information in expectation. Similarly, line 12 adds ρ S_k rather than a fixed projection of ρI, so bC_{k,damped} is not a sketch of C_k + ρI. Consequently, the preconditioner computed in lines 13-20 is not an approximation of (C_k+ρI)^{-1}, and the update analyzed in Appendix A, W_{k+1}=W_k-ηG(W_k)(A_nys+ρI)^{-1}, is not the update implemented by Algorithm 1. This mismatch invalidates Theorem 5.5 as a statement about NYSACT.
  2. [Appendix A, proof of Theorem 5.5] The descent inequality is derived with the wrong direction. After Taylor expansion the proof needs to upper-bound -η Tr(∇L(W_k)(A_nys+ρI)^{-1}G(W_k)^⊺), but it invokes Tr(ABC) ≤ ‖A‖_F‖B‖_2‖C‖_F, which is an upper bound on the trace, not on its negative. Since the trace can be negative, this yields a lower bound on -ηTr(...), so the displayed chain of inequalities is invalid. A lower bound on Tr (e.g., Tr(ABC) ≥ -‖A‖_F‖B‖_2‖C‖_F) is needed, and it is not supplied.
  3. [Eq. (6) and the O(1/T) claim] Even setting the sign issue aside, the algebra leading to Eq. (6) is incorrect and the claimed O(1/T) rate does not follow. After summing and dividing by T, the variance term should have denominator 2(λL-ε νL+ρ)^2/(λU+ε νU+ρ) - ηL, not 2(λL-ε νL+ρ)^2 - ηL(λU+ε νU+ρ). Moreover, with the constant learning rate specified in Theorem 5.5, the variance term in Eq. (6) is independent of T, so the right-hand side does not tend to zero as T→∞ unless η is also sent to zero. The advertised O(1/T) convergence for fixed η is therefore unsupported.
  4. [Abstract and Section 1.1] The abstract's claim that NYSACT 'achieves improved test accuracy compared to both first-order and second-order methods' is contradicted by the paper's own tables. In Table 2, FOOF (a first-order preconditioning baseline) is the best method in every reported CIFAR block, with NYSACT second; in Table 4, NYSACT-S on ImageNet ResNet-50 reaches 75.62%/78.77% top-1 accuracy at 100/200 epochs versus 78.05%/79.46% for SGD and 78.37%/79.69% for FOOF. The contribution bullet in Section 1.1 repeats the unsupported accuracy claim. The experimental section itself is more measured, but the abstract and contributions overstate the results.
minor comments (5)
  1. [Section 4, Eq. (5)] The symbol eC_k is used both for the EMA of sketched covariances and, in Eq. (5), for a Nyström sketch of a single covariance; please use distinct notation to avoid conflating the two objects.
  2. [Section 4, line 12] The sentence 'to ensure the positive semi-definiteness C_k' should refer to C_k + ρI; the damping is added to the sketched quantity, not to the raw covariance.
  3. [Section 5, Figure 2] The empirical support for Assumption 5.3 shows only the Frobenius norm of activation covariances; the assumption also requires a uniform lower bound on eigenvalues, which the figure does not address.
  4. [Tables 4-6] Only NYSACT-S is reported for ImageNet, while the algorithm defines both NYSACT-G and NYSACT-S; please specify which variant is used in each table and whether the reported times and accuracies correspond to that variant alone.
  5. [Table 7] The ImageNet row for NYSACT lists two values for weight decay ('0.00002 / 0.0001') and two values for r ('50 / 20') without explaining which applies to ResNet-50 and which to DeiT-S; please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper's claims rest on external prior work and standard assumptions rather than on fitted inputs or self-citation.

full rationale

No load-bearing circular step appears in NysAct. The method combines two external ingredients: Benzing's FOOF observation that the activation covariance, not the pre-activation gradient covariance, drives KFAC's performance [2], and Tropp et al.'s eigenvalue-shifted Nyström approximation [35]. Neither ingredient is derived from, nor fitted to, the paper's own test-accuracy claims. Assumption 5.4 posits a bound on the Nyström approximation error and is used inside Theorem 5.5, but this is an assumption about the approximation, not a restatement of the convergence theorem; the theorem's stated rate is then derived, not assumed. The hyperparameters are grid-searched in the experiments, but no fitted parameter is renamed as a prediction. The paper contains no self-citation of the authors' prior work, and no uniqueness theorem or ansatz is imported from the authors themselves. Concerns raised by a skeptical reader, such as the resampled sketching matrix in the EMA update not being a sketch of the EMA covariance, the sign of the trace bound in the proof of Theorem 5.5, and the ImageNet results showing NYSACT below SGD and FOOF on ResNet-50, are correctness or validation issues rather than circularity: they do not show that a claimed 'prediction' is equivalent by construction to its inputs. Accordingly, the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities. Its free parameters are standard optimizer hyperparameters plus the rank and damping; the main load-bearing assumptions are the smoothness and variance conditions and the unverified Nyström error and covariance bounds.

free parameters (2)
  • rank r = 10 (CIFAR), 50 and 20 (ImageNet)
    Approximation rank for the Nyström sketch; chosen by hand, controls trade-off between accuracy and compute.
  • damping factor ρ = 1.0 for NYSACT on both CIFAR and ImageNet
    Selected via grid search over [0.01, 0.05, 0.1, 0.5, 1, 5, 10] for FOOF and NYSACT (Appendix B).
assumptions (4)
  • domain assumption The loss is L-smooth (Assumption 5.1)
    Standard smoothness assumption for neural network objectives; invoked in the Taylor expansion in the proof.
  • domain assumption Stochastic gradients are unbiased with bounded variance (Assumption 5.2)
    Standard in SGD analysis; used to take expectations in the proof.
  • domain assumption Activation covariance eigenvalues are bounded away from zero and above (Assumption 5.3)
    Supported only by a toy LeNet-5 experiment (Figure 2); needed for the inverse preconditioner norm bounds.
  • ad hoc to paper The Nyström approximation error satisfies ||A - A_nys||_F ≤ ε ||A||_F for small ε (Assumption 5.4)
    Assumed without proof; the convergence rate and eigenvalue bounds depend on this unverified ε.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximation." pith.science (2026). https://pith.science/paper/CCEILMG7

@misc{pith2026250608360,
  author       = {Pith},
  title        = {Pith review of: NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CCEILMG7}},
  note         = {Machine review of arXiv:2506.08360}
}
read the original abstract

Adaptive gradient methods are computationally efficient and converge quickly, but they often suffer from poor generalization. In contrast, second-order methods enhance convergence and generalization but typically incur high computational and memory costs. In this work, we introduce NysAct, a scalable first-order gradient preconditioning method that strikes a balance between state-of-the-art first-order and second-order optimization methods. NysAct leverages an eigenvalue-shifted Nystrom method to approximate the activation covariance matrix, which is used as a preconditioning matrix, significantly reducing time and memory complexities with minimal impact on test accuracy. Our experiments show that NysAct not only achieves improved test accuracy compared to both first-order and second-order methods but also demands considerably less computational resources than existing second-order methods. Code is available at https://github.com/hseung88/nysact.

Figures

Figures reproduced from arXiv: 2506.08360 by the authors.

Figure 1
Figure 1. Comparison of exact activation covariance and Nyström-approximated activation covariance in ResNet-32 architecture [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Trajectories of the Frobenius norm of the activation covariance for LeNet-5 layers on CIFAR-10 for 100 epochs. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Comparison of training loss and test accuracy of optimizers on CIFAR-100 dataset using [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison of training loss and top-1 accuracy of optimizers on ImageNet dataset. The curves are truncated at thresholds [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Comparison of the effects of learning rate, EMA coefficient, damping, and mini-batch size on gradient preconditioning [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Comparison of wall-clock time for different inverse update frequencies during ResNet-110 training on the CIFAR-100 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Comparison of test accuracy and wall-clock time for different ranks of [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 32 canonical work pages

  1. [1]

    Efficient full-matrix adaptive regularization

    Naman Agarwal, Brian Bullins, Xinyi Chen, Elad Hazan, Karan Singh, Cyril Zhang, and Yi Zhang. Efficient full-matrix adaptive regularization. In International Conference on Machine Learning, 2020

    work page 2020

  2. [2]

    Gradient descent on neurons and its link to approximate second-order optimization

    Frederik Benzing. Gradient descent on neurons and its link to approximate second-order optimization. In International Conference on Machine Learning, 2022

    2022

  3. [3]

    C. G. Broyden. The convergence of a class of double-rank minimization algorithms 2. the new algorithm. Ima Journal of Applied Mathematics, 1970

    work page 1970

  4. [4]

    Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V . Le. Randaugment: Practical automated data augmentation with a reduced search space. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019

    work page 2019

  5. [5]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition, 2009

    work page 2009

  6. [6]

    Fletcher

    R. Fletcher. A new approach to variable metric algorithms. Comput. J., 1970

    work page 1970

  7. [7]

    Sketchysgd: reliable stochastic optimization via randomized curvature estimates

    Zachary Frangella, Pratik Rathore, Shipu Zhao, and Madeleine Udell. Sketchysgd: reliable stochastic optimization via randomized curvature estimates. arXiv, 2022

    work page 2022

  8. [8]

    Tropp, and Madeleine Udell

    Zachary Frangella, Joel A. Tropp, and Madeleine Udell. Randomized nyström preconditioning. SIAM J. Matrix Anal. Appl., 2023

    work page 2023

Show all 41 references
  1. [9]

    M-FAC: Efficient matrix-free approximations of second-order information

    Elias Frantar, Eldar Kurtic, and Dan Alistarh. M-FAC: Efficient matrix-free approximations of second-order information. In A. Beygelzimer, Y . Dauphin, P. Liang, and J. Wortman Vaughan, editors,Neural Information Processing Systems, 2021

    2021

  2. [10]

    A family of variable-metric methods derived by variational means.Mathematics of Computation, 1970

    Donald Goldfarb. A family of variable-metric methods derived by variational means.Mathematics of Computation, 1970

    1970

  3. [11]

    Practical quasi-newton methods for training deep neural networks

    Donald Goldfarb, Yi Ren, and Achraf Bahamou. Practical quasi-newton methods for training deep neural networks. In Neural Information Processing Systems, 2020

    2020

  4. [12]

    Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He

    Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. ArXiv, 2017

    2017

  5. [13]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, 2018

    2018

  6. [14]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016

    2016

  7. [15]

    Augment your batch: better training with larger batches

    Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: better training with larger batches. ArXiv, 2019. 11 NysAct: A Scalable Preconditioned Gradient Descent using Nyström Approximation

    2019

  8. [16]

    Train longer, generalize better: closing the generalization gap in large batch training of neural networks

    Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In Neural Information Processing Systems, 2017

    2017

  9. [17]

    Weinberger

    Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks.IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  10. [18]

    Weinberger

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, 2016

    2016

  11. [19]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015

    2015

  12. [20]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009

    2009

  13. [21]

    Lecun, L

    Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998

    1998

  14. [22]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training

    Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In International Conference on Learning Representations, 2024

    2024

  15. [23]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. International Conference on Learning Representations, 2016

    2016

  16. [24]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019

    2019

  17. [25]

    Optimizing neural networks with kronecker-factored approximate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International Conference on Machine Learning, 2015

    2015

  18. [26]

    Müller and Frank Hutter

    Samuel G. Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmentation.IEEE/CVF International Conference on Computer Vision, 2021

    2021

  19. [27]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  20. [28]

    Sublinear time approximation of text similarity matrices

    Archan Ray, Nicholas Monath, Andrew McCallum, and Cameron Musco. Sublinear time approximation of text similarity matrices. In AAAI Conference on Artificial Intelligence, 2021

    2021

  21. [29]

    A stochastic approximation method

    Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, 1951

    1951

  22. [30]

    David F. Shanno. Conditioning of quasi-newton methods for function minimization. Mathematics of Computation, 1970

    1970

  23. [31]

    Rethinking the inception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. IEEE Conference on Computer Vision and Pattern Recognition, 2015

    2015

  24. [32]

    Skfac: Training neural networks with faster kronecker-factored approximate curvature

    Zedong Tang, Fenlong Jiang, Maoguo Gong, Hao Li, Yue Wu, Fan Yu, Zidong Wang, and Min Wang. Skfac: Training neural networks with faster kronecker-factored approximate curvature. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021

    2021

  25. [33]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv’e J’egou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, 2020

    2020

  26. [34]

    Better SGD using second-order momentum

    Hoang Tran and Ashok Cutkosky. Better SGD using second-order momentum. In Neural Information Processing Systems, 2022

    2022

  27. [35]

    Tropp, Alp Yurtsever, Madeleine Udell, and V olkan Cevher

    Joel A. Tropp, Alp Yurtsever, Madeleine Udell, and V olkan Cevher. Fixed-rank approximation of a positive- semidefinite matrix from streaming data. In Neural Information Processing Systems, 2017

    2017

  28. [36]

    Christopher K. I. Williams and Matthias W. Seeger. Using the nyström method to speed up kernel machines. In Neural Information Processing Systems, 2000

    2000

  29. [37]

    Zhewei Yao, Amir Gholami, Sheng Shen, Kurt Keutzer, and Michael W. Mahoney. Adahessian: An adaptive second order optimizer for machine learning. In AAAI Conference on Artificial Intelligence, 2020

    2020

  30. [38]

    Cutmix: Regularization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Young Joon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. IEEE/CVF International Conference on Computer Vision, 2019. 12 NysAct: A Scalable Preconditioned Gra...

    2019

  31. [39]

    Dauphin, and David Lopez-Paz

    Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018

    2018

  32. [40]

    Eva: Practical second-order optimization with kronecker-vectorized approximation

    Lin Zhang, Shaohuai Shi, and Bo Li. Eva: Practical second-order optimization with kronecker-vectorized approximation. In International Conference on Learning Representations, 2023

    2023

  33. [41]

    Random erasing data augmentation

    Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. AAAI Conference on Artificial Intelligence, 2017. 13 NysAct: A Scalable Preconditioned Gradient Descent using Nyström Approximation A Proof of Theorem 5.5 The update rule of NYSAC...

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.