Pith. sign in

REVIEW 3 major objections 4 minor 67 references

Memory-Efficient 4-bit Preconditioned Stochastic Optimization

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read 4-bit Shampoo with compensated Cholesky quantization matches 32-bit Shampoo's accuracy at a fraction of the memory, with an $O(1/\sqrt{T})$ convergence rate for smooth nonconvex losses and stationary-point convergence for nonsmooth…

desk verdict Solid empirical recipe for 4-bit Shampoo via Cholesky quantization and error feedback, but the convergence proofs rest on a quantizer error bound that doesn't match the implemented quantizer and a positivity argument that doesn't go through. read the letter →

arxiv 2412.10663 v2 pith:6JDLNKJQ submitted 2024-12-14 cs.LG cs.CVmath.OC

Jingyang Li , Kuangyu Ding , Kim-Chuan Toh , Pan Zhou This is my paper · ORCID
classification cs.LGcs.CVmath.OC MSC 68T0790C2662L20
keywords 4-bitShampooCholeskyquantizationerrorfeedbackpreconditionedstochasticoptimizationmemory-efficienttrainingnonconvexconvergencelow-precisionoptimizer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Shampoo-style preconditioned optimizers can outperform diagonal preconditioners like Adam, but they carry memory-heavy full preconditioners that block large-scale training. This paper tries to establish that the preconditioners can be stored in 4-bit without losing the method's benefit: instead of quantizing the matrices themselves, it quantizes their Cholesky factors, whose triangular form halves storage and whose reconstruction is automatically symmetric positive definite, and it feeds the quantization error back through a cheap 4-bit error state stored in the unused upper triangle. If the claim holds, memory-hungry preconditioned optimizers become practical for billion-parameter models, including LLaMA-scale pretraining where 32-bit Shampoo runs out of GPU memory, while retaining accuracy close to the 32-bit baseline.

What carries the argument

The load-bearing object is the compensated Cholesky quantizer. Each Shampoo preconditioner $L_k$ is factored as $C_k C_k^T$; the lower-triangular Cholesky factor $C_k$ is quantized blockwise to 4-bit for off-diagonal entries while diagonals stay at 32-bit, and the quantization error is accumulated into a 4-bit error state $E_k$ that is exponentially averaged and added back to the next Cholesky factor before quantization. Because the error state is triangular with zero diagonal, it is stored in the upper-triangular half of the same matrix that holds the factor, so the scheme costs no more memory than vanilla 4-bit Shampoo. Reconstruction $D(C_k)D(C_k)^T$ is symmetric positive definite by construction, which is what keeps the inverse-fourth-root preconditioners spectrally close to the 32-bit versions; the convergence theorems then only need the reconstructed preconditioners to have bounded eigenvalues, which the paper attempts to guarantee through a diagonal-dominance argument on the unquantized inverse-root matrix.

What would settle it

Watch the condition in Proposition 5.1 during a run of Algorithm 1: for each refreshed block, check whether every row satisfies $|[M_k]_{ii}| > (1 + 2/(2^4-1)) \sum_{j\neq i} |[M_k]_{ij}|$, and measure the actual per-entry error of the linear-2 quantizer against the claimed $2^{-b}$ bound; a violation of either at any step would break the positive-definiteness assumption that the convergence theorems depend on.

Watch

Extended reading notes

Core claim

The paper's central claim is that compensated Cholesky quantization makes 4-bit Shampoo behave like 32-bit Shampoo. Experimentally, on CIFAR-100, Tiny-ImageNet, and ImageNet, the proposed 4-bit Shampoo outperforms vanilla 4-bit Shampoo and lands within a fraction of a percentage point of 32-bit Shampoo on ResNet, Swin, and ViT models, and on LLaMA pretraining it trains a 1B model that 32-bit Shampoo cannot fit, reaching perplexity 46.31 versus 48.39 for AdamW. Theoretically, the paper proves an $O(1/\sqrt{T})$ convergence rate for smooth nonconvex objectives, matching the optimal rate for first-order methods, and, for nonsmooth Whitney-stratifiable objectives such as ReLU networks, convergence of the iterates to the stationary-point set, which it describes as the first global convergence proof for preconditioned gradient descent.

Load-bearing premise

Everything rests on the assumption that the reconstructed 4-bit preconditioners stay well-behaved—symmetric, positive, and not exploding—for every training step, and the paper's guarantee of that fact assumes a row-dominance condition on the unquantized inverse-root matrix that the algorithm itself is not shown to satisfy.

Editorial extensions

If this is right

  • On ImageNet, 4-bit compensated Cholesky Shampoo reaches 78.00% accuracy on ResNet-50 and 75.01% on ViT-Base, within 0.06 and 0.46 percentage points of 32-bit Shampoo while cutting peak memory.
  • With LLaMA-1B on C4, 32-bit Shampoo exceeds the 80GB GPU memory while 4-bit Shampoo trains at perplexity 46.31, beating AdamW's 48.39.
  • The error state adds no memory beyond vanilla 4-bit Shampoo, since it is stored in the upper-triangular half of the Cholesky factor matrix.
  • For smooth nonconvex objectives, quantized Shampoo converges at the optimal $O(1/\sqrt{T})$ rate.
  • For nonsmooth stratifiable objectives such as ReLU networks, the paper claims global convergence to stationary points, a first for preconditioned gradient descent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same Cholesky-quantization-plus-error-feedback construction should transfer to other preconditioned optimizers whose preconditioners are symmetric positive definite, such as K-FAC-style block methods; the paper notes this generality only as future work.
  • Because the proof's spectrum bound hinges on diagonal dominance of the unquantized inverse-root matrix, a practical safeguard would be to monitor this condition during training and fall back to higher precision for stray blocks; the paper does not propose such a monitor.
  • The diagonal-in-32-bit, off-diagonal-in-4-bit split suggests a natural mixed-precision schedule: reserve more bits for diagonal entries when preconditioners become ill-conditioned late in training; this is a testable extension not explored in the paper.
  • Theorem 5.2's proof actually uses only bounded SPD preconditioners and Cesàro summability, so the same nonsmooth convergence argument would cover quantized variants of any preconditioned method satisfying those conditions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a memory-efficient 4-bit quantization scheme for Shampoo preconditioners. Instead of quantizing the preconditioning matrices L_k and R_k directly, the authors compute their Cholesky factors, quantize only the off-diagonal entries of those factors at 4-bit precision while keeping diagonals at 32-bit, and store an error-feedback state in the upper triangular part of the same storage. They report extensive image-classification experiments (VGG, ResNet, Swin, ViT on CIFAR-100, Tiny-ImageNet, ImageNet) and LLaMA pretraining experiments showing that the method nearly matches 32-bit Shampoo while reducing peak memory. The theoretical section claims an O(1/sqrt(T)) convergence rate for smooth nonconvex objectives and convergence to stationary points for nonsmooth stratifiable objectives, both under an assumption that the quantized preconditioner has uniformly bounded positive-definite spectrum.

Significance. If the convergence claims were fully supported, this would be a meaningful contribution: memory-efficient preconditioned optimization with a practical 4-bit implementation, a new Cholesky-quantization idea, and the first global-convergence result for quantized preconditioned gradient descent. The empirical evaluation is broad and internally consistent: Tables 3–6 cover several architectures and LLM scales, report peak memory and wall-clock time, and include ablations over the momentum parameters. The paper does not appear to fit constants to data or bake results into the derivation; the empirical claims are credible. The main weakness is the theoretical support: the positive-definiteness and bounded-spectrum assumption that underpins both convergence theorems rests on two unproven or incorrectly proven propositions. The contribution would still be of practical value if the theorems were restated as conditional on an explicit assumption, but the current text overclaims.

major comments (3)
  1. [Appendix B, Proposition B.1] Proposition B.1 claims the bound ||D(Q(x)) - x||_∞ <= ||x||_∞ / 2^b for any b-bit quantization, with the proof using a uniform grid of spacing Δ = 2/2^b. The algorithm actually uses the linear-2 mapping in Eq. (4), which is not uniform. For b = 4, the positive representable values are (i/15)^2 for odd i = 1,3,...,15, so the largest gap is 1 - (13/15)^2 = 56/225 and the worst scalar quantization error is 28/225 ≈ 0.124, which is larger than 1/16. Therefore the stated error bound fails for the implemented quantizer, and the subsequent use of the factor 2^{-b} in Proposition B.2 is not justified. The authors must either prove an analogous error bound for the linear-2 mapping (with the correct quantizer-dependent constant) or change the quantizer to one for which the bound holds.
  2. [Proposition 5.1 / Proposition B.2] The diagonal-dominance proof contains a sign error. From the Gershgorin lower bound |[M_k]_ii| - |[E_k]_ii| - Σ_{j≠i}(|[M_k]_ij| + |[E_k]_ij|), applying |[E_k]_ij| ≤ 2^{-b}|[M_k]_ij| gives (1 - 2^{-b})|[M_k]_ii| - (1 + 2^{-b})Σ_{j≠i}|[M_k]_ij|, not the stated (1 - 2^{-b})|[M_k]_ii| + (1 + 2^{-b})Σ_{j≠i}|[M_k]_ij|. The displayed expression is trivially positive and does not follow from the preceding line. A correct argument would require strict row-wise diagonal dominance of M_k with a margin depending on the quantizer, and this condition is neither derived from Algorithm 1 nor verified on the actual M_k matrices. Since Assumption 5.1c is justified by Proposition 5.1, and Assumption 5.2c is used by Theorem 5.2, both convergence theorems are currently conditional on an unproven premise.
  3. [Section 5.1, Assumption 5.1c and Figure 3] The text states that Assumption 5.1c is 'guaranteed by the implementation of the Schur–Newton method, the regularization step in Eq. (7), and Proposition 5.1.' Given the problems with Proposition B.1 and Proposition B.2, this guarantee is not established. Figure 3 only shows that eigenvalues of dequantized preconditioners are positive for VGG-19 on CIFAR-100 at a few epochs; it does not establish a uniform lower bound over all iterations, layers, and datasets. The upper bound is less problematic, but the lower-bound/positive-definiteness part must be either proved, verified exhaustively for the reported configurations, or explicitly assumed. As written, the statement that Assumption 5.1c is guaranteed is too strong.
minor comments (4)
  1. [Algorithm 1, line 2] The notation ∇L_k(W_k) appears to be a typo: the loss is defined as F(W) in Eq. (1), and there is no loss L_k defined. It should presumably be ∇F(W_k) or the corresponding mini-batch loss.
  2. [Section 4.3 and Figure 2] The joint-storage scheme places the Cholesky factor and the error state in the lower and upper triangular parts of one matrix, but the exact memory layout, the bit-width of the error state, and how the 32-bit diagonal interacts with the 4-bit off-diagonal storage are not fully specified. Since the reported memory savings in Tables 3–6 depend on this layout, a precise description would help reproducibility.
  3. [Proposition 5.1] The additive error term C_B n_k 2^{-b} I uses n_k as the number of rows of W_k, but the actual preconditioner in Eq. (14) is H_k = D(ˆR_k) ⊗ D(ˆL_k), which has dimension mn; the dependence of the bound on the matrix dimensions should be clarified or corrected.
  4. [Section 3.2] The linear-2 mapping in Eq. (4) is attributed to [15,32], but [15] is an 8-bit optimizer paper and the mapping may originate elsewhere; please ensure the credit is accurate and define b explicitly in Algorithm 1, since the algorithm is called '4-bit' but the quantizer mapping is stated for general b.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation chain; convergence proofs rest on explicit spectrum/PD assumptions, and the only self-citation is a non-load-bearing reference to standard error-feedback work.

full rationale

The paper's central derivation is not circular. The convergence theorems (5.1 and 5.2) are proven from explicitly stated Assumptions 5.1 and 5.2, using standard descent arguments and stochastic-approximation techniques. Assumption 5.1c (bounded eigenvalues of the preconditioner) is not derived from the convergence target; it is an external premise, with Proposition 5.1 offering a conditional positive-definiteness guarantee under an additional diagonal-dominance condition. That condition is not proven from the algorithm, and the quantizer error bound in Proposition B.1 is inconsistent with the nonlinear linear-2 mapping in Eq. (4), but these are mathematical correctness gaps rather than circular reductions. Similarly, Assumption 5.2c (Cesaro summability of the preconditioner sequence) is an explicit hypothesis, not a disguised form of the stationarity conclusion. No fitted parameter is renamed as a prediction, and the empirical results are compared against independent baselines rather than being forced by construction. The one self-citation, [56], is cited alongside [42,46] for the well-known error-feedback idea and is not load-bearing for any theorem or empirical claim. The central contribution—Cholesky quantization plus error feedback—stands independently of this citation. Accordingly, the paper has no significant circularity, and the score reflects only the presence of a minor, non-load-bearing self-citation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central theoretical claims rest on standard stochastic optimization assumptions plus two premises specific to this paper: a diagonal dominance condition for the quantized preconditioner that is not derived, and a quantization error bound that is proved for uniform quantization while the algorithm uses a nonlinear mapping. The experimental claims depend on several hand-chosen hyperparameters, but the ablation shows robustness to the momentum coefficients.

free parameters (5)
  • preconditioner momentum beta = 0.95
    Exponential moving average coefficient for L_k and R_k; chosen by hand and shown robust across 0.6-0.98 in ablation (Table 7).
  • error state momentum beta_e = 0.95
    EMA coefficient for quantization error states (Eq. 11); set equal to beta, affects error feedback behavior.
  • quantization block size B = 64
    Block-wise linear-2 quantization block size (Appendix C.3); balances accuracy and normalization-factor memory overhead.
  • regularization epsilon = 1e-6
    Stability constant for Cholesky and inverse 1/4-root computations (Eqs. 7 and 12).
  • update intervals T1, T2 = 100 and 500 (vision); 200 and 200 (LLM)
    Intervals for refreshing Cholesky factors and inverse roots; chosen to trade compute and memory.
assumptions (5)
  • domain assumption Smooth nonconvex loss with L-Lipschitz gradient; stochastic gradient unbiased with bounded relative variance (Assumption 5.1a,b).
    Standard stochastic optimization conditions assumed for the smooth convergence theorem.
  • domain assumption Quantized preconditioner eigenvalue bounds, sup_k lambda_max(H_k) finite and inf_k lambda_min(H_k) positive (Assumption 5.1c).
    Central to Theorem 5.1; claimed to follow from Proposition 5.1, which itself requires unproven diagonal dominance of M_k.
  • ad hoc to paper Strong row-wise diagonal dominance of M_k, |[M_k]_ii| > (1 + 2/(2^b-1)) * sum_{j != i} |[M_k]_ij| (Proposition 5.1).
    Imposed in the proposition to guarantee PSD of the quantized preconditioner; not established in general.
  • ad hoc to paper Uniform quantization error bound |Q(a) - a| <= 2^{-b} (Proposition B.1).
    Derived for uniform spacing but the algorithm uses the nonlinear linear-2 mapping (Eq. 4); used to bound the quantization error E_k.
  • domain assumption Whitney stratifiability, Lipschitz continuity, bounded noise, and Cesaro summability of H_k (Assumption 5.2a-c).
    Conditions for the nonsmooth convergence theorem (Theorem 5.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Memory-Efficient 4-bit Preconditioned Stochastic Optimization." pith.science (2026). https://pith.science/paper/6JDLNKJQ

@misc{pith2026241210663,
  author       = {Pith},
  title        = {Pith review of: Memory-Efficient 4-bit Preconditioned Stochastic Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6JDLNKJQ}},
  note         = {Machine review of arXiv:2412.10663}
}
read the original abstract

Preconditioned stochastic optimization algorithms, exemplified by Shampoo, outperform first-order optimizers by offering theoretical convergence benefits and practical gains in large-scale neural network training. However, they incur substantial memory overhead due to the storage demands of non-diagonal preconditioning matrices. To address this, we introduce 4-bit quantization for Shampoo's preconditioners. We introduce two key methods: First, we apply Cholesky decomposition followed by quantization of the Cholesky factors, reducing memory usage by leveraging their lower triangular structure while better preserving spectral properties to minimize information loss. To our knowledge, this is the first quantization approach applied to Cholesky factors of preconditioners. Second, we incorporate error feedback in the quantization process, efficiently storing Cholesky factor and error state in the lower and upper triangular parts of the same matrix. Through extensive experiments, we demonstrate that combining Cholesky quantization with error feedback enhances memory efficiency and algorithm performance in large-scale deep-learning tasks. Theoretically, we also provide convergence proofs for quantized Shampoo under both smooth and non-smooth stochastic optimization settings.

Figures

Figures reproduced from arXiv: 2412.10663 by the authors.

Figure 1
Figure 1. Comparison of test accuracy and peak memory usage for [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Efficient storage for Cholesky factor and error state. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Eigenvalue frequency of the dequantized preconditioners [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of training loss and test accuracy (%) for training ResNet-34 on CIFAR-100 and ViT-Small on Tiny-ImageNet. The [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 39 canonical work pages

  1. [1]

    Disentangling adaptive gradient methods from learning rates

    Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang. Disentangling adaptive gradient methods from learning rates. arXiv preprint arXiv:2002.11803, 2020. 5

    arXiv 2002

  2. [2]

    Qsgd: Communication-efficient sgd via gra- dient quantization and encoding

    Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan V ojnovic. Qsgd: Communication-efficient sgd via gra- dient quantization and encoding. Advances in neural infor- mation processing systems, 30, 2017. 2

    work page 2017

  3. [3]

    Scalable second order optimization for deep learning

    Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020. 1, 2, 8

    arXiv 2002

  4. [4]

    Stochas- tic approximations and differential inclusions

    Michel Bena ¨ım, Josef Hofbauer, and Sylvain Sorin. Stochas- tic approximations and differential inclusions. SIAM J. Con- trol and Optimization, 44(1):328–348, 2005. 3

    work page 2005

  5. [5]

    Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning

    J ´erˆome Bolte and Edouard Pauwels. Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning. Mathematical Programming, 188:19–51,

    work page

  6. [6]

    Stochastic approximation: a dynamical sys- tems viewpoint

    Vivek S Borkar. Stochastic approximation: a dynamical sys- tems viewpoint. Springer, 2009. 3

    work page 2009

  7. [7]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 1

    arXiv 2005

  8. [8]

    Lower bounds for finding stationary points ii: first- order methods

    Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sid- ford. Lower bounds for finding stationary points ii: first- order methods. Mathematical Programming, 185(1):315– 355, 2021. 6

    work page 2021

Show all 67 references
  1. [9]

    Optimization and nonsmooth analysis

    Frank H Clarke. Optimization and nonsmooth analysis . SIAM, 1990. 3

    1990

  2. [10]

    Autoaugment: Learning augmentation policies from data

    Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasude- van, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501, 2018. 4

    2018 arXiv

  3. [11]

    Randaugment: Practical automated data augmen- tation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmen- tation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703, 2020. 4

    2020

  4. [12]

    Pathological sub- gradient dynamics

    Aris Daniilidis and Dmitriy Drusvyatskiy. Pathological sub- gradient dynamics. SIAM Journal on Optimization , 30(2): 1327–1338, 2020. 6

    2020

  5. [13]

    Stochastic subgradient method converges on tame functions

    Damek Davis, Dmitriy Drusvyatskiy, Sham Kakade, and Ja- son D Lee. Stochastic subgradient method converges on tame functions. Foundations of Computational Mathematics , 20 (1):119–154, 2020. 6, 3, 4

    2020

  6. [14]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 6

    2009

  7. [15]

    8-bit optimizers via block-wise quantization

    Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettle- moyer. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861, 2021. 2, 3

    2021 arXiv

  8. [16]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. InInternational ...

    2020

  9. [17]

    Adaptive sub- gradient methods for online learning and stochastic opti- mization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive sub- gradient methods for online learning and stochastic opti- mization. Journal of Machine Learning Research , 12(7),

  10. [18]

    Stochastic methods for com- posite and weakly convex optimization problems

    John C Duchi and Feng Ruan. Stochastic methods for com- posite and weakly convex optimization problems. SIAM J. Optimization, 28(4):3229–3259, 2018. 3

    2018

  11. [19]

    A survey of quan- tization methods for efficient neural network inference

    Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quan- tization methods for efficient neural network inference. In Low-Power Computer Vision, pages 291–326. Chapman and Hall/CRC, 2022. 3

    2022

  12. [20]

    Practi- cal quasi-newton methods for training deep neural networks

    Donald Goldfarb, Yi Ren, and Achraf Bahamou. Practi- cal quasi-newton methods for training deep neural networks. Advances in Neural Information Processing Systems , 33: 2386–2396, 2020. 1, 2

    2020

  13. [21]

    A schur–newton method for the matrixp th root and its inverse.SIAM Journal on Matrix Analysis and Applications , 28(3):788–804, 2006

    Chun-Hua Guo and Nicholas J Higham. A schur–newton method for the matrixp th root and its inverse.SIAM Journal on Matrix Analysis and Applications , 28(3):788–804, 2006. 2

    2006

  14. [22]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In Interna- tional Conference on Machine Learning , pages 1842–1850. PMLR, 2018. 1, 2

    2018

  15. [23]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. 1, 2, 6, 4

    2016

  16. [24]

    Training compute-optimal large language mod- els

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language mod- els. arXiv preprint arXiv:2203.15556, 2022. 1

    2022 arXiv

  17. [25]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  18. [26]

    Adam: A method for stochastic opti- mization

    Diederik P Kingma. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980, 2014. 1, 2

    2014 arXiv

  19. [27]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 3, 6

    2009

  20. [28]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. Advances in neural information processing systems , 25, 2012. 4

    2012

  21. [29]

    Tiny imagenet visual recognition challenge

    Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015. 6

    2015

  22. [30]

    Biobert: a pre-trained biomedical language representation model for biomedical text mining

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240,

  23. [31]

    Vision transformer for small-size datasets

    Seung Hoon Lee, Seunghyun Lee, and Byung Cheol Song. Vision transformer for small-size datasets. arXiv preprint arXiv:2112.13492, 2021. 6, 4 9

    2021 arXiv

  24. [32]

    Memory efficient optimizers with 4-bit states

    Bingrui Li, Jianfei Chen, and Jun Zhu. Memory efficient optimizers with 4-bit states. Advances in Neural Information Processing Systems, 36, 2024. 2, 3

    2024

  25. [33]

    Relora: High-rank training through low-rank updates

    Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High-rank training through low-rank updates. arXiv preprint arXiv:2307.05695 , 2023. 6, 5

    2023 arXiv

  26. [34]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021. 1, 6, 4

    2021

  27. [35]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2019. 1, 6

    2019

  28. [36]

    Optimizing neural net- works with kronecker-factored approximate curvature

    James Martens and Roger Grosse. Optimizing neural net- works with kronecker-factored approximate curvature. InIn- ternational Conference on Machine Learning , pages 2408–

  29. [37]

    A new perspective on shampoo’s preconditioner.arXiv preprint arXiv:2406.17748,

    Depen Morwani, Itai Shapira, Nikhil Vyas, Eran Malach, Sham Kakade, and Lucas Janson. A new perspective on shampoo’s preconditioner.arXiv preprint arXiv:2406.17748,

    arXiv

  30. [38]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  31. [39]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 6

    2020

  32. [40]

    Ef21: A new, simpler, theoretically better, and practically faster error feedback

    Peter Richt ´arik, Igor Sokolov, and Ilyas Fatkhullin. Ef21: A new, simpler, theoretically better, and practically faster error feedback. Advances in Neural Information Processing Sys- tems, 34:4384–4396, 2021. 2, 4

    2021

  33. [41]

    A stochastic approxima- tion method

    Herbert Robbins and Sutton Monro. A stochastic approxima- tion method. The Annals of Mathematical Statistics , pages 400–407, 1951. 1

    1951

  34. [42]

    1-bit stochastic gradient descent and its application to data- parallel distributed training of speech dnns

    Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and its application to data- parallel distributed training of speech dnns. In Interspeech, pages 1058–1062. Singapore, 2014. 2, 4

    2014

  35. [43]

    A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale

    Hao-Jun Michael Shi, Tsung-Hsien Lee, Shintaro Iwasaki, Jose Gallego-Posada, Zhijing Li, Kaushik Rangadurai, Dheevatsa Mudigere, and Michael Rabbat. A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale. a...

    2023 arXiv

  36. [44]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 6

    2014 arXiv

  37. [45]

    On the importance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning, pages 1139–1147. PMLR, 2013. 1, 6

    2013

  38. [46]

    1-bit adam: Communication efficient large- scale training with adam’s convergence speed

    Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong He. 1-bit adam: Communication efficient large- scale training with adam’s convergence speed. In Inter- national Conference on Machine Learning , pages 10118...

    2021

  39. [47]

    Lecture 6.5- rmsprop: Divide the gradient by a running average of its re- cent magnitude

    Tijmen Tieleman, Geoffrey Hinton, et al. Lecture 6.5- rmsprop: Divide the gradient by a running average of its re- cent magnitude. COURSERA: Neural networks for machine learning, 4(2):26–31, 2012. 2

    2012

  40. [48]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1, 6

    2023 arXiv

  41. [49]

    Powersgd: Practical low-rank gradient compression for dis- tributed optimization

    Thijs V ogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for dis- tributed optimization. Advances in Neural Information Pro- cessing Systems, 32, 2019. 2

    2019

  42. [50]

    Soap: Improving and stabilizing shampoo using adam.arXiv preprint arXiv:2409.11321, 2024

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam.arXiv preprint arXiv:2409.11321, 2024. 2

    2024 arXiv

  43. [51]

    4-bit sham- poo for memory-efficient network training

    Sike Wang, Jia Li, Pan Zhou, and Hua Huang. 4-bit sham- poo for memory-efficient network training. arXiv preprint arXiv:2405.18144, 2024. 2, 3, 4

    2024 arXiv

  44. [52]

    Terngrad: Ternary gradients to reduce communication in distributed deep learning

    Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. Ad- vances in neural information processing systems , 30, 2017. 2

    2017

  45. [53]

    Resnet strikes back: An improved training procedure in timm.arXiv preprint arXiv:2110.00476, 2021

    Ross Wightman, Hugo Touvron, and Herv ´e J ´egou. Resnet strikes back: An improved training procedure in timm.arXiv preprint arXiv:2110.00476, 2021. 6

    2021 arXiv

  46. [54]

    Bloomberggpt: A large lan- guage model for finance

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large lan- guage model for finance. arXiv preprint arXiv:2303.17564,

    arXiv

  47. [55]

    Crystal graph convolu- tional neural networks for an accurate and interpretable pre- diction of material properties

    Tian Xie and Jeffrey C Grossman. Crystal graph convolu- tional neural networks for an accurate and interpretable pre- diction of material properties. Physical review letters, 120 (14):145301, 2018. 1

    2018

  48. [56]

    Loco: Low-bit communication adaptor for large-scale model training

    Xingyu Xie, Zhijie Lin, Kim-Chuan Toh, and Pan Zhou. Loco: Low-bit communication adaptor for large-scale model training. arXiv preprint arXiv:2407.04480, 2024. 2, 4

    2024 arXiv

  49. [57]

    Adan: Adaptive nesterov momentum algo- rithm for faster optimizing deep models

    Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algo- rithm for faster optimizing deep models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 1, 4

    2024

  50. [58]

    Zeroquant: Ef- ficient and affordable post-training quantization for large- scale transformers

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xi- aoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Ef- ficient and affordable post-training quantization for large- scale transformers. Advances in Neural Information Process- ing Systems, 35:27168–27183, 2022. 3

    2022

  51. [59]

    A general regret bound of preconditioned gradient method for dnn training

    Hongwei Yong, Ying Sun, and Lei Zhang. A general regret bound of preconditioned gradient method for dnn training. 10 In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7866–7875, 2023. 1, 2, 3

    2023

  52. [60]

    Cutmix: Regu- larization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu- larization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 6023–6032, 2019. 4

    2019

  53. [61]

    mixup: Beyond empirical risk minimiza- tion

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion. In International Conference on Learning Representa- tions, 2018. 4

    2018

  54. [62]

    Why are adaptive methods good for attention mod- els? Advances in Neural Information Processing Systems , 33:15383–15393, 2020

    Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention mod- els? Advances in Neural Information Processing Systems , 33:15383–15393, 2020. 1

    2020

  55. [63]

    Galore: Memory- efficient llm training by gradient low-rank projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory- efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024. 6, 5

    2024 arXiv

  56. [64]

    Random erasing data augmentation

    Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceed- ings of the AAAI conference on artificial intelligence , pages 13001–13008, 2020. 4 11 Memory-Efficient 4-bit Preconditioned Stochastic Optimization Supplementary Mate...

    2020

  57. [65]

    Definition B.2

    is defined as ∂f (x) := ( y : xk → x, ∇f (xk) → y, where f is differentiable at xk ) . Definition B.2. A locally Lipschitz function is C p-Whitney stratifiable [13], if the graph of f: graph(f ) := {(x, t) : f (x) = t} can be decomposed into finite C p manifolds, called strata...

  58. [66]

    For any two strata M1 and M2, the following implica- tion holds: M1 ∩ M2 ̸= ∅ =⇒ M1 ⊂ M2

  59. [67]

    Here NMi is the normal space of Mi

    For any sequence of points zk in a stratum M1 converg- ing to a point ¯z in a stratum M2, if the corresponding normal vectors vk ∈ NM1 (zk) converge to a vector v, then the inclusion v ∈ NM2 (¯z) holds. Here NMi is the normal space of Mi. For example, the function −|x| is a C ...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.