Pith. sign in

REVIEW 2 major objections 6 minor 43 references

Gefen cuts AdamW optimizer memory by about 8x while matching its training quality, by sharing second moments in auto-found blocks and quantizing first moments with an exact learned codebook.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 17:59 UTC pith:DHJX7BES

load-bearing objection Solid systems drop-in: automatic block sharing plus exact DP first-moment quantization gives a real ~8× AdamW-state cut with matched curves and measured throughput gains; the frozen first-step partition is the main heuristic, not a collapse. the 2 major comments →

arxiv 2606.13894 v2 pith:DHJX7BES submitted 2026-06-11 cs.LG cs.AIcs.CLcs.CV

Gefen: Optimized Stochastic Optimizer

Nadav Benedek , Tomer Koren , Ohad Fried This is my paper
classification cs.LG cs.AIcs.CLcs.CV
keywords memory-efficient optimizersAdamWsecond-moment sharingfirst-moment quantizationautomatic block partitioningdynamic-programming codebookHessian-aligned parametersthroughput
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

AdamW stores a first-moment and a second-moment value for every parameter, roughly doubling the memory cost of large-model training. Gefen claims you can drop that extra state to about one-eighth of AdamW’s size without losing validation quality. It does so by automatically grouping parameters whose squared gradients look similar after the first step, then sharing one second-moment scalar inside each group and quantizing the first-moment buffer with an exact dynamic-programming codebook that reuses the same groups. A short theoretical argument motivates the grouping: when two weights have a large mixed Hessian entry, their squared-gradient ratio is forced toward one, so they are natural candidates for a shared second moment. Because full Hessians are impractical, the method simply measures within-block variance of the first-step squared gradients over candidate periods and picks the first sharp drop. Across language and vision pretraining the resulting optimizer matches AdamW loss and accuracy, records the lowest peak optimizer memory among the compared methods, and frees enough space for larger microbatches that raise end-to-end throughput.

Core claim

A fully automatic, architecture-agnostic partition of each parameter tensor—chosen once from the first-step squared-gradient variance curve—plus an exact histogram-based dynamic-programming codebook for the first moment, reduces AdamW’s optimizer-state footprint by roughly eight times while preserving AdamW-level performance on language and vision tasks and enabling larger microbatches.

What carries the argument

Automatic block partitioning (Algorithm 2) that selects a period p from the first-step squared-gradient variance curve E(p), justified by Theorem 4.1 showing that large mixed Hessian entries contract squared-gradient ratios toward one; the same blocks are reused both for shared second-moment scalars and for exact DP first-moment quantization.

Load-bearing premise

The single period chosen from the first-step squared-gradient variance curve stays a good Hessian-aligned grouping for the whole training run, so one shared second-moment scalar per block never drifts far from full AdamW.

What would settle it

Re-run the Llama-3-1.5B or GPT-2-125M pretraining while recomputing the block period every few thousand steps from fresh squared gradients; if final validation loss or peak memory then diverges from the fixed-partition Gefen curve, the once-and-for-all partition claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. Gefen is a drop-in AdamW variant that reduces optimizer-state memory by ~8× (to ~0.25× model size) by (i) automatically partitioning each parameter tensor into blocks from the first-step squared-gradient variance curve E(p) and sharing one second-moment scalar per block, and (ii) quantizing the first moment with an exact histogram-based dynamic-programming codebook that reuses the same blocks and forces endpoints at ±1. The design is motivated by a contraction theorem for a two-layer MLP: large mixed Hessian entries force the ratio of squared gradients toward 1, so Hessian-aligned groups are natural candidates for shared second moments; at scale the Hessian is replaced by the first-step E(p) proxy. Empirically, on GPT-2-125M/OpenWebText, Llama-3-1.5B/C4, ResNet-18/101/ImageNet and a CNN/MNIST, Gefen matches AdamW validation curves under identical learning rates and defaults, reports the lowest peak optimizer memory among compared AdamW-like methods, and converts the memory savings into larger microbatches that raise measured FSDP/DDP throughput (e.g., +56% FSDP on Llama-3-1.5B).

Significance. If the empirical claims hold, Gefen is a practically useful systems contribution: an architecture-agnostic, hyperparameter-light AdamW replacement that frees ~6.5 GiB per billion parameters and can raise end-to-end throughput or enable larger models/batches under fixed GPU memory. Strengths that raise confidence include a complete public implementation with fused CUDA kernels, an exact (histogram) DP codebook rather than a hand-designed quantizer, matched-hyperparameter curves across both language and vision, and explicit peak vs. persistent memory tables plus distributed throughput measurements. The two-layer-MLP contraction theorem and small-model Hessian scatter plots supply independent motivation rather than circular evaluation. The main open risk is that the first-step partition is frozen for the whole run; that is a heuristic leap from the theory, but final AdamW-matched performance already constrains material late-training distortion within the reported regimes.

major comments (2)
  1. §5.1 / Algorithm 2: The automatic period p⋆ is chosen once from the first-step E(p) curve and never recomputed. The paper shows codebook stability (Fig. 5) but not partition stability. Because the central claim is that Hessian-aligned sharing introduces negligible distortion relative to full AdamW, please add either (a) an ablation that periodically re-selects p⋆ (or freezes a late-training partition) and reports validation curves, or (b) a short analysis of E(p) at later steps on at least one LLM and one vision model. Without this, the bridge from Theorem 4.1 / App. B to long runs remains an untested assumption, even though final performance already bounds the practical damage in the reported settings.
  2. §2 and §5 (desiderata / “no additional hyperparameters”): Algorithm 2 hard-codes Δ⋆ = 10^{-12} and rejects periods p⋆ < 8; Algorithm 3 uses a 16k-bin histogram and a fixed codebook size k (8-bit in the main results). These are free design choices, not AdamW defaults. Please either (i) state them explicitly as fixed defaults of the method and show that performance is insensitive to reasonable variations (e.g., min period ∈ {4,8,16}, k ∈ {4,8}), or (ii) soften the “no additional hyperparameters” claim to “no architecture-specific metadata or per-run tuning beyond AdamW defaults.” The current wording overstates plug-and-play purity relative to the algorithm as written.
minor comments (6)
  1. Table 1: Peak memory for Adam-mini and Adafactor exceeds persistent memory (temporary allocations). Briefly confirm in the caption or §6.1 that Gefen’s peak figures include all step-time intermediates under the fused CUDA path, so the comparison is apples-to-apples.
  2. Fig. 2 / §6.2: For SM3 and Adafactor the learning rates differ from AdamW (and SM3 required a manual reduction to avoid divergence). A short note that these are not drop-in under the same LR would make the “AdamW-like” vs. “non-AdamW-like” distinction clearer.
  3. Theorem 4.1 / App. A: The bound is stated for k ≠ k′; the text notes the same phenomenon is seen for k = k′ empirically. A one-sentence remark on whether the proof extends when the Kronecker term is present would help readers.
  4. §7 / Fig. 4: The 4-bit degradation is useful; consider adding the corresponding peak-memory number so readers can see the quality–memory trade-off quantitatively.
  5. Typos / polish: title line “OPTIMIZEDSTOCHASTICOPTIMIZER” lacks spaces; “performanceto” and similar missing spaces appear in §2; arXiv id and GitHub link are fine but ensure the camera-ready abstract matches the body on the 6.5 GiB/B figure.
  6. Related work (§3, App. E): A brief comparison to Apollo and Sophia on the “manual name-based rules / extra hypers” axis is already present; adding one sentence on whether Gefen’s blocks could compose with 1-bit Adam communication compression would complete the orthogonality discussion.

Circularity Check

0 steps flagged

No significant circularity; theory motivates block sharing independently of the empirical AdamW-matching claim, and period/codebook choices are frozen from first-step data rather than fitted to final metrics.

full rationale

The paper's central claim is empirical and systems-oriented: Gefen matches AdamW validation curves (and improves throughput via larger microbatches) on public datasets/models while cutting optimizer-state memory to ~0.25 imes model size. Theorem 4.1 derives a contraction of squared-gradient ratios under large mixed Hessian entries for a two-layer MLP (with explicit assumptions on D, U, V, C); Appendix B supplies independent small-model scatter plots confirming the same qualitative pattern. Algorithm 2 then selects a single period p* from the first-step E(p) variance curve and freezes it; Algorithm 3 learns an exact histogram DP codebook once from the same first-step gradients. Neither quantity is re-optimized against validation loss, so the final performance match is not forced by construction. There are no load-bearing self-citations, uniqueness theorems imported from the authors, or renamings of known results that close a definitional loop. The residual heuristic (that the first-step partition remains adequate for the whole run) is a genuine modeling assumption, not a circular reduction of a claimed prediction to its inputs. Evaluations use external AdamW baselines under identical hyperparameters, rendering the derivation chain self-contained against the reported evidence.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The method rests on standard AdamW update equations, a new but limited Hessian-ratio lemma for a two-layer MLP, and two algorithmic design choices (first-step period selection and exact DP quantization) whose free parameters are few and mostly fixed by defaults. No new physical entities are postulated; the invented objects are purely algorithmic.

free parameters (3)
  • minimum accepted period p⋆ = 8
    Algorithm 2 falls back to p=1 if the selected period is smaller than 8; the threshold 8 is a hand-chosen memory-vs-quality cutoff not derived from the theory.
  • histogram bin count (16k) and codebook size k = 16k bins, k=256 (8-bit)
    Exact DP operates on a 16k-bin histogram over [−1,1] and produces a k-level codebook (default 8-bit). Both are implementation choices that affect quantization error and memory.
  • drop threshold Δ⋆ initialization = 1e-12
    The “largest drop” search initializes Δ⋆ = 10^{-12}; the numerical floor is arbitrary though practically inert.
axioms (4)
  • domain assumption AdamW moment updates, bias correction and decoupled weight decay are the correct baseline dynamics.
    All comparisons and the Gefen update (Algorithm 1) inherit the standard AdamW equations without re-derivation.
  • standard math For a two-layer MLP with MSE loss, large mixed Hessian magnitude contracts the admissible range of the squared-gradient ratio toward 1 (Theorem 4.1 / A.1).
    Proved under explicit boundedness assumptions on D, U, V, C; used only as motivation for sharing.
  • ad hoc to paper First-step squared-gradient variance across candidate periods is a sufficient proxy for Hessian block structure throughout training.
    Stated in §5.1; never recomputed; justified only by final performance parity.
  • ad hoc to paper Forcing codebook extremes to −1 and +1 prevents scale contraction under per-step absmax re-scaling.
    Design choice in §5.2.1 to avoid the feedback loop observed with plain Lloyd-Max.
invented entities (2)
  • Automatic period selection via successive E(p) drops (Algorithm 2) no independent evidence
    purpose: Infer Hessian-aligned blocks from first-step gradients without architecture metadata.
    Core algorithmic novelty; independent evidence is only the empirical match to AdamW, not an external falsifiable prediction.
  • Exact histogram DP quantization codebook with forced endpoints (Algorithm 3) independent evidence
    purpose: Obtain an optimal 1-D codebook for first-moment quantization that re-uses the same blocks.
    Guarantees optimality for the histogram; independent of final model quality, but the decision to freeze it after step 0 is paper-specific.

pith-pipeline@v1.1.0-grok45 · 25604 in / 3162 out tokens · 32078 ms · 2026-07-14T17:59:51.337051+00:00 · methodology

0 comments
read the original abstract

AdamW is a default optimizer for modern deep learning, but its first and second moment states add roughly two parameter-sized buffers to training memory, increasing the already substantial cost of large-scale pretraining. We propose Gefen, a memory-efficient optimizer that automatically shares second-moment estimates across parameter blocks and quantizes the first moment using a learned codebook, thereby reducing AdamW's memory footprint by ~8x while maintaining the same performance, corresponding to a reduction of 6.5 GiB per billion parameters. The method is motivated by a theoretical result showing that large mixed Hessian entries constrain the ratio of squared gradients toward one, suggesting that Hessian-aligned parameters are natural candidates for sharing second-moment statistics. Since computing Hessians is impractical at scale, Gefen infers block structure from the initial squared gradients, requiring no architecture-specific metadata or hyperparameters beyond AdamW defaults. Gefen learns an exact histogram-based dynamic-programming quantization codebook and reuses the same blocks for first-moment scaling. Across diverse pretraining experiments, Gefen achieves the lowest peak optimizer memory among the compared AdamW-like methods while maintaining AdamW-level performance. In single-machine or distributed training, the reduced memory footprint enables larger microbatches and improves throughput significantly over AdamW, providing a practical drop-in replacement with lower memory usage that can increase throughput and enable training larger models or using larger global batch sizes. We provide the complete Python implementation, including fused CUDA kernels at https://github.com/ndvbd/Gefen

Figures

Figures reproduced from arXiv: 2606.13894 by Nadav Benedek, Ohad Fried, Tomer Koren.

Figure 1
Figure 1. Figure 1: Hessian of a toy trans￾former attention-query parameter, showing block-diagonal structure, and the corresponding squared gra￾dients of the same parameter below. Qualitatively, both exhibit a visible 16-block structure. For each parameter tensor, Gefen infers the sharing pattern from the first iteration gradient only. Let g0 ∈ R n be the flattened gradient of a parameter tensor at the first optimiza￾tion st… view at source ↗
Figure 3
Figure 3. Figure 3: Training throughput in two distributed settings. (a) With FSDP, Gefen increases through￾put by 56%. (b) For GPT2-1B, Gefen enables training where AdamW is infeasible and improves throughput by 21% over Adam-mini; similar be￾havior is observed in single-machine training. We next evaluate whether the reduced mem￾ory footprint translates into practical through￾put gains and whether the method is compat￾ible w… view at source ↗
Figure 2
Figure 2. Figure 2: Stable Quantization. A natural question for the ablation study is whether it is sufficient to learn the quantization codebook only at the beginning of training [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: The top two panels show histograms of gradient values after block-wise absmax normal [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Left: qualitative comparison of optimizer desiderata. Middle: peak optimizer memory rel [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Squared-gradient ratio on a log scale vs. Hessian for GPT-toy at the beginning of training: (a) first-layer query matrix; (b) fourth￾layer value matrix. As the Hessian entry moves away from zero, the squared-gradient ratio ap￾proaches one, as predicted by the theorem. 0.02 0.00 0.02 0.04 0.06 Hessian entry H[i,j] 10 6 10 4 10 2 10 0 10 2 10 4 10 6 grad_sq[i] / grad_sq[j] all pairs (a) 0.010 0.005 0.000 0.0… view at source ↗
Figure 9
Figure 9. Figure 9: Comparison of gradient quantization methods. The top panels show the gradient distribu [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 8 linked inside Pith

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Np-hardness of euclidean sum-of-squares clustering

    Daniel Aloise, Amit Deshpande, Pierre Hansen, and Preyas Popat. Np-hardness of euclidean sum-of-squares clustering. Machine learning, 75 0 (2): 0 245--248, 2009

    2009

  3. [3]

    Memory efficient adaptive optimization

    Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory efficient adaptive optimization. Advances in Neural Information Processing Systems, 32, 2019

    2019

  4. [4]

    An iterative algorithm for computing the best estimate of an orthogonal matrix

    ke Bj \"o rck and Clary Bowie. An iterative algorithm for computing the best estimate of an orthogonal matrix. SIAM Journal on Numerical Analysis, 8 0 (2): 0 358--364, 1971

    1971

  5. [5]

    Symbolic discovery of optimization algorithms

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36: 0 49205--49233, 2023

    2023

  6. [6]

    Large scale machine learning

    Ronan Collobert. Large scale machine learning. 2004

    2004

  7. [7]

    Random projection trees for vector quantization

    Sanjoy Dasgupta and Yoav Freund. Random projection trees for vector quantization. IEEE Transactions on Information Theory, 55 0 (7): 0 3229--3242, 2009

    2009

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. IEEE, 2009

    2009

  9. [9]

    8-bit optimizers via block-wise quantization

    Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861, 2021

    Pith/arXiv arXiv 2021

  10. [10]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12 0 (7), 2011

    2011

  11. [11]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  12. [12]

    Functions of matrices: theory and computation

    Nicholas J Higham. Functions of matrices: theory and computation. SIAM, 2008

    2008

  13. [13]

    Gpipe: Efficient training of giant neural networks using pipeline parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019

    2019

  14. [14]

    Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization

    Shuoran Jiang, Qingcai Chen, Youcheng Pan, Yang Xiang, Yukang Lin, Xiangping Wu, Chuanyi Liu, and Xiaobao Song. Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 18363--18371, 2024

    2024

  15. [15]

    Muon: An optimizer for hidden layers in neural networks

    Keller Jordan. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/, 2024. Blog post

    2024

  16. [16]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    Pith/arXiv arXiv 2014

  17. [17]

    Numba: A llvm-based python jit compiler

    Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert. Numba: A llvm-based python jit compiler. In Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC, pp.\ 1--6, 2015

    2015

  18. [18]

    Handbuch der Lehre von der Verteilung der Primazahlen, volume 1

    Edmund Landau. Handbuch der Lehre von der Verteilung der Primazahlen, volume 1. BG Teubner, 1909

    1909

  19. [19]

    Memory efficient optimizers with 4-bit states

    Bingrui Li, Jianfei Chen, and Jun Zhu. Memory efficient optimizers with 4-bit states. Advances in Neural Information Processing Systems, 36: 0 15136--15171, 2023

    2023

  20. [20]

    Pytorch distributed: Experiences on accelerating data parallel training

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704, 2020

    Pith/arXiv arXiv 2006

  21. [21]

    Torchtitan: One-stop pytorch native solution for production ready LLM pretraining

    Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Purandare, Gokul Nadathur, and Stratos Idreos. Torchtitan: One-stop pytorch native solution for production ready LLM pretraining. In The Thirteenth International Conference on Learning Representations, 2025. URL http...

    2025

  22. [22]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training

    Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In International Conference on Learning Representations, volume 2024, pp.\ 1621--1650, 2024

    2024

  23. [23]

    Sparse mezo: Less parameters for better performance in zeroth-order llm fine-tuning

    Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse mezo: Less parameters for better performance in zeroth-order llm fine-tuning. Advances in Neural Information Processing Systems, 38: 0 21139--21169, 2026

    2026

  24. [24]

    Least squares quantization in pcm

    Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory, 28 0 (2): 0 129--137, 1982

    1982

  25. [25]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  26. [26]

    The planar k-means problem is np-hard

    Meena Mahajan, Prajakta Nimbhorkar, and Kasturi Varadarajan. The planar k-means problem is np-hard. Theoretical computer science, 442: 0 13--21, 2012

    2012

  27. [27]

    Fine-tuning language models with just forward passes

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36: 0 53038--53075, 2023

    2023

  28. [28]

    Quantizing for minimum distortion

    Joel Max. Quantizing for minimum distortion. IRE Transactions on Information Theory, 6 0 (1): 0 7--12, 1960

    1960

  29. [29]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  30. [30]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: international conference for high performance computing, networking, storage and analysis, pp.\ 1--16. IEEE, 2020

    2020

  31. [31]

    On the convergence of adam and beyond

    Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237, 2019

    Pith/arXiv arXiv 1904

  32. [32]

    Adafactor: Adaptive learning rates with sublinear memory cost

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International conference on machine learning, pp.\ 4596--4604. PMLR, 2018

    2018

  33. [33]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    Pith/arXiv arXiv 1909

  34. [34]

    Multivariate stochastic approximation using a simultaneous perturbation gradient approximation

    James C Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions on automatic control, 37 0 (3): 0 332--341, 2002

    2002

  35. [35]

    1-bit adam: Communication efficient large-scale training with adam’s convergence speed

    Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong He. 1-bit adam: Communication efficient large-scale training with adam’s convergence speed. In International Conference on Machine Learning, pp.\ 10118--10129. PMLR, 2021

    2021

  36. [36]

    Lecture 6.5--- RMSProp : Divide the gradient by a running average of its recent magnitude

    Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5--- RMSProp : Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012

    2012

  37. [37]

    Haizhou Wang and Mingzhou Song. Ckmeans. 1d. dp: optimal k-means clustering in one dimension by dynamic programming. The R journal, 3 0 (2): 0 29, 2011

    2011

  38. [38]

    Adam-mini: Use fewer learning rates to gain more

    Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik Durk Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. In International Conference on Learning Representations, volume 2025, pp.\ 28033--28063, 2025

    2025

  39. [39]

    Pytorch fsdp: experiences on scaling fully sharded data parallel

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023

    Pith/arXiv arXiv 2023

  40. [40]

    Pan, Zhangyang Wang, and Jinwon Lee

    Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z. Pan, Zhangyang Wang, and Jinwon Lee. Apollo: Sgd-like memory, adamw-level performance, 2024. URL https://arxiv.org/abs/2412.05270

    Pith/arXiv arXiv 2024

  41. [41]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  42. [42]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  43. [43]

    attn" and

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 2026