Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Low-precision training fails because additive steps vanish into the rounding grid at large weights; M+Adam adds a multiplicative branch that keeps progress where additive steps zero out.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:11 UTC pith:F4US5VX7

load-bearing objection The paper's central mechanism—multiplicative progress at large magnitudes—does not hold in the implemented algorithm; the stress-test note is correct. the 4 major comments →

arxiv 2607.10611 v2 pith:F4US5VX7 submitted 2026-07-12 cs.LG

M+Adam: Low-Precision Training via Additive-Multiplicative Optimization

Xiaoyuan Liang , Sebastian Loeschcke , Mads Toftrup , Anima Anandkumar This is my paper
classification cs.LG
keywords low-precision trainingadditive-multiplicative optimizationmultiplicative updatesmaster weightsBF16FP8FP4/NVFP4LLM pretraining
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper identifies why training neural networks with low-precision master weights (BF16, FP8, FP4) loses accuracy when no FP32 copy is kept: additive optimizer updates, such as Adam's, fall below the floating-point quantization step at large weight magnitudes and get rounded back to zero. It then proposes M+Adam, which adds a multiplicative (Madam-style) branch that rescales weights by a relative factor, so progress continues exactly where additive steps are lost. The two branches have complementary failure modes, and the paper proves a monotone-descent guarantee for their combination under a local smoothness bound. Across 60M–1B LLaMA-style models at 1–8× Chinchilla budgets, M+Adam consistently beats AdamW, and also beats AdamW with stochastic rounding or compensated accumulation, with the largest gains in the most aggressive FP4 setting. If the claim holds, low-precision training can drop FP32 master weights and still stay near full-precision accuracy, cutting memory and enabling cheaper end-to-end low-precision training.

Core claim

M+Adam claims that the accuracy loss in low-precision training without FP32 master weights is not primarily an arithmetic-accumulation problem but an optimizer-geometry problem: when weights are stored in BF16, FP8, or FP4, the spacing between representable values grows with |w|, so Adam-style additive updates below the local bin size are rounded to zero and training stalls at large magnitudes. The paper's fix is to compose an Adam-style additive update u^(a) with a Madam-style multiplicative update u^(m), yielding w_{t+1} = w_t + w_t u^(m) + u^(a). The multiplicative branch is derived from the gradient with respect to the exponent, g^(e) = (ln 2) w_t g_t, so it is scale-invariant and keeps

What carries the argument

The central object is the combined update w_{t+1} = w_t + w_t u^(m) + u^(a), where u^(a) is the Adam-style adaptive additive step and u^(m) is a Madam-style relative step built from the exponent-gradient g^(e) = (ln 2) w_t g_t, normalized by its own second moment, divided by ρ_t = max(|w_t|, τ), and clipped. The multiplicative gradient identity g^(e) = (ln 2) w_t g_t connects the weight-space gradient to the exponent-space gradient in base-2 floating point, so the relative update can be computed without explicitly extracting exponents. The accompanying descent theorem (Theorem 4.1) supplies conditions under which composing an additive displacement and a multiplicative rescaling decreases the

Load-bearing premise

The central claim stands on the assumption that low-precision master-weight rounding at large magnitudes is the dominant source of optimization error, and on Table 2's single best-over-sweep perplexity values (no seeds) with FP8/NVFP4 storage simulated through explicit quantize/dequantize loops; if native hardware behaves differently or the sweeps are noisy, the reported consistent gains could shrink or reverse.

What would settle it

Run the 350M FP8/FP8 and NVFP4/FP8 comparisons on hardware with native FP8/FP4 master-weight update support (no simulated quantize/dequantize loops), or repeat the BF16/BF16 sweeps with multiple seeds and report median rather than best perplexity; if M+Adam's edge over AdamW (e.g., 19.395 vs 21.674 at FP8/FP8, 20.796 vs 24.941 at NVFP4/FP8) disappears, the claimed gains are an artifact of the simulation or of single-run sweep noise.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Low-precision master weights (BF16, FP8, NVFP4) can be used without FP32 copies and without stochastic rounding or compensated accumulation; M+Adam recovers most of the perplexity gap to FP32-master training.
  • The advantage grows as weight precision coarsens: relative gains over AdamW are largest in the NVFP4/FP8 regime (e.g., 350M: 20.796 vs 24.941), consistent with the claim that coarse master-weight rounding is the dominant error source.
  • The extra multiplicative-branch second-moment state can be compressed independently (Apollo-E, rank 4) to within about 0.4–0.8 bytes per parameter of BF16-state AdamW while retaining most of M+Adam's perplexity benefit.
  • The method trains stably across model scale (60M–1B) and token budgets (1–8× Chinchilla), indicating the improvement is not an early-training artifact.
  • A monotone-descent theorem covers the idealized scale-then-add update, providing a theoretical basis for the combined geometry.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the underlying diagnosis is right, any optimizer that makes effective step size proportional to the local quantization bin at large magnitudes—not necessarily via a multiplicative branch—should narrow the gap; the key property is scale-awareness, not the specific additive-plus-multiplicative decomposition.
  • The paper keeps the two branch coefficients fixed globally; a natural extension would be to modulate the multiplicative learning rate by the block scale in block-scaled formats (the paper reports a null result for one simple version), or to make the branch mixture depend on local |w|, which could further improve the NVFP4 regime.
  • The comparison against stochastic rounding and compensated accumulation is limited to the BF16/BF16 setting; extending those baselines to FP8/NVFP4 would test whether the paper's gains are unique to the update geometry or partially shared by any principled rounding or compensation scheme.
  • If the gains persist at larger scales with native low-bit hardware kernels (rather than simulated quantize/dequantize loops), end-to-end low-precision pretraining without FP32 master weights becomes a practical memory and energy saving.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes M+Adam, an optimizer that combines an Adam-style additive branch with a Madam-style multiplicative branch for training neural networks whose master weights are stored in low precision (BF16, FP8, NVFP4). The motivation is that additive updates are rounded away at large weight magnitudes, while multiplicative updates provide relative progress there; conversely, multiplicative updates fail near zero and across sign changes, which the additive branch handles. The authors prove a local descent theorem for an idealized scale-then-add update, validate the complementary failure modes with a BF16 matrix-fitting diagnostic, and report LLaMA-style pretraining results at 60M–1B scale across several Chinchilla budgets and precision regimes, claiming consistent improvements over AdamW, AdamW+SR, and AdamW+Kahan. The central claim is that switching the optimizer geometry, rather than patching accumulation, recovers most of the accuracy lost by removing FP32 master weights.

Significance. If the empirical gains are real, this is a potentially significant step toward low-precision training without FP32 master weights: the method is conceptually simple, requires no auxiliary correction buffers, and is evaluated across a broad and relevant set of model sizes, token budgets, and precisions. The paper is commendably thorough: it releases code, includes extensive hyperparameter ablations in the appendix, reports runtime overhead, and is explicit about several limitations. However, the main mechanistic claim is currently contradicted by the algorithm as written: the multiplicative branch's normalization cancels the |w| factor that is supposed to provide scale-aware progress, and the toy diagnostic that appears to validate the mechanism actually tests an unnormalized variant. The empirical evaluation also rests on single best-over-sweep runs without seeds or error bars. The contribution's significance cannot be fully assessed until the algorithm, theory, and experiments are reconciled.

major comments (4)
  1. [§4.1, Algorithm 1] The central mechanism is not realized by the implemented multiplicative branch. In Algorithm 1, g^(e) = (ln 2) w_t g, and v^(e) is the EMA of (g^(e))^2. After a short transient, √(v^(e)) ≈ (ln 2)|w_t|·RMS(g), so the normalized proposal is g^(e)/(√(v^(e))+ϵ) ≈ sign(w_t) g / RMS(g). Dividing by ρ = max(|w_t|, τ) yields u^(m) ≈ −η_m sign(w_t) g/(RMS(g)|w_t|), and the weight-space contribution is w_t u^(m) ≈ −η_m g/RMS(g), which is independent of |w_t|. Thus the multiplicative branch does not make relative progress proportional to |w_t|, and for |w_t| large enough that η_m ≪ |w_t|·2^{−7} (the BF16 spacing), the update is rounded to zero just like an additive update. This contradicts the claimed behavior in §1, Fig. 1, and Table 1. The branch would need to omit the ρ division, or rescale differently, to deliver the advertised scale-aware progress.
  2. [Appendix B, Eq. (9) / Fig. 3(d)] The toy diagnostic that supports the large-magnitude mechanism uses a different algorithm from the main text. In Appendix B, the multiplicative update is defined as ẽu^(m)_t = −η_m g^(e)_t with no second-moment normalization, followed by division by ρ_t and clipping. The main Algorithm 1 instead normalizes by √(v^(e)) before dividing by ρ, which cancels the |w_t| factor. Consequently, Fig. 3(d) validates the intended but unimplemented mechanism, not the one in the main algorithm. The paper should either test the actual Algorithm 1 in the diagnostic or explicitly state that the diagnostic uses an unnormalized variant that is not what is trained in the LLM experiments.
  3. [§4.2, Theorem 4.1] The descent theorem is conditional on alignment and size assumptions that are not shown to hold for Algorithm 1, and the normalization analysis above shows they fail in an essential way. Eq. (5)–(6) assume ⟨g^(e), z⟩ ≤ −η_m∥g^(e)∥² and ∥z∥ ≤ η_m∥g^(e)∥. With z ≈ log(1+u^(m)) ≈ u^(m), the implemented branch gives g^(e)·z ≈ −η_m ln 2 ∥g∥²/RMS(g), while ∥g^(e)∥ scales as |w_t|∥g∥. The required inequalities can hold only with an effective η_m that depends on |w_t|, which the algorithm does not provide. Thus Theorem 4.1 does not prove descent for the implemented M+Adam, and the abstract's 'We prove monotone descent for M+Adam' overstates the result. Please either prove the alignment/size inequalities for the actual update or clearly label the theorem as an idealized sufficient condition for a family of updates not instantiated here.
  4. [Table 2, §5.2] The empirical central claim rests on single best-over-sweep validation values with no seeds or error bars. Table 2's caption states 'Each entry is the best value achieved over the corresponding hyperparameter sweep,' and the reported gaps—especially BF16/BF16 at 350M (19.156 vs. 16.607) and NVFP4 at 350M (24.941 vs. 20.796)—are single draws that could be inflated by tuning noise or selection bias. Given that the paper's title and abstract promise consistent improvements, the authors should report means and standard deviations over at least 3 independent seeds, or otherwise provide a matched evaluation protocol that controls the number of hyperparameter evaluations per method. Without this, the magnitude and consistency of the gains cannot be reliably assessed.
minor comments (5)
  1. [Algorithms 1 and 2] The two pseudocode listings are inconsistent: Algorithm 1 has no clipping of the multiplicative update, while Algorithm 2 includes clipping (lines 12, 16–19). The theoretical analysis matches neither. Please unify the pseudocode and state precisely which version is used in the experiments and which is analyzed.
  2. [Table 2 caption vs. §5.2] Table 2 says the NVFP4/FP8 setting 'uses true NVFP4 master-weight storage,' while §5.2 states that FP8/FP4 storage is 'simulated through explicit quantize/dequantize operations.' This contradiction should be resolved; if hardware support is not native, the simulation caveat should appear wherever 'true' appears.
  3. [§4.1 notation] The threshold is written τ_t in the main text, τ in Algorithm 1, and simply 'small constant τ > 0' in Algorithm 2. It is also unclear whether ρ is a scalar or per-coordinate. Please clarify the notation and the intended elementwise semantics.
  4. [Appendix I / Table 2] The appendix reports validation losses (e.g., 3.36) while Table 2 reports perplexities (e.g., 29.03). The relationship (presumably perplexity = exp(loss)) is never stated. Please add a sentence connecting the two metrics.
  5. [General] The phrase 'non-zero update at large |w|' in Table 1 is misleading if, as argued in the major comments, the implemented multiplicative update is also rounded to zero for sufficiently large |w|. The table should reflect the actual behavior of the algorithm, not the intended one.

Circularity Check

1 steps flagged

The descent theorem's alignment assumptions are the very descent property it claims to prove; the empirical central claim is independently benchmarked.

specific steps
  1. other [Section 4.2, Theorem 4.1 (Eqs. 5-6); Appendix D.1]
    "Suppose the branch outputs satisfy, for some ηa, ηm > 0, ⟨g, u(a)⟩ ≤ −ηa∥g∥2, ∥u(a)∥ ≤ ηa∥g∥, ⟨g(e), z⟩ ≤ −ηm∥g(e)∥2, ∥z∥ ≤ ηm∥g(e)∥. If ... then L(wt+1) ≤ L(wt) − ..."

    The descent bound is a direct algebraic rearrangement of the assumed branch-alignment inequalities. These inequalities are exactly the per-branch descent properties that Algorithm 1's update rule would need to satisfy, but the paper never derives them from the Adam-style moments or the ν(e) normalization. Moreover, the two conditions ⟨g,u(a)⟩ ≤ −ηa∥g∥2 and ∥u(a)∥ ≤ ηa∥g∥ force equality in Cauchy-Schwarz, so u(a) = −ηa g; similarly z = −ηm g(e). Thus the theorem applies only to plain scaled gradient descent, not to M+Adam's adaptive branches. The claimed 'monotone descent for M+Adam' is therefore an input assumption, not a property extracted from the algorithm.

full rationale

The central empirical claim — that M+Adam improves low-precision pretraining against AdamW, AdamW+SR, and AdamW+Kahan across BF16/FP8/NVFP4 — is a direct experimental comparison and is not circular. Self-citations to Madam (Bernstein et al. 2020) and LNS-Madam (Zhao et al. 2022) are background motivation, not load-bearing evidence for this paper's results. The main circularity is in the theoretical section: Theorem 4.1 assumes per-branch alignment/descent inequalities and then concludes overall descent, but it never shows that the concrete M+Adam updates satisfy these inequalities; in fact the assumptions force the updates to be plain gradient descent, so the theorem does not establish descent for the implemented optimizer. The toy diagnostic (App. B) additionally omits the ν(e) normalization used in Algorithm 1, so its large-weight success validates a different multiplicative rule; this is a limitation/correctness risk rather than a further circular step. Overall, the empirical contribution has independent content, but the theoretical 'prediction' of monotone descent reduces to its own assumptions, warranting a partial circularity score.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central empirical claim rests on standard experimental conventions (C4, LLaMA-style models) plus the authors' tuning protocol; the only non-standard assumptions introduced by the paper are the toy-diagnostic design and the theorem's local smoothness/alignment conditions. The appendix documents the tuned hyperparameters (η_a, η_m, β1, β2, ε, warmup, weight decays). No new physical or formal entities are introduced.

free parameters (5)
  • additive learning rate η_a = 8e-5 (typical; swept per regime)
    Tuned per setting on validation perplexity; controls the Adam-style additive branch.
  • multiplicative learning rate η_m = 0.004 (typical; swept per regime)
    Tuned per setting; controls the exponent-space multiplicative branch.
  • threshold τ = 1e-8
    Chosen by hand to prevent division by near-zero weights (App. B).
  • multiplicative clipping r_max = 0.75 (with ε0 margin)
    Chosen by hand to keep log(1+u^(m)) stable (App. B, Algorithm 2).
  • second-moment decay β2 = 0.98 (typical)
    Tuned along with β1, ε, warmup and weight decays; affects both branches.
axioms (4)
  • domain assumption Local coordinate smoothness bound Eq. (5) with constants La, Le, Lae
    Theorem 4.1 assumes this quadratic upper bound in (additive, exponent) coordinates; no verification for neural loss landscapes.
  • ad hoc to paper Branch outputs satisfy alignment and size inequalities (⟨g,u^(a)⟩ ≤ −η_a||g||², ||u^(a)|| ≤ η_a||g||, etc.)
    These are exactly the descent properties the proof needs; the paper does not show Algorithm 1/2's moment-based branches satisfy them.
  • domain assumption FP8/NVFP4 storage simulation via explicit quantize/dequantize is faithful to hardware
    Section 5.2 and App. E: experiments use simulated low-precision weight storage rather than native kernels; if hardware rounding differs, results may shift.
  • domain assumption Single best-over-sweep validation perplexity is a representative metric for comparing optimizers
    Main claims are based on single best-over-sweep values without seeds; noise or tuning luck could change the ranking.

pith-pipeline@v1.3.0-alltime-deepseek · 55767 in / 17662 out tokens · 171939 ms · 2026-08-02T07:11:25.171258+00:00 · methodology

0 comments
read the original abstract

Training with quantized weights can reduce costs but often results in degraded accuracy, especially when optimization is carried out in low precision, without storing high-precision copies. We identify a key failure mode: under low precision, standard optimizers can get stuck and not make progress, especially at large weight magnitudes due to coarse mantissa resolution. To overcome this, multiplicative updates have been previously proposed, in place of additive updates in standard optimizers. While successful under extremely low precision, such as under the logarithmic number system, they suffer from failures near zero and across sign changes. The failure modes of additive and multiplicative updates are therefore complementary. To exploit this, we propose M+Adam, which combines both update types: additive steps handle sign changes and small magnitudes, while multiplicative steps ensure progress at large magnitudes when additive updates are zeroed out under rounding. We prove monotone descent for M+Adam under standard smoothness assumptions. Across LLaMA-style pretraining with 60M-1B models, 1x-8x Chinchilla budgets, and using only BF16, FP8, and FP4 master weights, M+Adam consistently improves low-precision training.

Figures

Figures reproduced from arXiv: 2607.10611 by Anima Anandkumar, Mads Toftrup, Sebastian Loeschcke, Xiaoyuan Liang.

Figure 1
Figure 1. Figure 1: Additive (Adam), multiplicative (Madam), and com￾bined M+Adam updates under low precision. FP grids under low precision cause additive updates to be lost to rounding at large magnitudes and multiplicative updates to stall near zero, while M+Adam avoids both failure modes. Adam Madam M+ADAM Additive updates ✓ × ✓ Multiplicative updates × ✓ ✓ Can escape zero ✓ × ✓ Non-zero update at large |w| × ✓ ✓ [PITH_FU… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of M+Adam. (1) Compute the two gradients: the standard weight gradient g = ∇wL(wt), and gradient with respect to exponent, which evaluates as g (e) = (ln 2) wtg. (2) Compute the two updates: use gradient g to compute standard additive update u (a) and g (e) for relative multiplicative update u (m) . (3) Combine the two updates: apply the combined update wt+1 = wt + wtu (m) + u (a) to obtain the ne… view at source ↗
Figure 3
Figure 3. Figure 3: BF16-stored matrix-fitting diagnostics. Additive and Multiplicative use the optimal learning rate per task and M+Adam uses a shared learning rate across all tasks. (a) Additive and M+Adam can change signs to correct sign errors, while Multiplicative is sign-preserving. (b) Multiplicative is zero-absorbing, while Additive and M+Adam recover zero weights through the additive branch. (c) For small weights, Ad… view at source ↗
Figure 4
Figure 4. Figure 4: C4/EN perplexity scaling across model sizes and precision regimes. Validation perplexity as a function of Chinchilla ratio for 60M, 130M, and 350M LLaMA-style models. Each panel overlays tuned AdamW and M+Adam runs across BF16-weight/BF16-compute, BF16-weight/FP8-compute, and FP8-weight/FP8-compute regimes. AdamW with stochastic rounding is included in the BF16/BF16 setting. Lower perplexity is better. Hyp… view at source ↗
Figure 5
Figure 5. Figure 5: Learning-rate sweep curves for the BF16-stored matrix-fitting diagnostic. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Learning-rate sensitivity in BF16 (AdamW vs. M+Adam). Columns correspond to model sizes (60M, 130M, 350M). The top row shows AdamW validation loss versus learning rate. The bottom row shows M+Adam, where the bottom x-axis sweeps lrm and the top x-axis sweeps lre; bars correspond to the same sweep index for the paired rates. Overall, M+Adam exhibits a stable region across learning-rate choices. D. Proofs D.… view at source ↗
Figure 7
Figure 7. Figure 7: BF16 End-to-End Training Dataflow with BF16 GEMMs and BF16 Optimizer States Forward/backward use BF16 activations and BF16 weights; GEMM partial sums accumulate in FP32; gradients and optimizer states are maintained in BF16. Input BF16 (X) Output BF16 (Y ) Weight Gradient BF16 (dW) Weight BF16 (W) Optimizer States BF16 Input Gradient BF16 (dX) Output Gradient BF16 (dY ) Forward GEMM Y = XW FP8 GEMM, acc FP… view at source ↗
Figure 8
Figure 8. Figure 8: BF16 Training Dataflow with TE FP8 GEMMs and BF16 Optimizer States Activations, outputs, and gradients are stored in BF16. Transformer Engine casts GEMM operands to FP8 (delayed scaling) and accumulates partial sums in FP32 before returning BF16 results; optimizer states are maintained in BF16 and applied to update BF16 weights. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: BF16 Training with TE FP8 GEMMs and True FP8-Master Weights Activations, outputs, and backpropagated gradients are stored in BF16. For the GEMM kernels, Transformer Engine casts BF16 operands to FP8 and accumulates partial sums in FP32 before returning BF16 outputs/gradients. For selected Linear/MLP weights, the only weight value is FP8 (q) with FP16 block scales (s); a BF16 compute copy is refreshed by de… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Reference Traces for Auditing Invisible Weight Updates and Guiding Exact-Budget Protection

    cs.LG 2026-07 conditional novelty 6.5

    Weight updates that fall below half a ULP freeze coordinates deterministically, and freeze time is predictable a priori from a high-precision trajectory and mantissa length alone.

  2. Reference Traces for Auditing Invisible Weight Updates and Guiding Exact-Budget Protection

    cs.LG 2026-07 conditional novelty 5.0

    A matched high-precision reference trace forecasts invisible weight-update events in controlled low-precision runs, and a pre-frozen 10% fp32-master mask recovers ~61% of the RNE-vs-fp32 loss gap on three unseen targets.

Reference graph

Works this paper leans on

68 extracted references · 12 linked inside Pith · cited by 1 Pith paper

  1. [1]

    IEEE Transactions on Computers , volume =

    Jiawei Zhao and Steve Dai and Rangharajan Venkatesan and Brian Zimmer and Mustafa Ali and Ming. IEEE Transactions on Computers , volume =. 2022 , doi =

    2022

  2. [2]

    Aberger and Christopher De Sa , year=

    Pedram Zamirai and Jian Zhang and Christopher R. Aberger and Christopher De Sa , year=. 2010.06192 , archivePrefix=

    Pith/arXiv arXiv 2010

  3. [3]

    Proceedings of the 38th International Conference on Machine Learning , series =

    High-Performance Large-Scale Image Recognition Without Normalization , author =. Proceedings of the 38th International Conference on Machine Learning , series =. 2021 , editor =

    2021

  4. [4]

    arXiv preprint arXiv:2309.14053 , year =

    Revisiting LARS for Large Batch Training Generalization of Neural Networks , author =. arXiv preprint arXiv:2309.14053 , year =. doi:10.48550/arXiv.2309.14053 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2309.14053

  5. [5]

    Machine Learning , volume =

    Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm , author =. Machine Learning , volume =. 1988 , doi =

    1988

  6. [6]

    Information and Computation , volume =

    Exponentiated gradient versus gradient descent for linear predictors , author =. Information and Computation , volume =. 1997 , doi =

    1997

  7. [7]

    arXiv preprint arXiv:2406.00655 , year =

    Generalized Exponentiated Gradient Algorithms and Their Application to On-Line Portfolio Selection , author =. arXiv preprint arXiv:2406.00655 , year =

    Pith/arXiv arXiv

  8. [8]

    Proceedings of the 35th International Conference on Machine Learning , series =

    Adafactor: Adaptive Learning Rates with Sublinear Memory Cost , author =. Proceedings of the 35th International Conference on Machine Learning , series =. 2018 , editor =

    2018

  9. [9]

    bioRxiv , year =

    Brain-like Learning with Exponentiated Gradients , author =. bioRxiv , year =. doi:10.1101/2024.10.25.620272 , url =

    doi:10.1101/2024.10.25.620272 2024

  10. [10]

    Journal of Computer and System Sciences , volume =

    A decision-theoretic generalization of on-line learning and an application to boosting , author =. Journal of Computer and System Sciences , volume =. 1997 , doi =

    1997

  11. [11]

    Operations Research Letters , volume =

    Mirror descent and nonlinear projected subgradient methods for convex optimization , author =. Operations Research Letters , volume =. 2003 , doi =

    2003

  12. [12]

    Neurocomputing , volume =

    Multiplicative update rules for accelerating deep learning training and increasing robustness , author =. Neurocomputing , volume =. 2024 , doi =

    2024

  13. [13]

    Nishida, Keigo and K. Log. arXiv preprint arXiv:2506.17768 , year =. doi:10.48550/arXiv.2506.17768 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.17768

  14. [14]

    G. G. Turrigiano and K. R. Leslie and N. S. Desai and L. C. Rutherford and S. B. Nelson , title =. Nature , volume =

  15. [15]

    Elliott and N

    T. Elliott and N. R. Shadbolt , title =. Neural Computation , volume =

  16. [16]

    Jouppi and Doe Hyun Yoon and George Kurian and Sheng Li and Nishant Patil and James Laudon and Cliff Young and David Patterson , title =

    Norman P. Jouppi and Doe Hyun Yoon and George Kurian and Sheng Li and Nishant Patil and James Laudon and Cliff Young and David Patterson , title =. Communications of the. 2020 , volume =

    2020

  17. [17]

    Introducing NVFP4 for Efficient and Accurate Low-Precision Inference , year =

  18. [18]

    M. C. W. van Rossum and G. Q. Bi and G. G. Turrigiano , title =. The Journal of Neuroscience , volume =

  19. [19]

    You and I

    Y. You and I. Gitman and B. Ginsburg , title =. arXiv preprint arXiv:1708.03888 , year =

    Pith/arXiv arXiv

  20. [20]

    and Hseu, Jonathan and Kumar, Sanjiv and Bhojanapalli, Srinadh and Song, Xiaodan and Demmel, James and Keutzer, Kurt and Hsieh, Cho

    You, Yang and Li, Jing and Reddi, Sashank J. and Hseu, Jonathan and Kumar, Sanjiv and Bhojanapalli, Srinadh and Song, Xiaodan and Demmel, James and Keutzer, Kurt and Hsieh, Cho. Large Batch Optimization for Deep Learning: Training. International Conference on Learning Representations , year =

  21. [21]

    Bernstein and A

    J. Bernstein and A. Vahdat and Y. Yue and M.-Y. Liu , title =. Advances in Neural Information Processing Systems , volume =

  22. [22]

    Jacob and S

    B. Jacob and S. Kligys and B. Chen and M. Zhu and M. Tang and A. G. Howard and H. Adam and D. Kalenichenko , title =. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , pages =

  23. [23]

    Advances in Neural Information Processing Systems 31 (NeurIPS 2018) , year =

    Scalable Methods for 8-bit Training of Neural Networks , author =. Advances in Neural Information Processing Systems 31 (NeurIPS 2018) , year =

    2018

  24. [24]

    Zhou and Y

    S. Zhou and Y. Wu and Z. Ni and X. Zhou and H. Wen and Y. Zou , title =. arXiv preprint arXiv:1606.06160 , year =

    Pith/arXiv arXiv

  25. [25]

    Miyashita and E

    D. Miyashita and E. Lee and B. Murmann , title =. arXiv preprint arXiv:1603.01025 , year =

    Pith/arXiv arXiv

  26. [26]

    International Conference on Learning Representations , year =

    Mixed Precision Training , author =. International Conference on Learning Representations , year =

  27. [27]

    Paszke and others , title =

    A. Paszke and others , title =. Advances in Neural Information Processing Systems , volume =

  28. [28]

    D. P. Kingma and J. Ba , title =. International Conference on Learning Representations , year =

  29. [29]

    Automatic Mixed Precision package --- torch.amp , howpublished =

  30. [30]

    Courbariaux and Y

    M. Courbariaux and Y. Bengio and J.-P. David , title =. Advances in Neural Information Processing Systems , volume =

  31. [31]

    Gupta and A

    S. Gupta and A. Agrawal and K. Gopalakrishnan and P. Narayanan , title =. Proceedings of the 32nd International Conference on Machine Learning (ICML) , pages =

  32. [32]

    NeurIPS , year =

    Bernstein, Jeremy and Zhao, Jiawei and Meister, Markus and Liu, Ming-Yu and Anandkumar, Anima and Yue, Yisong , title =. NeurIPS , year =

  33. [33]

    2501.02379 , archivePrefix=

    Sebastian Loeschcke and David Pitt and Robert Joseph George and Jiawei Zhao and Cheng Luo and Yuandong Tian and Jean Kossaifi and Anima Anandkumar , year=. 2501.02379 , archivePrefix=

    Pith/arXiv arXiv

  34. [34]

    NeurIPS , year =

    Banner, Ron and Hubara, Itay and Hoffer, Elad and Soudry, Daniel , title =. NeurIPS , year =

  35. [35]

    Zhu, Hanqing and Zhang, Zhenyu and Cong, Wenyan and Liu, Xi and Park, Sem and Chandra, Vikas and Long, Bo and Pan, David Z and Wang, Zhangyang and Lee, Jinwon , journal=

  36. [36]

    Advances in Neural Information Processing Systems , year =

    Post-training 4-bit quantization of convolution networks for rapid-deployment , author =. Advances in Neural Information Processing Systems , year =

  37. [37]

    Journal of Machine Learning Research , volume =

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , author =. Journal of Machine Learning Research , volume =. 2020 , url =

    2020

  38. [38]

    arXiv preprint arXiv:2302.13971 , year =

    LLaMA: Open and Efficient Foundation Language Models , author =. arXiv preprint arXiv:2302.13971 , year =

    Pith/arXiv arXiv

  39. [39]

    Advances in Neural Information Processing Systems 35 (NeurIPS 2022) , year =

    Training Compute-Optimal Large Language Models , author =. Advances in Neural Information Processing Systems 35 (NeurIPS 2022) , year =

    2022

  40. [40]

    2022 , eprint=

    FP8 Formats for Deep Learning , author=. 2022 , eprint=

    2022

  41. [41]

    TorchAO Documentation , howpublished =

  42. [42]

    , title =

    Ibata, Keita and Sun, Qingxiu and Turrigiano, Gina G. , title =. Neuron , year =

  43. [43]

    International Conference on Learning Representations , year=

    8-bit Optimizers via Block-wise Quantization , author=. International Conference on Learning Representations , year=

  44. [44]

    2024 , publisher=

    Jiawei Zhao and Zhenyu Zhang and Beidi Chen and Zhangyang Wang and Anima Anandkumar and Yuandong Tian , booktitle=. 2024 , publisher=

    2024

  45. [45]

    CoRR , volume =

    Jesse Dodge and Maarten Sap and Ana Marasovic and William Agnew and Gabriel Ilharco and Dirk Groeneveld and Matt Gardner , title =. CoRR , volume =. 2021 , url =. 2104.08758 , timestamp =

    Pith/arXiv arXiv 2021

  46. [46]

    and Fox, Kevin and Gerstner, Wulfram and Haydon, Philip G

    Keck, Tara and Toyoizumi, Taro and Chen, Lu and Doiron, Brent and Feldman, Daniel E. and Fox, Kevin and Gerstner, Wulfram and Haydon, Philip G. and Hübener, Mark and Lee, Hey-Kyoung and Lisman, John and Rose, Tobias and Sengpiel, Frank and Stellwagen, David and Stryker, Michael P. and Turrigiano, Gina and van Rossum, Mark C. W. , title =. Philosophical Tr...

  47. [47]

    Current Opinion in Neurobiology , year =

    Zenke, Friedemann and Gerstner, Wulfram and Ganguli, Surya , title =. Current Opinion in Neurobiology , year =

  48. [48]

    , title =

    Turrigiano, Gina G. , title =. Cold Spring Harbor Perspectives in Biology , year =. doi:10.1101/cshperspect.a005736 , pmid =

    doi:10.1101/cshperspect.a005736

  49. [49]

    DeepGEMM: Clean and Efficient FP8 GEMM Kernels with Fine-Grained Scaling , year =

  50. [50]

    International Conference on Learning Representations , year =

    Fantastic Pretraining Optimizers and Where to Find Them , author =. International Conference on Learning Representations , year =

  51. [51]

    Advances in Neural Information Processing Systems , volume=

    Loeschcke, Sebastian and Toftrup, Mads and Kastoryano, Michael and Belongie, Serge and Sn. Advances in Neural Information Processing Systems , volume=

  52. [52]

    Proceedings of the 32nd International Conference on Machine Learning , series =

    Deep Learning with Limited Numerical Precision , author =. Proceedings of the 32nd International Conference on Machine Learning , series =. 2015 , publisher =

    2015

  53. [53]

    SIAM Journal on Scientific Computing , volume =

    Stochastic Rounding and Its Probabilistic Backward Error Analysis , author =. SIAM Journal on Scientific Computing , volume =

  54. [54]

    Stochastic Rounding for

    Kaan Ozkara and Tao Yu and Youngsuk Park , booktitle=. Stochastic Rounding for. 2025 , publisher=

    2025

  55. [55]

    and Panferov, Andrei and Tabesh, Soroush and Sieberling, Oliver and Chen, Jiale and Nikdan, Mahdi and Ashkboos, Saleh and Alistarh, Dan , booktitle =

    Castro, Roberto L. and Panferov, Andrei and Tabesh, Soroush and Sieberling, Oliver and Chen, Jiale and Nikdan, Mahdi and Ashkboos, Saleh and Alistarh, Dan , booktitle =. Quartet: Native. 2025 , url =

    2025

  56. [56]

    2023 , eprint=

    PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel , author=. 2023 , eprint=

    2023

  57. [57]

    2020 , eprint=

    ZeRO: Memory Optimizations Toward Training Trillion Parameter Models , author=. 2020 , eprint=

    2020

  58. [58]

    2026 , howpublished =

    2026

  59. [59]

    2026 , howpublished =

    Getting Started with. 2026 , howpublished =

    2026

  60. [60]

    Proceedings of the 26th

    Rasley, Jeff and Rajbhandari, Samyam and Ruwase, Olatunji and He, Yuxiong , title =. Proceedings of the 26th. 2020 , doi =

    2020

  61. [61]

    2403.03507 , archivePrefix =

    Zhao, Jiawei and Zhang, Zhenyu and Chen, Beidi and Wang, Zhangyang and Anandkumar, Anima and Tian, Yuandong , year =. 2403.03507 , archivePrefix =

    Pith/arXiv arXiv

  62. [62]

    2502.16982 , archivePrefix =

    Liu, Jingyuan and Su, Jianlin and Yao, Xingcheng and Jiang, Zhejun and Lai, Guokun and Du, Yulun and Qin, Yidao and Xu, Weixin and Lu, Enzhe and Yan, Junjie and Chen, Yanru and Zheng, Huabin and Liu, Yibo and Liu, Shaowei and Yin, Bohong and He, Weiran and Zhu, Han and Wang, Yuzhi and Wang, Jianzhou and Dong, Mengnan and Zhang, Zheng and Kang, Yongsheng a...

    Pith/arXiv arXiv

  63. [63]

    2024 , publisher =

    Zhao, Jiawei and Zhang, Zhenyu and Chen, Beidi and Wang, Zhangyang and Anandkumar, Anima and Tian, Yuandong , booktitle =. 2024 , publisher =

    2024

  64. [64]

    Fishman, Maxim and Chmiel, Brian and Banner, Ron and Soudry, Daniel , booktitle =. Scaling. 2025 , url =

    2025

  65. [65]

    Advances in Neural Information Processing Systems , volume =

    Memory Efficient Optimizers with 4-bit States , author =. Advances in Neural Information Processing Systems , volume =. 2023 , publisher =

    2023

  66. [66]

    2025 , eprint =

    Benchmarking Optimizers for Large Language Model Pretraining , author =. 2025 , eprint =

    2025

  67. [67]

    International Conference on Learning Representations , year =

    A Convergence Analysis of Adaptive Optimizers under Floating-point Quantization , author =. International Conference on Learning Representations , year =

  68. [68]

    International Conference on Learning Representations , year =

    Decoupled Weight Decay Regularization , author =. International Conference on Learning Representations , year =