Pith. sign in

REVIEW 3 major objections 5 minor 36 references

Stable end-to-end NVFP4 pretraining of LLMs is achievable by fixing three module-specific numerical failure modes, reaching a 1.47% loss gap to BF16 on a 3B model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 19:16 UTC pith:B6DMMX2E

load-bearing objection Solid modular full-stack NVFP4 recipe with clean ablations; the 1.47% claim is real under fake quant but still needs hardware and scale confirmation. the 3 major comments →

arxiv 2607.04422 v1 pith:B6DMMX2E submitted 2026-07-05 cs.LG cs.AI

Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention

Siyu Ding , Mingchuan Ma , Jiabo Tong , Xingrun Xing , Ziming Wang , Guoqi Li This is my paper
classification cs.LG cs.AI
keywords NVFP44-bit pretrainingLLM trainingLoRA-SVDquantized optimizersmixed-precision attentionMuonRoot optimizer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Four-bit pretraining has so far been limited to transformer linear layers because optimizer states, optimizer arithmetic, and attention each break under NVFP4 in different ways. Linear layers hit a hard quantization-noise ceiling that grows with hidden dimension; AdamW second moments are heavy-tailed, non-negative, and fragile as denominators; attention has error-sensitive paths that require strict forward-backward consistency. Full-Stack FP4 supplies a tailored fix for each module: LoRA-SVD dilutes noise in projections and lowers the linear-only gap from 1.40% to 0.61%; a transformed-state pipeline plus native NVFP4 Newton–Schulz stabilizes AdamW and Root; mixed-precision attention quantizes Q/K/V and dS while keeping the fragile PV and dOV^T paths in BF16. On a 3B model trained for 64B tokens with every block quantized and no BF16 blocks reserved, the full recipe stays within 1.47% of a BF16 baseline. A sympathetic reader cares because this would turn native 4-bit training on modern GPUs from a partial experiment into a practical, near-lossless option for the whole pretraining stack.

Core claim

Stable full-stack NVFP4 pretraining is feasible. By addressing three distinct numerical failure modes with module-wise precision policies—LoRA-SVD for linear projections, transformed second-moment storage and native NVFP4 Newton–Schulz for optimizers, and mixed-precision attention that quantizes Q/K/V and dS while protecting sensitive paths—the authors obtain a 1.47% training-loss gap to BF16 on a 3B model for 64B tokens, with every transformer block quantized.

What carries the argument

LoRA-SVD: a lightweight structural decomposition that keeps a low-rank principal subspace (and activations) in BF16 while computing the residual weight in NVFP4, with a subspace-constrained backward pass and periodic randomized SVD realignment that systematically dilutes quantization noise. It is paired with a four-step transform for AdamW’s second moment, shape-optimized native NVFP4 Newton–Schulz for Root, and mixed-precision attention that reuses quantized Q/K/V for forward-backward consistency.

Load-bearing premise

The claim that stable full-stack NVFP4 pretraining is achievable rests on fake-quantization simulations of a 3B model for 64B tokens—well below Chinchilla scale—being representative of real NVFP4 Tensor Cores and larger models.

What would settle it

Train the same 3B architecture end-to-end with native NVFP4 Tensor Cores, or a larger model to Chinchilla scale, and check whether final loss still stays within roughly 1.5% of an identical BF16 run; a substantially larger gap or divergence would falsify the claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Linear-only NVFP4 can be brought from a 1.40% to a 0.61% loss gap via LoRA-SVD alone.
  • Optimizer states and Newton–Schulz iterations can run natively in NVFP4 without the mixed-precision auxiliary ops earlier low-bit Muon work required.
  • Quantizing only Q/K/V and dS (keeping PV and dOV^T in BF16) covers 4 of 7 attention matmuls, cuts activation IO, and preserves stability.
  • Full-stack 4-bit pretraining can stay within ~1.5% of BF16 loss at 3B/64B tokens with every block quantized.
  • The modules are modular and additive, so partial adoption still yields cumulative stability and efficiency gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the error-growth analysis holds, larger hidden dimensions will force more frequent SVD realignment or higher LoRA rank, creating a practical scaling trade-off above 3B.
  • Dynamic precision switching (full 4-bit early, gradual high-precision restoration late) sketched in the outlook could become the industrial default once native Blackwell kernels exist.
  • The simplex-violation and distribution-misalignment arguments that protect PV/dOV^T may apply to other softmax-heavy operators such as cross-attention or MoE routing.
  • Fake-quantization results this close to BF16 suggest that, once real NVFP4 kernels arrive, remaining gaps will be dominated by kernel numerics rather than algorithmic instability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes Full-Stack FP4, an end-to-end NVFP4 pretraining recipe that jointly quantizes linear projections, optimizer states/computation, and attention. It identifies three module-specific failure modes (quantization-noise ceiling and dimension-scaled error growth in linears; heavy-tailed, denominator-sensitive AdamW second moments and fragile Newton–Schulz iterations; path-sensitive attention requiring forward–backward consistency) and addresses them with LoRA-SVD residual decomposition, a four-step transformed-state pipeline for AdamW vt plus native NVFP4 Newton–Schulz for Root/Muon, and mixed-precision attention that quantizes Q/K/V and dS while keeping PV and dOV⊤ in BF16 with tensor reuse. Ablations on 1B/20B tokens (Table 3) isolate each component; the main 3B/64B-token run reports a 1.47% loss gap to a BF16 Root+AdamW baseline with every block quantized. Appendices supply variance analysis for quantized matmuls and a theoretical account of attention-path sensitivity.

Significance. If the result holds under real NVFP4 Tensor-Core execution and longer/larger training, the work would be the first demonstrated full-stack 4-bit pretraining stack covering linears, optimizers, and attention, closing a clear gap left by prior linear-only NVFP4 recipes and low-bit Muon studies. The modular design, relative-MSE diagnostics, and explicit analyses of error scaling (Appendix A) and attention sensitivity (Appendix D) are concrete engineering contributions that other groups can reuse. The paper is candid about its current scale and simulation setting, which strengthens rather than weakens the claim that stable full-stack NVFP4 is at least feasible in principle.

major comments (3)
  1. Section 4 and Limitations: the central claim that “stable full-stack NVFP4 LLM pretraining is achievable” (1.47% gap, every block quantized) rests on a single 3B/64B-token run under fake quantization on A800 GPUs. Fake quantization does not reproduce Blackwell Tensor-Core accumulation order, fused scale application, or exact rounding/saturation behavior. Because stability depends on residual BF16 paths (LoRA-SVD, transformed vt reconstruction, PV/dOV⊤), a hardware discrepancy could re-open the error modes analyzed in Appendices A and D. At minimum the claim should be scoped to “under NVFP4 simulation,” or a short native-kernel or multi-seed validation should be added.
  2. Table 3 / Figure 5: all reported loss gaps are single-run point estimates with no error bars, multi-seed statistics, or sensitivity sweeps over free parameters (LoRA rank r, β, SVD realignment interval, NS coefficients a,b,c). The 0.61% linear-only and 1.47% full-stack gaps are therefore hard to interpret as robust. A modest multi-seed or hyperparameter-sensitivity check on the 1B ablation setting would make the quantitative claims load-bearing rather than anecdotal.
  3. Appendix D / Table 2: the decision to keep PV and dOV⊤ in BF16 is justified by simplex-violation and distribution-misalignment arguments that assume 1×16 block scaling and Softmax skew. The analysis is useful but remains qualitative; it does not quantify how large the SNR degradation becomes under realistic attention distributions, nor does it test whether alternative block sizes or stochastic rounding on those paths could close the gap. Without that, the mixed-precision split is empirically motivated but not fully stress-tested as a necessary design choice.
minor comments (5)
  1. Figure 1 is dense; the LoRA-SVD and optimizer call-outs would benefit from a clearer legend or a simplified schematic of dataflow per training step.
  2. Notation for the four-step vt pipeline (Section 3.3) mixes tile-wise and block-wise means; a short pseudocode block would remove ambiguity about reconstruction order.
  3. Table 1 marks several prior works as “—” for attention or optimizer computation; a footnote clarifying partial coverage (e.g., Attn-QAT, SageAttn3) would avoid overstating the novelty gap.
  4. Appendix G loss-landscape figures are informative but the color scales differ across panels; a shared scale would make the “steeper yet smooth” claim easier to verify visually.
  5. Minor typos: “Root (Muon)” vs “Root/Muon” inconsistency; “sg” global scale sometimes written without subscript; “4/6” scaling referenced without restating the adaptive rule on first use in the main text.

Circularity Check

0 steps flagged

No circular derivation: the 1.47% gap and module ablations are measured against external BF16 baselines; appendices use standard quantization variance and Softmax arguments that do not define the target loss.

full rationale

This is an empirical systems paper whose central claim (stable full-stack NVFP4 pretraining with a 1.47% loss gap on 3B/64B tokens) is a measured training-loss comparison to a BF16 Root+AdamW baseline, not a quantity recovered from a fitted constant or a self-defined identity. Ablations (Table 3) isolate LoRA-SVD, Q-AdamW, and Q-Attn by the same external metric. Appendix A derives Var(δY)=Θ(N) from independent zero-mean quantization noise under SR; Appendix D argues PV / dOV⊤ sensitivity from simplex violation and P–dP misalignment under 1×16 block scaling. Neither analysis fits a parameter to the reported loss gap and then re-labels it as a prediction. Citations (SVDQuant, Root, NVFP4, 4/6, Attn-QAT, etc.) supply prior formats or optimizers; none is a same-author uniqueness theorem that forces the recipe. Minor engineering choices (periodic SVD every 4k steps, Root coefficients, mixed-precision split) are design decisions validated by Rel. MSE and loss curves, not circular reductions. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 4 invented entities

The central empirical claim rests on a handful of free design choices (rank, realignment interval, NS coefficients, learning-rate schedule) and on standard but non-trivial modeling assumptions about quantization noise and attention sensitivity. No new physical entities are postulated; the invented constructs are algorithmic modules whose independent evidence is the measured loss and MSE curves themselves.

free parameters (5)
  • LoRA rank r and scale β
    Chosen to keep the BF16 branch light; exact values are not ablated beyond the presence/absence of LoRA-SVD, yet they control how much noise is diverted from the NVFP4 residual.
  • SVD realignment interval (every 4k steps)
    Hand-chosen period that trades subspace drift against compute; no sensitivity study is reported.
  • Newton–Schulz coefficients (a,b,c) for Root
    Shape-optimized beyond the original Muon values; treated as free parameters that also act as implicit error control under quantization.
  • Peak learning rates (1.5e-4 main, 3e-4 LoRA) and cosine schedule
    Standard but fitted to the 3B setting; the paper notes a steeper loss landscape may prefer smaller LR for NVFP4.
  • 4/6 adaptive block scaling thresholds and Hadamard block size 16
    Inherited from prior NVFP4 work and fixed for all experiments; they determine the effective quantization grid.
axioms (4)
  • domain assumption Quantization error variance of an inner product grows linearly (Θ(N)) with hidden dimension under independent zero-mean rounding (Appendix A).
    Used to motivate why direct NVFP4 hits a noise ceiling that LoRA-SVD must break; holds exactly only for stochastic rounding with independent draws.
  • domain assumption AdamW second-moment vt is non-negative, heavy-tailed and denominator-sensitive, so direct NVFP4 storage is unstable without the four-step transform.
    Stated as the stability challenge in §3.3 and consistent with COAT; treated as given rather than re-derived.
  • ad hoc to paper The PV and dOV⊤ attention paths are inherently sensitive to NVFP4 because of simplex violation and distribution misalignment (Appendix D).
    Theoretical justification for keeping those paths in BF16; the derivation is paper-specific and underpins the mixed-precision design.
  • domain assumption Fake quantization on A800 faithfully predicts behavior of native NVFP4 Tensor Cores on Blackwell.
    Implicit throughout §4; never validated on real hardware.
invented entities (4)
  • LoRA-SVD linear projection independent evidence
    purpose: Structural decomposition that routes principal subspace to BF16 LoRA and residual to NVFP4, escaping the direct-quantization error ceiling.
    New algorithmic construct; independent evidence is the measured 10% Rel-MSE reduction and the linear-only gap drop from 1.40% to 0.61%.
  • Four-step transformed-state pipeline for AdamW vt independent evidence
    purpose: Reshape second-moment distribution (sqrt, mean-subtract, Hadamard, quantize) so it survives NVFP4 storage.
    Paper-specific transform; evidence is the ~60% error reduction in Figure 3a and the claim that its absence causes immediate failure.
  • Native NVFP4 Newton–Schulz iteration for Root/Muon independent evidence
    purpose: Run the orthogonalization matmuls entirely in NVFP4 with shape-tuned coefficients, avoiding mixed-precision auxiliaries.
    Extends low-bit Muon work; evidence is the Rel-MSE curves in Figure 3b.
  • Mixed-precision Q-Attn with unified tensor reuse independent evidence
    purpose: Quantize Q/K/V and dS while protecting PV/dOV paths and enforcing forward-backward consistency.
    Design choice justified by Appendix D and Figure 4; evidence is the ablation loss gaps.

pith-pipeline@v1.1.0-grok45 · 22639 in / 3556 out tokens · 41192 ms · 2026-07-11T19:16:47.780385+00:00 · methodology

0 comments
read the original abstract

Recent NVFP4 pretraining methods mainly target transformer linear layers, leaving optimizer states, optimizer arithmetic and attention underexplored in 4-bit pipelines. This critical gap blocks stable full-stack 4-bit pretraining, as the three core modules exhibit unique numerical failure patterns: linear layers hit hard quantization noise limits with dimension-propagated error amplification; AdamW second moments are heavy-tailed non-negative values fragile to low-precision denominators; attention carries error-prone computation paths demanding strict forward-backward quantization consistency. We propose Full-Stack FP4, the first complete NVFP4 pretraining framework resolving all three stability bottlenecks via module-wise precision strategies. For linear projections, LoRA-SVD lightweight decomposition suppresses quantization noise and breaks the direct-quantization error ceiling, shrinking the linear-only loss gap from 1.40% to 0.61%. For optimizers, we design AdamW second-moment transformation for robust NVFP4 storage and fully support native NVFP4 Newton-Schulz iterations for the Root (Muon) optimizer. For attention, a mixed-precision scheme quantizes Q/K/V and backward dS while guarding vulnerable paths in BF16, paired with unified tensor reuse to sustain forward-backward alignment. We further analyze fast error accumulation in naive low-bit matrix multiplication and the extreme sensitivity of PV / dOV^T attention branches. All modules are plug-and-play with cumulative stability and efficiency improvements. Our 3B/64B-token pretraining validates near-BF16 performance with merely 1.47% loss gap, verifying feasible stable end-to-end NVFP4 LLM pretraining.

Figures

Figures reproduced from arXiv: 2607.04422 by Guoqi Li, Jiabo Tong, Mingchuan Ma, Siyu Ding, Xingrun Xing, Ziming Wang.

Figure 1
Figure 1. Figure 1: Full-Stack FP4 system overview. The left panel shows the transformer architecture with Full-Stack NVFP4 modules. The right panel details our linear projection design with LoRA￾SVD 3.2. The framework supports two specialized low-precision optimizers for NVFP4 pre-training: Q-AdamW with our NVFP4-compression pipeline (Sec 3.3), and Q-Root with native NVFP4 NS￾iterations (Sec 3.4). Q-Attn denotes our mixed-pr… view at source ↗
Figure 2
Figure 2. Figure 2: Linear projection quantization errors analysis. LoRA-SVD consistently reduces quantization error by ∼10% for outputs, input gradients, and weight gradients compared to naive NVFP4. Overall, the trends remain stable throughout training. are routed to the BF16 LoRA branch, while quantization noise and residual components are assigned to the NVFP4 residual weight. To avoid redundant computation during gradien… view at source ↗
Figure 3
Figure 3. Figure 3: Optimizer quantization errors analysis. (a) Our transformed-state pipeline reduces AdamW second-moment error by ∼60%. (b) Optimized coefficients further stabilize Newton–Schulz iterations in NVFP4. Overall, the trends remain stable throughout training. We enable NS iterations to run natively in NVFP4 with shape-optimized coefficients, preserving orthogonality and stability. The first order momentum or Root… view at source ↗
Figure 4
Figure 4. Figure 4: Attention quantization errors analysis. Quantizing P V or dOV ⊤ causes large accuracy drops, confirming these paths are sensitive that should be kept in higher precision. This trend remains stable throughout training [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Training loss on 3B/64B-token pretraining. Our full-stack FP4 training achieves a convergence gap of only 1.47% relative to the BF16 Root+AdamW baseline, retaining nearly identical convergence behavior. This demonstrates that stable full-stack NVFP4 pretraining is achievable with negligible performance degradation. Appendix D theoretically derives the numerical origins of this sensitivity, justifying our n… view at source ↗
Figure 6
Figure 6. Figure 6: Quantization MSE of the final-block output activation under increasing hidden dimension. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Gradient norm curves during 3B model pretraining with 64B training tokens. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Loss landscape comparison across four training steps (arranged vertically from top to [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 15 linked inside Pith

  1. [1]

    Pretraining large language models with nvfp4.arXiv preprint arXiv:2509.25149, 2025

    Felix Abecassis, Anjulie Agrusa, Dong Ahn, Jonah Alben, Stefania Alborghetti, Michael Andersch, Sivakumar Arayandi, Alexis Bjorlin et al. Pretraining large language models with nvfp4.arXiv preprint arXiv:2509.25149, 2025

    arXiv 2025

  2. [2]

    Fp4 all the way: Fully quantized training of llms.arXiv preprint arXiv:2505.19115, 2025

    Brian Chmiel, Maxim Fishman, Ron Banner, and Daniel Soudry. Fp4 all the way: Fully quantized training of llms.arXiv preprint arXiv:2505.19115, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Quartet: Native fp4 training can be optimal for large language models.arXiv preprint arXiv:2505.14669, 2025

    Roberto L Castro, Andrei Panferov, Soroush Tabesh, Oliver Sieberling, Jiale Chen, Mahdi Nikdan, Saleh Ashkboos, and Dan Alistarh. Quartet: Native fp4 training can be optimal for large language models.arXiv preprint arXiv:2505.14669, 2025

    arXiv 2025

  4. [4]

    Dissecting outlier dynamics in llm nvfp4 pretraining.arXiv preprint arXiv:2602.02047, 2026

    Peijie Dong, Ruibo Fan, Yuechen Tao, Di Mou, Wenhu Hu, Zhenheng Tang, Yinghao Yu, Jiamang Wang et al. Dissecting outlier dynamics in llm nvfp4 pretraining.arXiv preprint arXiv:2602.02047, 2026

    arXiv 2026

  5. [5]

    Four over six: More accurate nvfp4 quantization with adaptive block scaling.arXiv preprint arXiv:2512.02010, 2025

    Jack Cook, Junxian Guo, Guangxuan Xiao, Yujun Lin, and Song Han. Four over six: More accurate nvfp4 quantization with adaptive block scaling.arXiv preprint arXiv:2512.02010, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Oscillation-reduced mxfp4 training for vision transformers.arXiv preprint arXiv:2502.20853, 2025

    Yuxiang Chen, Haocheng Xi, Jun Zhu, and Jianfei Chen. Oscillation-reduced mxfp4 training for vision transformers.arXiv preprint arXiv:2502.20853, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Tetrajet-v2: Accurate nvfp4 training for large language models with oscillation suppression and outlier control.arXiv preprint arXiv:2510.27527, 2025

    Yuxiang Chen, Xiaoming Xu, Pengle Zhang, Michael Beyer, Martin Rapp, Jun Zhu, and Jianfei Chen. Tetrajet-v2: Accurate nvfp4 training for large language models with oscillation suppression and outlier control.arXiv preprint arXiv:2510.27527, 2025. 9

    Pith/arXiv arXiv 2025

  8. [8]

    Muon: An optimizer for hidden layers in neural networks, 2024.URL https://kellerjordan

    Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.URL https://kellerjordan. github. io/posts/muon, 6(3):4, 2024

    2024

  9. [9]

    Muon is scalable for llm training, 2025

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu et al. Muon is scalable for llm training, 2025. URL https://arxiv.org/abs/ 2502.16982

    Pith/arXiv arXiv 2025

  10. [10]

    Deepseek-v4: Towards highly efficient million-token context intelligence, 2026

    DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026

    2026

  11. [11]

    Achieving low-bit muon through subspace preservation and grid quantization

    Huaijin Wu, Bingrui Li, Yebin Yang, Yi Tu, Zhanpeng Zhou, Jianfei Chen, and Junchi Yan. Achieving low-bit muon through subspace preservation and grid quantization. InThe F ourteenth International Conference on Learning Representations, 2026. URL https://openreview. net/forum?id=g2l9bg9DWx

    2026

  12. [12]

    Coat: Compressing optimizer states and activation for memory-efficient fp8 training.arXiv preprint arXiv:2410.19313, 2024

    Haocheng Xi, Han Cai, Ligeng Zhu, Yao Lu, Kurt Keutzer, Jianfei Chen, and Song Han. Coat: Compressing optimizer states and activation for memory-efficient fp8 training.arXiv preprint arXiv:2410.19313, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Metis: Training llms with fp4 quantization.arXiv preprint arXiv:2509.00404, 2025

    Hengjie Cao, Mengyi Chen, Yifeng Yang, Ruijun Huang, Fang Dong, Jixian Zhou, Anrui Chen, Mingzhi Dong et al. Metis: Training llms with fp4 quantization.arXiv preprint arXiv:2509.00404, 2025

    arXiv 2025

  14. [14]

    Attn-qat: 4-bit attention with quantization-aware training.arXiv preprint arXiv:2603.00040, 2026

    Peiyuan Zhang, Matthew Noto, Wenxuan Tan, Chengquan Jiang, Will Lin, Wei Zhou, and Hao Zhang. Attn-qat: 4-bit attention with quantization-aware training.arXiv preprint arXiv:2603.00040, 2026

    arXiv 2026

  15. [15]

    Sageattention3: Microscaling fp4 attention for inference and an exploration of 8-bit training.arXiv preprint arXiv:2505.11594, 2025

    Jintao Zhang, Jia Wei, Pengle Zhang, Xiaoming Xu, Haofeng Huang, Haoxu Wang, Kai Jiang, Jianfei Chen et al. Sageattention3: Microscaling fp4 attention for inference and an exploration of 8-bit training.arXiv preprint arXiv:2505.11594, 2025

    arXiv 2025

  16. [16]

    Microscaling data formats for deep learning

    Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Root: Robust orthogonalized optimizer for neural network training.arXiv preprint arXiv:2511.20626, 2025

    Wei He, Kai Han, Hang Zhou, Hanting Chen, Zhicheng Liu, Xinghao Chen, and Yunhe Wang. Root: Robust orthogonalized optimizer for neural network training.arXiv preprint arXiv:2511.20626, 2025

    arXiv 2025

  18. [18]

    Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344–16359, 2022

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344–16359, 2022

    2022

  19. [19]

    Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Training llms with mxfp4.arXiv preprint arXiv:2502.20586, 2025

    Albert Tseng, Tao Yu, and Youngsuk Park. Training llms with mxfp4.arXiv preprint arXiv:2502.20586, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Svdquant: Absorbing outliers by low-rank components for 4-bit diffusion models

    Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng et al. Svdquant: Absorbing outliers by low-rank components for 4-bit diffusion models. arXiv preprint arXiv:2411.05007, 2024

    arXiv 2024

  22. [22]

    Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions.SIAM review, 53 (2):217–288, 2011

    Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions.SIAM review, 53 (2):217–288, 2011

    2011

  23. [23]

    Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model, 2025

    NVIDIA, :, Aarti Basant, Abhijit Khairnar, Abhijit Paithankar, Abhinav Khattar, Adithya Renduchintala, Aditya Malte et al. Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model, 2025. URLhttps://arxiv.org/abs/2508.14444

    Pith/arXiv arXiv 2025

  24. [24]

    Training and inference with integers in deep neural networks

    Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. InInternational Conference on Learning Representations, 2018. 10

    2018

  25. [25]

    Fp8 formats for deep learning.arXiv preprint arXiv:2209.05433, 2022

    Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke et al. Fp8 formats for deep learning.arXiv preprint arXiv:2209.05433, 2022. URLhttps://arxiv.org/abs/2209.05433

    Pith/arXiv arXiv 2022

  26. [26]

    Stable and low-precision training for large-scale vision-language models.Advances in Neural Information Processing Systems, 36:10271–10298, 2023

    Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low-precision training for large-scale vision-language models.Advances in Neural Information Processing Systems, 36:10271–10298, 2023

    2023

  27. [27]

    Scaling fp8 training to trillion-token llms.arXiv preprint arXiv:2409.12517, 2024

    Maxim Fishman, Brian Chmiel, Ron Banner, and Daniel Soudry. Scaling fp8 training to trillion-token llms.arXiv preprint arXiv:2409.12517, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Quarot: Outlier-free 4-bit inference in rotated llms.Advances in Neural Information Processing Systems, 37:100213–100240, 2024

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler et al. Quarot: Outlier-free 4-bit inference in rotated llms.Advances in Neural Information Processing Systems, 37:100213–100240, 2024

    2024

  30. [30]

    Optimizing large language model training using fp4 quantization.arXiv preprint arXiv:2501.17116, 2025

    Ruizhe Wang, Yeyun Gong, Xiao Liu, Guoshuai Zhao, Ziyue Yang, Baining Guo, Zhengjun Zha, and Peng Cheng. Optimizing large language model training using fp4 quantization.arXiv preprint arXiv:2501.17116, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    8-bit optimizers via block-wise quantization.arXiv preprint arXiv:2110.02861, 2021

    Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization.arXiv preprint arXiv:2110.02861, 2021

    Pith/arXiv arXiv 2021

  32. [32]

    Memory efficient optimizers with 4-bit states.Advances in Neural Information Processing Systems, 36:15136–15171, 2023

    Bingrui Li, Jianfei Chen, and Jun Zhu. Memory efficient optimizers with 4-bit states.Advances in Neural Information Processing Systems, 36:15136–15171, 2023

    2023

  33. [33]

    Effective quantization of muon optimizer states.arXiv preprint arXiv:2509.23106, 2025

    Aman Gupta, Rafael Celente, Abhishek Shivanna, DT Braithwaite, Gregory Dexter, Shao Tang, Hiroto Udagawa, Daniel Silva et al. Effective quantization of muon optimizer states.arXiv preprint arXiv:2509.23106, 2025

    arXiv 2025

  34. [34]

    Sageat- tention: Accurate 8-bit attention for plug-and-play inference acceleration.arXiv preprint arXiv:2410.02367, 2024

    Jintao Zhang, Jia Wei, Haofeng Huang, Pengle Zhang, Jun Zhu, and Jianfei Chen. Sageat- tention: Accurate 8-bit attention for plug-and-play inference acceleration.arXiv preprint arXiv:2410.02367, 2024

    arXiv 2024

  35. [35]

    Sageat- tention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization

    Jintao Zhang, Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen. Sageat- tention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. arXiv preprint arXiv:2411.10958, 2024

    arXiv 2024

  36. [36]

    variance

    Xiao Sun, Jungwook Choi, Chia-Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalak- shmi Viji Srinivasan, Xiaodong Cui, Wei Zhang et al. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks.Advances in neural information processing systems, 32, 2019. 11 A Appendix: Quantization Noise in Matrix Products We derive the varian...

    2019