Pith. sign in

REVIEW 4 major objections 6 minor 21 references

Stable FP4 LLM training works once forward and backward passes use the same block scales after transposition.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 05:00 UTC pith:YXMOX7HI

load-bearing objection Useful 30B-scale FP4 training result with a clean diagnosis, but the 2D-geometry causal claim is not isolated from known stabilizers. the 4 major comments →

arxiv 2607.24953 v1 pith:YXMOX7HI submitted 2026-07-27 cs.LG cs.AI

Stable FP4 Training via Transposition-Invariant Block Quantization

Mehdi Rahimifar , Amin Darabi , Mehran Taghian Jazi , Xing Huang , Yao Wang , Zhijun Tu , Yufei Cui , Yunke Peng
show 1 more author
Hongliang Li
This is my paper
classification cs.LG cs.AI
keywords FP4 trainingmicroscaling2D block quantizationtransposition invariancestochastic roundingmixed-precision attentionLLM efficiencyforward-backward consistency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Training large language models in 4-bit floating point has been unstable even when block-wise microscaling reduces local quantization error. This paper argues the missing piece is consistency: standard 1D blocks regroup values after matrix transpose, so the same weight or gradient is scaled differently on the forward and backward passes and gradients become biased. Square 2D blocks keep each group of values under one shared scale through transposition, and pairing that with truncation-free scales and stochastic rounding keeps gradients unbiased enough to train. With MXFP8 only on query and key projections, dense models up to 7B and a 30B MoE train stably for 100B tokens and stay within about 1% of BF16 on perplexity and reasoning tasks. If that diagnosis is right, practical FP4 training is mainly a consistency problem, not an impossible precision problem.

Core claim

The central claim is that transposition-induced scale mismatch in 1D microscaling is a primary failure mode of FP4 training, and that enforcing forward–backward scale consistency with 2D square-block FP4 quantization—plus truncation-free scaling, stochastic rounding, and MXFP8 for Q/K—is sufficient for stable end-to-end FP4 training at scales up to 30B parameters and 100B tokens with less than 1.3% degradation versus BF16.

What carries the argument

Transposition-invariant 2D block FP4 quantization: partition tensors into b×b square blocks that share one scale so a block and its transpose keep the same maximum and the same scale (S(B)=S(B⊤)), eliminating the forward/backward scale mismatch of 1D layouts.

Load-bearing premise

That software-emulated FP4 microscaling on hardware without native FP4 tensor cores behaves like real MXFP4/NVFP4 hardware for the stability conclusions that matter.

What would settle it

Train the same OLMo-1B/7B or 30B MoE recipe on native FP4 tensor-core hardware (or a verified bit-accurate emulator) with 1D vs 2D blocks only, and check whether 1D still shows the forward–backward scale mismatch and early divergence while 2D stays within ~1% of BF16 loss and downstream accuracy through 100B tokens.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Stable FP4 training at LLM scale can be reached mainly by making quantization transpose-invariant, not only by shrinking blocks or raising scale precision.
  • Most transformer linear layers can stay FP4; only Q/K need higher precision (MXFP8) in this design.
  • Weight memory and ideal linear throughput can improve substantially (paper estimates ~65% weight memory cut and up to ~3.6× linear throughput on OLMo-1B) once native kernels exist.
  • Larger models can close the FP4–BF16 gap further once systematic scale bias is removed.
  • Hardware and formats should expose square-block or otherwise transpose-consistent scaling paths, not only 1D microscaling.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Any low-precision path that secretly changes scales under the adjoint (transpose, reshape, gather) may hit the same bias even outside FP4.
  • If consistency is the bottleneck, learned or adaptive 2D blocks and fused scale metadata could push past the 32×32 efficiency–accuracy trade-off without new number formats.
  • Selective higher precision may generalize: other noise-amplifying ops (softmax, layer-norm, router logits in MoE) could be the next places to keep above FP4 rather than whole layers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper identifies transposition-induced scale inconsistency in 1D microscaling formats (MXFP4/NVFP4) as a source of FP4 training instability: after transposition in backpropagation, values are regrouped into different scaling blocks, so forward and backward passes quantize the same tensor with different scales, biasing gradients under the STE. The proposed fix is 2D square-block FP4 quantization (32×32) for weights and gradients, which preserves block membership under transposition, combined with truncation-free scaling (to avoid overflow) and stochastic rounding in the backward pass (for unbiased gradients), plus MXFP8 for query/key projections. Experiments train OLMo-1B, OLMo-7B, and a Qwen 30B MoE on 100B tokens under software emulation, reporting stable convergence within ~0.8–1.6% of BF16 perplexity and ~1 point on downstream benchmarks.

Significance. If the results hold, this is a useful contribution to low-precision LLM training: stable end-to-end FP4 training up to 30B parameters / 100B tokens with ~1% of BF16 would be a practical step beyond FP8, and the transposition-invariance argument for 2D blocks is simple and mechanistically plausible. Notable strengths: a fully specified, reproducible training configuration (Appendix B: optimizer, schedule, batch size, quantization placement per tensor type); an explicit transposition-invariance proof for square blocks (Appendix D); matched hyperparameters across BF16 and FP4 variants so differences are attributable to the format; and evaluation against external BF16 baselines on public benchmarks rather than internal proxies. The paper is also appropriately candid about the software-emulation limitation (§4.1, §5), and its central design prediction — 2D block geometry should matter independently of scaling/rounding choices — is directly falsifiable with one additional experiment.

major comments (4)
  1. [§4.2, Fig. 4, Fig. 5, Table 4] The paper's central causal claim — that transposition-induced scale inconsistency is the dominant source of FP4 training instability, and that 2D block geometry is what fixes it — is never isolated experimentally. The only controlled ablations (Fig. 5) vary truncation-free scaling and stochastic rounding, both of which are established stabilizers in the cited prior work (e.g., refs [5, 6, 17]). Table 4 varies 2D block *size* (8×8 to 64×64) but never compares 1D vs 2D geometry at a matched recipe. Figure 4 compares against unnamed 'alternative microscaling schemes' whose configurations are unspecified. The load-bearing experiment is missing: 1D MXFP4 (1×32) with identical truncation-free scaling, stochastic rounding, and MXFP8 Q/K vs the proposed 2D-FP4 with the same components. This experiment would be especially informative because a 32×32 block shares one scale across 1024 values versu
  2. [§3.2, Table 2, Table 3, Appendix G] The MXFP8 Q/K variant consistently performs *worse* than the base 2D-FP4 variant across all reported metrics: Table 2 shows larger perplexity degradation for +MXFP8 at every scale (e.g., 1.6% vs 1.1% avg on OLMo-1B; 1.2% vs 0.9% on OLMo-7B), Table 3 shows lower average accuracy at every scale, and §4.2 acknowledges the loss gap 'increases to 1.6%.' Yet §3.2 motivates MXFP8 Q/K as necessary for stability ('directly quantizing Q and K to FP4 introduces errors that... distort the attention distribution'), and Appendix G reframes it as 'a stability-oriented mixed-precision option.' If the FP4-only configuration is uniformly closer to BF16 and equally stable in all reported runs, the case for the MXFP8 component is unsupported by the paper's own evidence. Either (a) show the FP4-Q/K configuration that fails or destabilizes (currently absent — the only divergence shown is 'naive FP4 without mi
  3. [Abstract, Table 2, Appendix B] The abstract states 'less than 1.3% degradation in perplexity and downstream accuracy,' but Table 2 reports 1.6% average perplexity degradation for the 2D-FP4+MXFP8 configuration on OLMo-1B, and Fig. 4/§4.2 repeat the 1.6% figure. The claim is only true for the base 2D-FP4 variant. Since the +MXFP8 configuration is presented as the proposed practical design (§3.2, Table 1, Appendix B where 'Query/Key Precision: MXFP8' appears in the default training configuration), the headline number is inconsistent with the paper's own tables. Please reconcile the abstract claim with Table 2, and clarify which configuration the claim refers to. Separately, Appendix B lists 'Query/Key Precision MXFP8' as the default for all models while the main text treats 2D-FP4 (without MXFP8) as 'Ours' — the default configuration should be stated unambiguously.
  4. [Fig. 4, §4.2] Figure 4 compares the method against 'alternative microscaling schemes' and 'competing methods' that are never named or cited in the caption or text, and no direct numerical comparison against the most relevant prior FP4 training work — MXFP4 training [17], NVFP4 pretraining [1], or Quartet [4] — appears anywhere in Tables 2–3. Given that [1] reports NVFP4 pretraining results and [4] reports native FP4 training, the absence of any controlled comparison to these baselines makes it impossible to assess the claimed improvement ('final loss gap approximately 1.1%, compared to around 2% for competing methods'). At minimum, the baselines in Fig. 4 must be identified, configured, and cited; ideally one named prior recipe should be run under the same token budget and hyperparameters.
minor comments (6)
  1. [Fig. 1] Fig. 1(d) reports 'effective weight deviation' but the quantity is never defined in the text or appendix. Please give a formula.
  2. [Table 1] Table 1's 'ideal linear throughput ~3.56×' and 'activation bandwidth ~0.30×' are described as 'idealized'; given that all experiments are software emulation (§4.1) and no kernel measurements exist, consider moving Table 1 to the appendix or clearly labeling it as a projection, since it does not reflect any measured system.
  3. [Table 4] Table 4 reports ranges ('~0.8–1.0', '~0.9–1.1') rather than single measurements, unlike Tables 2–3. Please report actual measured values with the model/token budget at which they were obtained.
  4. [Table 3, §4.2] The claim that FP4 exceeding BF16 on SciQ/COPA (Table 3) 'may be attributed to a regularization effect induced by quantization noise' is speculative; either provide supporting evidence (e.g., variance over seeds) or remove. Relatedly, no seed variance or error bars are reported anywhere — for gaps as small as 0.8–1.6%, run-to-run variance matters.
  5. [§3.1, Table 2] Notation: Qp and Qn are used in the scale formula (§3.1) before being defined; the E2M1 bounds should be stated explicitly. Also '∆' column header in Table 2 is undefined at first use (defined only in Appendix F).
  6. [§3, Appendix C] Activations use 1D-FP4 (Appendix B/C) while weights/gradients use 2D — but per the paper's own thesis, activations also participate in transposed backward GEMMs (∇X = ∇Y W). A brief discussion of why the 1D activation layout does not reintroduce the inconsistency the paper identifies would close an apparent gap in the argument.

Circularity Check

0 steps flagged

No circularity: method is a constructive design validated against external BF16 baselines, not a prediction forced by its inputs.

full rationale

The paper’s chain is: (i) 1D microscaling reassigns scales under transposition, inducing forward–backward mismatch; (ii) square 2D blocks preserve block maxima under transposition by construction, so S_fwd = S_bwd; (iii) add truncation-free scaling and stochastic rounding; (iv) evaluate end-to-end training against BF16 on held-out perplexity and reasoning benchmarks. Step (ii) is a design identity (square blocks are transposition-invariant because they contain the same values after transpose), not a fitted quantity renamed as a prediction, nor a uniqueness theorem imported from the authors. There is no parameter fit that is then reported as an independent forecast; block size 32×32 and learning rates are ordinary hyperparameters. Citations for STE, stochastic rounding, and microscaling formats are external prior art, not load-bearing self-citations that force the result. Empirical claims are checked against BF16 and public-style datasets (Wikitext, Pile, C4, SciQ, COPA, ARC-E, HellaSwag). Missing 1D-vs-2D ablations affect causal isolation of the mechanism, not circularity of the derivation. Score 0.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 1 invented entities

The central claim rests on standard autodiff/quantization practice (STE, block max scaling, stochastic rounding), the empirical premise that software FP4 emulation matches future hardware numerics, and design choices (square block size, 1D activations, MXFP8 Q/K) that are justified experimentally rather than derived. No new physical entities. Free parameters are ordinary ML hyperparameters and the chosen block size.

free parameters (3)
  • 2D block size (default 32×32) = 32×32
    Chosen for hardware tiling and accuracy/efficiency trade-off (Table 4); not derived from first principles. Final loss gap and metadata overhead depend on this choice.
  • Peak learning rates per model = 4e-4 (1B), 3e-4 (7B), 1e-4 (30B MoE)
    Hand-set per scale (4e-4 / 3e-4 / 1e-4) in Table 5; standard but load-bearing for matching BF16 trajectories.
  • FP4 format bounds (Qp, Qn) inside scale formula = E2M1 representable bounds (format-defined)
    Enter the truncation-free scale S = 2^ceil(log2(2M/(Qp-Qn))); concrete E2M1 bounds are taken as given by the format definition.
axioms (5)
  • domain assumption Straight-through estimator: ∂L/∂X ≈ ∂L/∂X̂ through the non-differentiable quantizer
    Invoked in Section 3 to allow end-to-end training; standard but approximate.
  • standard math Stochastic rounding yields unbiased gradients in expectation: E[round_stoch(x)] = x
    Used in Section 3.1 / Appendix E to justify backward FP4 gradients.
  • standard math Square b×b blocks preserve max-abs and therefore scale under transposition
    Core invariance lemma in Section 3 and Appendix D; mathematically immediate for equal block shapes.
  • ad hoc to paper Software emulation of MX/NV-style FP4 matches the stability behavior of future native tensor-core FP4
    Section 4.1 explicitly trains under emulation; Discussion lists native hardware as future work. Load-bearing for external validity of 'practical FP4 training'.
  • domain assumption Activations may remain 1D-quantized while weights/gradients are 2D without reintroducing fatal inconsistency
    Stated trade-off in Section 3; empirically accepted but not proved necessary or optimal.
invented entities (1)
  • Transposition-invariant 2D block FP4 training recipe (2D blocks + truncation-free scale + stochastic rounding + MXFP8 Q/K) no independent evidence
    purpose: Package the components that jointly deliver stable FP4 LLM training
    Not a new physical object; a composite method. Listed for completeness because the paper treats the combination as the unit of contribution.

pith-pipeline@v1.2.0-grok45-kimik3 · 15950 in / 3791 out tokens · 59778 ms · 2026-07-31T05:00:12.874736+00:00 · methodology

0 comments
read the original abstract

Reducing training precision is a key lever for improving the e ciency of large language model (LLM) training, but pushing beyond FP8 to 4-bit oating point (FP4) remains challenging due to instability during optimization. We identify a fundamental source of this instability in existing microscaling approaches: scale inconsistency induced by tensor transposition. In conventional 1D block quantization, forward and backward passes assign di erent scaling factors to the same values after transposition, leading to biased and unstable gradient updates. To address this issue, we propose a low-precision training framework based on 2D block FP4 quantization, which enforces transposition-invariant scaling and preserves consistency between forward and backward computations. We further combine this with truncation-free scaling and stochastic rounding to control quantization error and maintain unbiased gradients. To handle the sensitivity of attention mechanisms, we adopt MXFP8 quantization for query and key projections, yielding a practical mixed-precision design. We evaluate our method on dense LLMs up to 7B parameters and a 30B Mixture-of-Experts model, trained on up to 100B tokens. Across all settings, our approach achieves stable end-to-end FP4 training and closely matches BF16 performance, with less than 1.3% degradation in perplexity and downstream accuracy. These results demonstrate that enforcing forwardbackward scaling consistency is su cient to enable practical FP4 training at scale, providing a simple and e ective pathway toward more e cient LLM training.

Figures

Figures reproduced from arXiv: 2607.24953 by Amin Darabi, Hongliang Li, Mehdi Rahimifar, Mehran Taghian Jazi, Xing Huang, Yao Wang, Yufei Cui, Yunke Peng, Zhijun Tu.

Figure 1
Figure 1. Figure 1: Forward–backward inconsistency in 1D microscaling vs 2D blocks. (a) In 1D block quantization, transposition reassigns values to different scaling blocks, causing forward– backward mismatch and distribution shift. (b) 2D block quantization preserves block structure under transposition, yielding consistent scaling and aligned distributions. (c) Scale difference maps show large mismatch in 1D but near-zero er… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of 2D block FP4 quantization. Tensors are partitioned into square blocks that share a scale, enabling consistent scaling across forward and backward passes. Under the straight-through estimator (STE) [3], gradients are propagated as if Xˆ ≈ X, but this mismatch introduces a systematic bias: ∇Lbwd ̸= ∇Lfwd. In contrast, if the quantization scheme is invariant to transposition (i.e., Sfwd = Sbwd), b… view at source ↗
Figure 3
Figure 3. Figure 3: MXFP8 attention pathway. Query and key projections are quantized using MXFP8 to improve numerical stability while retaining low-precision computation in the rest of the model [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics under FP4 quantization across model scales. Naive FP4 without microscaling diverges early, while microscaling-based methods remain stable. The proposed 2D block FP4 method consistently converges closest to BF16, with the performance gap decreasing as model size increases. experiments, we compare BF16 training with our 2D block FP4 method, with and without MXFP8 attention. We begin by anal… view at source ↗
Figure 5
Figure 5. Figure 5: Ablation of truncation-free scaling and stochastic rounding. Both components are necessary for stable FP4 training: removing truncation-free scaling leads to overflow-induced instability, while disabling stochastic rounding introduces bias and degrades convergence [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 13 linked inside Pith

  1. [1]

    Pretraining large language models with nvfp4.arXiv preprint arXiv:2509.25149, 2025

    Felix Abecassis, Anjulie Agrusa, Dong Ahn, Jonah Alben, Stefania Alborghetti, Michael An- dersch, Sivakumar Arayandi, Alexis Bjorlin, Aaron Blakeman, Evan Briones, et al. Pretraining large language models with nvfp4.arXiv preprint arXiv:2509.25149, 2025

    arXiv 2025

  2. [2]

    gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025

    Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432, 2013

    Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432, 2013

    Pith/arXiv arXiv 2013

  4. [4]

    Quartet: Native fp4 training can be optimal for large language models.arXiv preprint arXiv:2505.14669, 2025

    Roberto L Castro, Andrei Panferov, Soroush Tabesh, Oliver Sieberling, Jiale Chen, Mahdi Nikdan, Saleh Ashkboos, and Dan Alistarh. Quartet: Native fp4 training can be optimal for large language models.arXiv preprint arXiv:2505.14669, 2025

    arXiv 2025

  5. [5]

    Oscillation-reduced mxfp4 training for vision transformers.arXiv preprint arXiv:2502.20853, 2025

    Yuxiang Chen, Haocheng Xi, Jun Zhu, and Jianfei Chen. Oscillation-reduced mxfp4 training for vision transformers.arXiv preprint arXiv:2502.20853, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Fp4 all the way: Fully quantized training of llms.arXiv preprint arXiv:2505.19115, 2025

    Brian Chmiel, Maxim Fishman, Ron Banner, and Daniel Soudry. Fp4 all the way: Fully quantized training of llms.arXiv preprint arXiv:2505.19115, 2025. 11

    Pith/arXiv arXiv 2025

  7. [7]

    A survey of low-bit large language models: Basics, systems, and algorithms.Neural Networks, page 107856, 2025

    Ruihao Gong, Yifu Ding, Zining Wang, Chengtao Lv, Xingyu Zheng, Jinyang Du, Yang Yong, Shiqiao Gu, Haotong Qin, Jinyang Guo, et al. A survey of low-bit large language models: Basics, systems, and algorithms.Neural Networks, page 107856, 2025

    2025

  8. [8]

    Low-precision training of large language models: Methods, challenges, and opportunities.arXiv preprint arXiv:2505.01043, 2025

    Zhiwei Hao, Jianyuan Guo, Li Shen, Yong Luo, Han Hu, Guoxia Wang, Dianhai Yu, Yonggang Wen, and Dacheng Tao. Low-precision training of large language models: Methods, challenges, and opportunities.arXiv preprint arXiv:2505.01043, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Towards fully fp8 gemm llm training at scale.arXiv preprint arXiv:2505.20524, 2025

    Alejandro Hernández-Cano, Dhia Garbaya, Imanol Schlag, and Martin Jaggi. Towards fully fp8 gemm llm training at scale.arXiv preprint arXiv:2505.20524, 2025

    arXiv 2025

  10. [10]

    How hungry is ai? benchmarking energy, water, and carbon footprint of llm inference, 2025

    Nidhal Jegham, Marwan Abdelatti, Chan Young Koh, Lassad Elmoubarki, and Abdeltawab Hendawi. How hungry is ai? benchmarking energy, water, and carbon footprint of llm inference, 2025

    2025

  11. [11]

    A comprehensive study on quantization tech- niquesforlargelanguagemodels

    Jiedong Lang, Zhehao Guo, and Shuyu Huang. A comprehensive study on quantization tech- niquesforlargelanguagemodels. In2024 4th International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC), pages 224–231. IEEE, 2024

    2024

  12. [12]

    Large language models: A survey, 2025

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, 2025

    2025

  13. [13]

    Recipes for pre-training llms with mxfp8.arXiv preprint arXiv:2506.08027, 2025

    Asit Mishra, Dusan Stosic, Simon Layton, and Paulius Micikevicius. Recipes for pre-training llms with mxfp8.arXiv preprint arXiv:2506.08027, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Quartet ii: Accu- rate llm pre-training in nvfp4 by improved unbiased gradient estimation.arXiv preprint arXiv:2601.22813, 2026

    Andrei Panferov, Erik Schultheis, Soroush Tabesh, and Dan Alistarh. Quartet ii: Accu- rate llm pre-training in nvfp4 by improved unbiased gradient estimation.arXiv preprint arXiv:2601.22813, 2026

    Pith/arXiv arXiv 2026

  15. [15]

    Microscaling data formats for deep learning.arXiv preprint arXiv:2310.10537, 2023

    Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, et al. Microscaling data formats for deep learning.arXiv preprint arXiv:2310.10537, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Scaling laws for floating point quantization training

    Xingwu Sun, Shuaipeng Li, Ruobing Xie, Weidong Han, Kan Wu, Zhen Yang, Yixing Li, An Wang, Shuai Li, Jinbao Xue, et al. Scaling laws for floating point quantization training. arXiv preprint arXiv:2501.02423, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Training llms with mxfp4.arXiv preprint arXiv:2502.20586, 2025

    Albert Tseng, Tao Yu, and Youngsuk Park. Training llms with mxfp4.arXiv preprint arXiv:2502.20586, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Optimizing large language model training using fp4 quantization.arXiv preprint arXiv:2501.17116, 2025

    Ruizhe Wang, Yeyun Gong, Xiao Liu, Guoshuai Zhao, Ziyue Yang, Baining Guo, Zhengjun Zha, and Peng Cheng. Optimizing large language model training using fp4 quantization.arXiv preprint arXiv:2501.17116, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Coat: Compressing optimizer states and activation for memory-efficient fp8 training.arXiv preprint arXiv:2410.19313, 2024

    Haocheng Xi, Han Cai, Ligeng Zhu, Yao Lu, Kurt Keutzer, Jianfei Chen, and Song Han. Coat: Compressing optimizer states and activation for memory-efficient fp8 training.arXiv preprint arXiv:2410.19313, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    An empirical study of microscaling formats for low-precision llm training

    Hanmei Yang, Summer Deng, Amit Nagpal, Maxim Naumov, Mohammad Janani, Tongping Liu, and Hui Guan. An empirical study of microscaling formats for low-precision llm training. In2025 IEEE 32nd Symposium on Computer Arithmetic (ARITH), pages 1–8. IEEE Computer Society, 2025. 12

    2025

  21. [21]

    Towards efficient pre-training: Exploring fp4 precision in large language models.arXiv preprint arXiv:2502.11458, 2025

    Jiecheng Zhou, Ding Tang, Rong Fu, Boni Hu, Haoran Xu, Yi Wang, Zhilin Pei, Zhongling Su, Liang Liu, Xingcheng Zhang, et al. Towards efficient pre-training: Exploring fp4 precision in large language models.arXiv preprint arXiv:2502.11458, 2025. 13 A Supplementary Material Thisappendixprovidesadditionalimplementationandevaluationdetailsfortheproposedtransp...

    Pith/arXiv arXiv 2025