Pith. sign in

REVIEW 11 cited by

Training LLMs with MXFP4

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.20586 v3 pith:SGZ57SD7 submitted 2025-02-27 cs.LG

Albert Tseng , Tao Yu , Youngsuk Park This is my paper · ORCID
classification cs.LG
keywords mxfp4trainingbf16timesdirectlyduringgemmshowever
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Low precision (LP) datatypes such as MXFP4 can accelerate matrix multiplications (GEMMs) and reduce training costs. However, directly using MXFP4 instead of BF16 during training significantly degrades model quality. In this work, we present the first near-lossless training recipe that uses MXFP4 GEMMs, which are $2\times$ faster than FP8 on supported hardware. Our key insight is to compute unbiased gradient estimates with stochastic rounding (SR), resulting in more accurate model updates. However, directly applying SR to MXFP4 can result in high variance from block-level outliers, harming convergence. To overcome this, we use the random Hadamard tranform to theoretically bound the variance of SR. We train GPT models up to 6.7B parameters and find that our method induces minimal degradation over mixed-precision BF16 training. Our recipe computes $>1/2$ the training FLOPs in MXFP4, enabling an estimated speedup of $>1.3\times$ over FP8 and $>1.7\times$ over BF16 during backpropagation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models

    cs.LG 2026-07 conditional novelty 7.0 of 10

    In end-to-end 4-bit RL post-training, rollout activation underflow, not training quantization, is the main accuracy killer; a sparse residual correction closes most of the gap to BF16.

  2. Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Full-stack NVFP4 LLM pretraining is stable with a 1.47% loss gap to BF16 via LoRA-SVD projections, transformed AdamW/Root optimizers, and mixed-precision attention.

  3. OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration

    cs.LG 2025-07 conditional novelty 7.0 of 10

    OASIS enables efficient LLM inference with non-uniform 4-bit weights and activations via precomputed Cartesian product lookup tables and a parallel outlier-compensation branch, at a reported 1.94-2.05% average accuracy drop.

  4. FP4 All the Way: Fully Quantized Training of LLMs

    cs.LG 2025-05 conditional novelty 7.0 of 10

    A 7-billion parameter LLM trained almost entirely in 4-bit float precision matches a standard 16-bit baseline after a brief quantization-aware fine-tuning phase.

  5. Reference Traces for Auditing Invisible Weight Updates and Guiding Exact-Budget Protection

    cs.LG 2026-07 conditional novelty 6.5 of 10

    Weight updates that fall below half a ULP freeze coordinates deterministically, and freeze time is predictable a priori from a high-precision trajectory and mantissa length alone.

  6. Stable FP4 Training via Transposition-Invariant Block Quantization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Transposition-invariant 2D block FP4 quantization plus truncation-free scaling and stochastic rounding enables stable end-to-end FP4 LLM training within ~1% of BF16.

  7. CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits

    cs.AI 2026-07 conditional novelty 6.0 of 10

    CANN Bench is an Ascend-NPU benchmark with 53 operators and 1,060 released test cases that scores agent-generated kernels on compilation, correctness, and hardware-anchored performance.

  8. Recipes for Pre-training LLMs with MXFP8

    cs.LG 2025-05 conditional novelty 6.0 of 10

    MXFP8-E4M3 with round-up scale computation matches BF16 pre-training accuracy for LLMs up to 8B parameters, enabling 2x faster training on Blackwell.

  9. Scaling Law for Quantization-Aware Training

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A unified QAT scaling law predicts 4-bit quantization error from model size, training tokens, and group size, showing activation outliers in the FC2 layer are the main W4A4 bottleneck.

  10. ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs

    cs.LG 2025-02 conditional novelty 6.0 of 10

    ProxSparse learns 2:4 semi-structured sparsity masks for pretrained LLMs via regularized proximal-gradient optimization, outperforming heuristic baselines on seven models.

  11. What is New in Stochastic Rounding: a Survey on Theory, Hardware, and Applications

    math.NA 2026-03 accept novelty 3.5 of 10

    An update survey of stochastic rounding (2022–2026) that centers limited-precision SR, commercial hardware, probabilistic error bounds, and applications in ML and scientific computing.

Pith tools