REVIEW 9 cited by
Recipes for Pre-training LLMs with MXFP8
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Using fewer bits to represent model parameters and related tensors during pre-training has become a required technique for improving GPU efficiency without sacrificing accuracy. Microscaling (MX) formats introduced in NVIDIA Blackwell generation of GPUs represent a major advancement of this technique, making it practical to combine narrow floating-point data types with finer granularity per-block scaling factors. In turn, this enables both quantization of more tensors than previous approaches and more efficient execution of operations on those tensors. Effective use of MX-formats requires careful choices of various parameters. In this paper we review these choices and show how MXFP8-E4M3 datatype and a specific number conversion algorithm result in training sessions that match those carried out in BF16. We present results using models with up to 8B parameters, trained on high-quality datasets of up to 15T tokens.
Forward citations
Cited by 9 Pith papers
-
Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
Four Over Six adaptively scales blocks in NVFP4 quantization to smaller FP4 values, making representable value distributions more uniform and reducing quantization error especially for near-maximal values.
-
Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference
AdaMX puts per-block format and enhancement metadata into spare MX exponent bits, recovering 83% of MXFP4's commonsense accuracy loss at the same 4-bit storage budget.
-
Stable FP4 Training via Transposition-Invariant Block Quantization
Transposition-invariant 2D block FP4 quantization plus truncation-free scaling and stochastic rounding enables stable end-to-end FP4 LLM training within ~1% of BF16.
-
SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
SOAP and Muon, stabilized by per-step QR eigenbasis updates and KL-Shampoo covariance accumulation, beat AdamW on large-batch LLM pretraining up to 100M-token batches.
-
StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
StoSignSGD resolves SignSGD divergence on non-smooth objectives via structural stochasticity, matching optimal convex rates and improving non-convex bounds while delivering 1.44-2.14x speedups in FP8 LLM pretraining.
-
ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning
ACRL stabilizes FP8-quantized LLM reinforcement learning by adaptively scaling token-level importance ratios with the measured training-inference discrepancy.
-
Stochastic Rounding Increases Small Singular Values
Stochastic rounding lifts clusters of small singular values even in constant aspect ratio matrices, extending its role as a spectral regularizer.
-
OSP-Next: Efficient High-Quality Video Generation with Sparse Sequence Parallelism, HiF8 Quantization, and Reinforcement Learning
OSP-Next reports 83.73% VBench score and up to 2.27x speedup via hybrid sparse attention, SSP parallelism, HiF8 quantization, and Mix-GRPO on diffusion transformers.
-
A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models
A hybrid-granularity FP8 training recipe achieves near-lossless accuracy versus BF16 on reasoning benchmarks with up to 22% faster training and 14% lower memory.
Discussion (0). Sign in to comment.