REVIEW 14 cited by
FP8-LM: Training FP8 Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
In this paper, we explore FP8 low-bit data formats for efficient training of large language models (LLMs). Our key insight is that most variables, such as gradients and optimizer states, in LLM training can employ low-precision data formats without compromising model accuracy and requiring no changes to hyper-parameters. Specifically, we propose a new FP8 automatic mixed-precision framework for training LLMs. This framework offers three levels of FP8 utilization to streamline mixed-precision and distributed parallel training for LLMs. It gradually incorporates 8-bit gradients, optimizer states, and distributed learning in an incremental manner. Experiment results show that, during the training of GPT-175B model on H100 GPU platform, our FP8 mixed-precision training framework not only achieved a remarkable 39% reduction in real memory usage but also ran 75% faster than the widely adopted BF16 framework (i.e., Megatron-LM), surpassing the speed of Nvidia Transformer Engine by 37%. This largely reduces the training costs for large foundation models. Furthermore, our FP8 mixed-precision training methodology is generic. It can be seamlessly applied to other tasks such as LLM instruction tuning and reinforcement learning with human feedback, offering savings in fine-tuning expenses. Our FP8 low-precision training framework is open-sourced at {https://github.com/Azure/MS-AMP}{aka.ms/MS.AMP}.
Forward citations
Cited by 14 Pith papers
-
FP4 All the Way: Fully Quantized Training of LLMs
A 7-billion parameter LLM trained almost entirely in 4-bit float precision matches a standard 16-bit baseline after a brief quantization-aware fine-tuning phase.
-
GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining
Transforming gradients into K-FAC-based coordinates before FP8 quantization reduces communication error and improves downstream task preservation over Euclidean FP8, with a 7.6% end-to-end speedup on 64 GH200 GPUs.
-
DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce
DynamiQ compresses gradients for multi-hop all-reduce with per-block bit-width allocation, block reordering, and fused recompression, matching BF16 accuracy at ~5 bits per coordinate and improving time-to-accuracy by ...
-
Compute Requirements for Algorithmic Innovation in Frontier AI Models
Estimated development compute for 36 LLM pretraining innovations shows half would remain possible under GPT-2-level or 8-H100 compute caps.
-
Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networks
LMD, a log-normal multiplicative-weight optimizer, trains ViT and GPT-2 from scratch and keeps accuracy under MXFP6 forward-pass quantization.
-
Thunder-LLM: Efficiently Adapting LLMs to Korean with Minimal Resources
A cost-effective recipe consisting of tokenizer extension, continual pretraining, FP8 training, and SFT/DPO post-training yields Korean-English bilingual 8B models with top Korean benchmark scores.
-
Scaling Law for Quantization-Aware Training
A unified QAT scaling law predicts 4-bit quantization error from model size, training tokens, and group size, showing activation outliers in the FC2 layer are the main W4A4 bottleneck.
-
Gaussian Weight Sampling for Scalable, Efficient and Stable Pseudo-Quantization Training
Using a rounded Gaussian noise distribution for pseudo-quantization makes low-precision FP weight training stable and cheap, matching or approaching BF16 baseline loss in LLM pretraining.
-
QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache
QuantSpec accelerates long-context LLM decoding by giving the self-draft model a 4-bit hierarchical KV cache shared with an INT8-quality target cache, achieving up to ~2.5x speedup and mostly >90% acceptance.
-
A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models
A hybrid-granularity FP8 training recipe achieves near-lossless accuracy versus BF16 on reasoning benchmarks with up to 22% faster training and 14% lower memory.
-
Foundation Models for Clean Energy Forecasting: A Comprehensive Review
A survey of foundation model methods, data, and open problems for renewable energy forecasting, built from roughly 218 cited works.
-
Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities
A review that organizes low-precision LLM training research into fixed-point, floating-point, and custom numeric format categories and discusses open challenges.
-
DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models
A survey of DeepSeek's V3 and R1 models covering MLA, MoE, MTP, GRPO, and training engineering, with no new experimental results.
-
Resource-Efficient Language Models: Quantization for Fast and Accessible Inference
A survey of post-training quantization techniques for large language models, covering schemes, granularities, and popular methods, with no new experimental results.
Discussion (0). Continue with ORCID to comment.