REVIEW 5 cited by
Revisiting BFloat16 Training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Revisiting BFloat16 Training
read the original abstract
State-of-the-art generic low-precision training algorithms use a mix of 16-bit and 32-bit precision, creating the folklore that 16-bit hardware compute units alone are not enough to maximize model accuracy. As a result, deep learning accelerators are forced to support both 16-bit and 32-bit floating-point units (FPUs), which is more costly than only using 16-bit FPUs for hardware design. We ask: can we train deep learning models only with 16-bit floating-point units, while still matching the model accuracy attained by 32-bit training? Towards this end, we study 16-bit-FPU training on the widely adopted BFloat16 unit. While these units conventionally use nearest rounding to cast output to 16-bit precision, we show that nearest rounding for model weight updates often cancels small updates, which degrades the convergence and model accuracy. Motivated by this, we study two simple techniques well-established in numerical analysis, stochastic rounding and Kahan summation, to remedy the model accuracy degradation in 16-bit-FPU training. We demonstrate that these two techniques can enable up to 7% absolute validation accuracy gain in 16-bit-FPU training. This leads to 0.1% lower to 0.2% higher validation accuracy compared to 32-bit training across seven deep learning applications.
Forward citations
Cited by 5 Pith papers
-
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
A recurrent-depth architecture enables language models to improve reasoning performance by iterating computation in latent space, achieving gains equivalent to much larger models on benchmarks.
-
Reference Traces for Auditing Invisible Weight Updates and Guiding Exact-Budget Protection
Weight updates that fall below half a ULP freeze coordinates deterministically, and freeze time is predictable a priori from a high-precision trajectory and mantissa length alone.
-
M+Adam: Low-Precision Training via Additive-Multiplicative Optimization
M+Adam combines additive and multiplicative update branches, avoiding low-precision rounding stalls and improving LLaMA-style pretraining perplexity compared with AdamW at BF16, FP8, and FP4 master-weight storage.
-
M+Adam: Low-Precision Training via Additive-Multiplicative Optimization
Combining additive and multiplicative Adam-style updates yields stable BF16/FP8/FP4 master-weight training and lower C4 perplexity than AdamW on 60M–1B LLaMA-style models.
-
Reference Traces for Auditing Invisible Weight Updates and Guiding Exact-Budget Protection
A matched high-precision reference trace forecasts invisible weight-update events in controlled low-precision runs, and a pre-frozen 10% fp32-master mask recovers ~61% of the RNE-vs-fp32 loss gap on three unseen targets.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.