REVIEW 22 cited by
Scaling Laws for Precision
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Low precision training and inference affect both the quality and cost of language models, but current scaling laws do not account for this. In this work, we devise "precision-aware" scaling laws for both training and inference. We propose that training in lower precision reduces the model's "effective parameter count," allowing us to predict the additional loss incurred from training in low precision and post-train quantization. For inference, we find that the degradation introduced by post-training quantization increases as models are trained on more data, eventually making additional pretraining data actively harmful. For training, our scaling laws allow us to predict the loss of a model with different parts in different precisions, and suggest that training larger models in lower precision may be compute optimal. We unify the scaling laws for post and pretraining quantization to arrive at a single functional form that predicts degradation from training and inference in varied precisions. We fit on over 465 pretraining runs and validate our predictions on model sizes up to 1.7B parameters trained on up to 26B tokens.
Forward citations
Cited by 22 Pith papers
-
Which Decisions Low-Bit Quantization Breaks, and How to Predict Them
Quantization reduces a model's decision margin by a multiplicative factor that collapses at low bit-widths, making per-decision flip probabilities predictable with small error, while the constants stay model-specific.
-
OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration
OASIS enables efficient LLM inference with non-uniform 4-bit weights and activations via precomputed Cartesian product lookup tables and a parallel outlier-compensation branch, at a reported 1.94-2.05% average accuracy drop.
-
Characterization and Mitigation of Training Instabilities in Microscaling Formats
Block-scaled MX low-precision training is unstable because quantization of tightly clustered layer-norm weights and some activations injects multiplicative gradient bias, and this can be fixed by keeping activations i...
-
QuEST: Stable Training of LLMs with 1-Bit Weights and Activations
A quantization-aware training method with Hadamard normalization and a trust gradient mask trains Llama models stably down to 1-bit weights and activations and makes 4-bit precision Pareto-optimal in accuracy per memory.
-
Reference Traces for Auditing Invisible Weight Updates and Guiding Exact-Budget Protection
Weight updates that fall below half a ULP freeze coordinates deterministically, and freeze time is predictable a priori from a high-precision trajectory and mantissa length alone.
-
Reliability Scaling Laws for Quantized Large Language Models
Reliability of quantized LLMs peaks nonlinearly at 4-bit precision under fixed total model bits, while accuracy scales monotonically, and quantization can improve robustness to natural perturbations.
-
CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training
CAGE, a curvature-aware correction that adds the quantization error to the gradient, reduces loss in low-bit quantization-aware training, letting 3-bit CAGE-trained models match 4-bit baseline-trained models.
-
LRM-1B: Towards Large Routing Model
A 1B-parameter routing model beats existing multi-task neural solvers on synthetic VRP benchmarks, and the authors fit power-law scaling curves for model size, trajectories, and compute.
-
Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models
A newly fitted scaling law with model-size-dependent data exponents predicts LLM loss more accurately than Chinchilla, including at a held-out 25.1B model.
-
Kinetics: Rethinking Test-Time Scaling Laws
A memory-aware test-time scaling law shows small models are overestimated and sparse attention is needed for efficient scaling.
-
Unified Scaling Laws for Compressed Representations
A representation capacity derived from Gaussian fitting error predicts the training efficiency of sparse, quantized, and hybrid compressed models, and this capacity approximately multiplies across combined compression types.
-
Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMs
Post-training an LLM on economic reasoning problems improves accuracy on economic benchmarks and, without game-specific training, raises its Nash equilibrium frequency and win rates in strategic games.
-
Scaling Law for Quantization-Aware Training
A unified QAT scaling law predicts 4-bit quantization error from model size, training tokens, and group size, showing activation outliers in the FC2 layer are the main W4A4 bottleneck.
-
Gaussian Weight Sampling for Scalable, Efficient and Stable Pseudo-Quantization Training
Using a rounded Gaussian noise distribution for pseudo-quantization makes low-precision FP weight training stable and cheap, matching or approaching BF16 baseline loss in LLM pretraining.
-
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
A joint scaling law fitted to over 280 models shows that, under fixed memory or total-parameter budgets, MoE models can achieve lower loss than dense models when trained on more tokens.
-
Scaling Inference-Efficient Language Models
An inference-aware scaling law that includes model aspect ratio ranks model shapes by loss and latency, producing a 1B model that is 1.8x faster without losing accuracy.
-
Physics of Skill Learning
The paper introduces Geometry, Resource, and Domino models that reproduce the sequential Domino effect in skill learning and link it to scaling laws, optimizers, and modularity.
-
INTELLECT-1 Technical Report
The INTELLECT-1 report demonstrates the first 10B-parameter language model trained over the internet across 3 continents, achieving 83-96% compute utilization with a new fault-tolerant framework called PRIME.
-
Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens
The paper derives a scaling law for quantization-induced loss increase as a function of model size, training tokens, and bit width, and uses it to argue that low-bit quantization will hurt future fully trained LLMs.
-
QS4D: Quantization-aware training for efficient hardware deployment of structured state-space sequential models
Quantization-aware training allows S4D sequence models to run at much lower precision, cutting estimated hardware costs by up to two orders of magnitude while keeping accuracy.
-
The Race to Efficiency: A New Perspective on AI Scaling Laws
A simple formula generalizes static AI scaling laws to time-varying efficiency, showing that efficiency-doubling rates around 2 per year keep loss reduction on a near-exponential path.
-
A Layered Architecture for Developing and Enhancing Capabilities in Large Language Model-based Software Systems
A layered architecture with model, inference, and application layers, plus a capability-mapping process, guides where to implement features like structured output and domain knowledge in LLM systems.
Discussion (0). Continue with ORCID to comment.