REVIEW 6 cited by
Algorithmic progress in language models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we find that the compute required to reach a set performance threshold has halved approximately every 8 months, with a 95% confidence interval of around 5 to 14 months, substantially faster than hardware gains per Moore's Law. We estimate augmented scaling laws, which enable us to quantify algorithmic progress and determine the relative contributions of scaling models versus innovations in training algorithms. Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period. Though limited by noisy benchmark data, our analysis quantifies the rapid progress in language modeling, shedding light on the relative contributions from compute and algorithms.
Forward citations
Cited by 6 Pith papers
-
Life-Cycle Emissions of AI Hardware: A Cradle-To-Grave Approach and Generational Trends
First cradle-to-grave life-cycle assessment of AI accelerators based on Google's internal fleet data, with a new CO2e/FLOP metric showing 3x improvement from TPU v4i to TPU v6e.
-
DICE: Data Influence Cascade in Decentralized Learning
DICE defines and approximates multi-hop data influence in decentralized learning, showing that influence is shaped by data, topology, and loss curvature.
-
A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search
A skill-graph random-walk model gives closed-form accuracy-versus-compute formulas for four reasoning strategies and connects them to training scaling.
-
Towards Responsible Governing AI Proliferation
The paper proposes a 'Proliferation' paradigm of AI, where small, hidden, augmented, decentralized, and open-weight models challenge compute-centric governance.
-
Technical Requirements for Halting Dangerous AI Activities
A taxonomy of compute-centric technical interventions, graded by readiness and mapped to five AI governance plans, argues that halting dangerous AI requires substantial control over AI compute.
-
Meek Models Shall Inherit the Earth
Under fixed-distribution neural scaling laws, the capability gap between state-of-the-art and low-compute AI models shrinks over time toward zero.
Discussion (0). Continue with ORCID to comment.