Pith. sign in

REVIEW 5 major objections 5 minor 35 references

Meek Models Shall Inherit the Earth

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Under a fixed-distribution next-token objective, diminishing returns to compute make frontier and cheap 'meek' models converge in capability.

desk verdict Honest and transparent, but the central convergence claim rests on an unproven smooth loss-to-capability map; the algebra is a correct corollary of Chinchilla, not an empirical discovery. read the letter →

arxiv 2507.07931 v1 pith:IPVMEIOB submitted 2025-07-10 cs.AI cs.CY

Hans Gundlach , Jayson Lynch , Neil Thompson This is my paper
classification cs.AIcs.CY
keywords AIscalinglawsdiminishingreturnsmeekmodelscapabilityconvergenceChinchillainferencesequentialprobabilityratiotestgovernance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the capability gap between the largest AI models and cheap, low-budget 'meek' models is shrinking and will keep shrinking under current scaling practices. The driving mechanism is the power-law scaling of training loss with compute: each additional unit of compute buys less loss reduction, so an exponentially growing compute budget eventually buys only a vanishing loss advantage over a fixed budget. The paper supports this with the Chinchilla exponent $\alpha=0.155$, a sigmoid mapping from loss to benchmark scores, an information-theoretic argument about the number of tokens needed to tell two models apart, and empirical trends in MMLU Pro performance at a fixed inference budget. If the model is right, capital and compute are not long-term moats for frontier AI, and governance must shift from limiting compute to other levers.

What carries the argument

The central object is the Chinchilla power-law scaling law $L_{\mathrm{opt}}(C)=AC^{-\alpha}+L_0$ with $\alpha=0.155$, which fixes how loss responds to compute. The argument is carried by the loss-gap equation that subtracts the loss of an exponentially growing compute budget from that of a fixed budget; the positive exponent in $C^{-\alpha}$ makes the gap's leading term decay as $(g_{\mathrm{alg}}g_h Tap)^{-\alpha t}$. Supporting machinery includes a sigmoid function mapping loss to benchmark accuracy (MMLU), a Sequential Probability Ratio Test in which the expected number of discrimination tokens is inversely proportional to the loss gap $\Delta L$, and an inference-cost decomposition that turns falling per-token prices into larger effective model sizes.

What would settle it

Track the gap between the best overall model and the best model available at a fixed inference budget (around $1 per million tokens) on broad benchmarks such as MMLU Pro over the next several years: if the gap grows or stays flat instead of shrinking, the convergence claim fails. A sharper falsifier would be a task family where a tiny loss difference produces a discontinuous jump in accuracy, which would show the sigmoid loss-to-capability bridge is the wrong model.

Watch

Extended reading notes

Core claim

Under a fixed-distribution next-token objective, the training-loss advantage of a model whose compute grows exponentially over a fixed-budget model is $A((g_{\mathrm{alg}}g_h)^tC_0)^{-\alpha} - A((g_{\mathrm{alg}}g_h g_i)^tC_0)^{-\alpha}$, where $\alpha=0.155$ is the Chinchilla scaling exponent. Because $\alpha$ is positive, this gap decays toward zero as $t$ grows: the shared algorithmic and hardware progress in the base term dominates the investor's extra growth factor $g_i$. The paper therefore claims that even companies scaling models exponentially faster than others end up with little capability advantage. The same conclusion is argued to hold for inference-time scaling when cheap inference compute compounds through hardware and algorithmic improvements, and for benchmark performance when loss is mapped through a monotone sigmoid. A sequential-hypothesis-testing view shows that the expected number of tokens needed to statistically distinguish a meek model from a SOTA model grows exponentially as the loss gap shrinks.

Load-bearing premise

The load-bearing premise is that the loss gap between models translates directly into a capability gap, so that exponentially small differences in loss really mean no meaningful difference in what models can do.

Editorial extensions

If this is right

  • Compute thresholds used by AI regulation, such as $10^{26}$ or $10^{25}$ flops, will become less effective because capable models will be widely available below those thresholds.
  • Money will not be a durable moat for frontier model providers; large capital advantages produce only temporary capability leads.
  • There will be a governance window in which centralized actors can develop safety experience before capable systems become ubiquitous, but also a risk of power concentration during that window.
  • Cheap inference will let most users run effectively frontier-level models, making access to powerful AI as common as computer ownership.
  • Techniques like speculative sampling, which exploit the indistinguishability of small and large models on most tokens, should become more widely used.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension: if $\alpha$ stays near 0.155, the MMLU-Pro gap at a fixed inference price should continue falling on a roughly predictable schedule; if new training paradigms lower $\alpha$, that schedule slips.
  • The convergence argument is specific to a fixed data distribution; under reinforcement learning, synthetic data, or adversarial objectives, the distribution itself moves, and the paper's Section 6 concedes that its model may not apply.
  • The same loss-gap math implies an economic prediction the authors do not foreground: the market value of access to the very largest model should decline over time unless new capability types emerge.
  • If benchmark saturation hides real divergence, the empirical support weakens; a robustness check would compare loss gaps on newly created task distributions rather than saturated benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper argues that, under a fixed-distribution next-token objective, the marginal returns to raw training compute shrink quickly enough that models trained with a fixed, modest budget ("meek" models) converge toward the capabilities of best-funded state-of-the-art models. The core model uses the Chinchilla scaling law L = A C^{-alpha} + L0 with alpha = 0.155, combined with exponential growth rates for algorithmic progress, hardware price-performance, and compute investment, to derive a training-loss gap that decays over time (Eq. 3) and an inflection time for the maximum SOTA advantage (Eq. 4). The paper extends the argument to benchmark performance via a sigmoid mapping (Eq. 6), to multi-step tasks (Eq. 7), to a sequential hypothesis-testing view (Eq. 8), and to a zero-shot inference budget model (Eq. 11). Empirical MMLU/MMLU-Pro data are presented as suggestive but inconclusive. The final sections draw governance implications about compute-based regulation, arguing that frontier capabilities will become widely accessible.

Significance. The paper's algebra is correct and transparent, and the inflection-time formula is a concrete, falsifiable prediction that could be compared with future capability-gap trajectories. If the capability-convergence claim held, it would have substantial implications for AI governance, export controls, and the economics of model development. However, the central claim is load-bearing on an assumed smooth, effectively Lipschitz mapping from training loss to capability, which the authors themselves qualify in Section 6 by acknowledging threshold-like capabilities and exponentially small loss differences that can produce large capability differentials. The paper also extrapolates the Chinchilla scaling law far outside its fitted compute range. The strengths are the clean model structure and the explicit acknowledgment of caveats; the weaknesses are the absence of fitted parameters for the sigmoid, the lack of quantification of the threshold failure mode, and the inconclusive empirical section. As a modeling argument with clear assumptions it is interesting, but as an empirical or predictive claim it currently overreaches.

major comments (5)
  1. [3.1 (Eq. 6)] The sigmoid fit parameters (A, k, x0, b) are never reported, nor is the fit quality or the data used. The shape of the capability gap over time in Figure 3, and especially the convergence time, depends directly on k: for large k the benchmark gap decays roughly as k times the loss gap, and for a hard threshold it does not decay at all. Without these values, the central claim that capability gaps shrink on the same timescale as loss gaps is unsubstantiated. Please report the fitted parameters, uncertainty, and a diagnostic (e.g., residuals or R^2), and explore the sensitivity of the convergence time to k.
  2. [6] The paper states that 'exponentially small differences in overall loss may correspond to large capability differentials' and cites the quantization model of Michaud et al. This is a direct counterexample to the inference from Eq. (3) to capability convergence used in the abstract and in Sections 3 and 4. Under a threshold capability, two models with loss gap delta_L can sit on opposite sides of a threshold, and the capability gap remains full-scale as delta_L -> 0. The manuscript does not quantify when thresholds matter or bound the loss-capability curve. To support the claim, the authors need either to restrict the claim to smooth/saturating capabilities, or to analyze a mixture model of smooth and threshold capabilities and show that the aggregate capability gap still converges.
  3. [2 (Eqs. 1-3)] The predicted loss-gap convergence is a direct algebraic consequence of the assumed power law L = A C^{-alpha} + L0 with alpha > 0 and constant growth rates; it is not an independent prediction from data. More importantly, alpha and A are fitted in Chinchilla et al. over compute budgets around 10^21 FLOPs, while the 'meek' budget C0 = $1000 is about 10^17 FLOPs, and the gap is evaluated for t -> infinity. The scaling law is used far outside its fitted range. If alpha changes with compute (e.g., as loss approaches L0) or if the power-law form breaks down, the sign or rate of the gap's decay could change. Please provide a robustness analysis with time-varying alpha or alternative scaling forms beyond the parameter sweeps in Figures 6 and 7.
  4. [5] Section 5 concedes that the data 'looks inconclusive' and that the largest observed MMLU Pro gap in 2023-2024 'falls outside the trajectory forecast by our training-based predictions.' This admission undermines the paper's empirical support for the training-based convergence claim. Please either refine the empirical analysis (e.g., accounting for reasoning-time compute and open- vs closed-weight models) or explicitly present Figure 5 as preliminary and not as model validation.
  5. [3.2 (Eq. 23)] The SPRT argument gives the number of tokens needed to distinguish two models' predictive distributions, but it inherits the same delta_L from Eq. (3) and therefore is not an independent check. It also assumes a stationary, ergodic source and an ideal observer, which the paper acknowledges. Most importantly, the SPRT result does not address capability thresholds: a tiny delta_L can be undetectable for an enormous number of tokens while a threshold capability difference persists. The section should be reframed as supporting the loss-convergence claim only, not capability convergence.
minor comments (5)
  1. [Notation] The symbol alpha is used both for the Chinchilla exponent (Eq. 1) and for the hypothesis-test error probability (Eqs. 15 and 23). Rename one of these to avoid confusion.
  2. [Abstract and Section 5] The abstract says 'we analyze empirical data on the capability difference of AI models over time,' but Section 5 concludes the data are inconclusive; consider aligning the wording with the actual strength of the empirical evidence.
  3. [Figure 3 caption] The caption reads '2 and 5 benchmark performance,' which should be 'p = 2 and p = 5 benchmark performance' to match Eq. (7).
  4. [Section 4] The sentence 'This means we are overestimating because Peff also takes into account this growth' is unclear; please rephrase to explain which quantity is overestimated and why the approximation is conservative.
  5. [Section 2, first paragraph] The source for GPU Flops per dollar is a blog post (Li 2021); consider citing a peer-reviewed or independently verified measurement to strengthen the quantitative baseline.

Circularity Check

3 steps flagged · score 6.0 of 10

The central 'prediction' of capability convergence is a corollary of the assumed Chinchilla power law and a fitted MMLU sigmoid; the one independent empirical section is inconclusive and partly contradicts the model.

  1. fitted input called prediction [Section 2, Eq. (3)]
    "Training Loss Difference = Loss Meek − Loss SOTA = A((galggh)tC0)−α − A((galgghgi)tC0)−α"

    The gap-decay conclusion is not independently derived: it is the algebraic consequence of inserting the assumed Chinchilla power law L = A C^{-alpha} + L0 for both models. Because both terms share the same A and alpha and both approach the same irreducible loss L0, their difference necessarily tends to zero as t grows. The rate is set by the externally fitted alpha = 0.155. Thus 'marginal capability returns shrink' is a restatement of the assumed power law, not a result tested by the paper's data.

  2. fitted input called prediction [Section 3.1, Eq. (6), Figures 2-3]
    "In Figure 2, we make an updated sigmoid fit of MMLU scores to loss. We determined the loss for each model using Chinchilla scaling laws from data on parameters, and data for each model from Owen [2024]."

    The benchmark-gap curves are computed by feeding the already-assumed loss gap of Eq. (3) through a sigmoid fitted to the same MMLU data. The 'prediction' of a capability gap that first rises and then converges is therefore forced by the fitted sigmoid shape and the constructed loss gap, not by an independent test. The p-step ceiling effect is likewise an artifact of the 80% MMLU ceiling in the fitted data, as the figure caption admits.

1 more flagged steps
  1. other [Section 3.2, Eq. (8), Figure 9]
    "Equation 8 shows that as the loss difference decreases, the number of tokens necessary for discrimination increases. An exponential decrease in loss leads to an exponential increase in necessary discriminator tokens."

    The SPRT token-count result is just the reciprocal of the loss gap divided by a constant: E[N] = constant / Delta L. Since the shrinking Delta L was already assumed in Eq. (3), the growth in discrimination tokens is a restatement of the same input in different units, not independent evidence. The argument also assumes that 'better prediction of text' is the capability of interest, which is the very premise under question.

full rationale

The paper's central loss-gap and benchmark-gap predictions reduce, by the paper's own equations, to the assumed power law and fitted sigmoid parameters. Equation (3) is a direct substitution of the Chinchilla law with shared A, alpha, and L0, so the convergence of the loss gap is guaranteed by the functional form rather than demonstrated empirically. Similarly, the MMLU benchmark gap curves are the fitted sigmoid evaluated at that constructed loss gap; the qualitative 'slow, then rapid, then saturating' pattern is inherited from the sigmoid. The SPRT analysis in Section 3.2 restates the same loss gap as an inverse token count, adding no independent confirmation. The empirical leaderboard comparison in Section 5 is genuinely external, but the paper itself calls the parameter-count-based data 'inconclusive' and notes that the largest observed MMLU Pro gap falls 'outside the trajectory forecast by our training-based predictions,' so it does not rescue the derivation from circularity. The paper is transparent about limitations: Section 6 admits that 'exponentially small differences in overall loss may correspond to large capability differentials,' which undermines the loss-to-capability bridge needed for the convergence conclusion, but that is a correctness risk rather than an additional circular step. The use of Ho et al. (2024) for galg is a self-citation involving author Thompson, but the qualitative convergence does not depend on that parameter's value, so it is not load-bearing. Overall, the 'predictions' are best described as consequences of the chosen scaling-law and sigmoid assumptions, hence a score of 6 rather than full circularity.

Assumptions & free parameters 10 free parameters · 7 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities; 'meek models' is a category label, not a postulated object. The central claim rests on fitted scaling-law parameters, chosen growth rates, and the assumption that loss maps monotonically to capability.

free parameters (10)
  • Chinchilla exponent alpha = 0.155
    Fitted to language model scaling data in Hoffmann et al. 2022; drives the decay rate of the loss gap in Eq. 3 and the inflection time in Eq. 4.
  • Chinchilla coefficient A = 1070
    Fitted coefficient in Eq. 1; scales the loss gap.
  • Irreducible loss L0 = 1.7
    Fitted offset in Eq. 1; cancels in the loss-difference expression.
  • Algorithmic progress growth per year galg = 2.8
    Taken from Ho et al. 2024 for language models; enters the shared growth factor for meek and SOTA compute.
  • Hardware price-performance growth gh = 1.4
    Moore's-law-inspired estimate; enters the shared growth factor.
  • SOTA compute investment growth gi = 3.57 (5/1.4)
    Computed from Sevilla et al. 2022 compute growth divided by hardware growth; sets the rate at which SOTA budgets outgrow meek budgets.
  • Initial compute budget C0 = $1000, about 1e17 flops (Li 2021)
    Chosen baseline defining t = 0 in Eq. 3; the paper also considers GPT-2-era budgets for calibration.
  • Inference efficiency growth ginf = 9x per year
    Slowest estimate from Cottier et al. 2025; drives the inference convergence model in Eqs. 9-12.
  • Fixed inference budget for meek users = 1e-8 dollars per token
    Chosen as an example budget in Section 4; determines when meek inference asymptotes to SOTA.
  • MMLU sigmoid parameters A, k, x0, b = not reported
    Fit to MMLU scores versus inferred loss in Section 3.1 (Eq. 6); the values and fit diagnostics are not given, yet these generate the p-step benchmark gap curves in Fig. 3.
assumptions (7)
  • domain assumption Chinchilla scaling law L = A C^-alpha + L0 holds for all relevant compute scales and over time
    The model extrapolates a power law fitted on a specific compute range to future and smaller budgets; Section 2 Eq. 1. If the exponent or form changes, the convergence rate changes.
  • domain assumption Loss differences are a monotonic proxy for capability differences
    Section 3 assumes benchmark performance is a sigmoid of loss and that capability is monotonic in loss; Section 6 concedes adversarial or skill-based tasks may violate this.
  • domain assumption Human text distribution is stationary and ergodic for the SPRT argument
    Section 3.2 and Appendix A rely on stationarity and ergodicity to apply Wald's theorem; the paper notes natural language can depend on arbitrarily far words, breaking this assumption.
  • domain assumption Algorithmic and hardware progress multiply effective compute for both meek and SOTA models
    Section 2 assumes both budgets grow by galg*gh, so only gi separates them. If algorithmic progress is not multiplicative in this way, the shared-growth cancellation changes.
  • domain assumption Inference compute can be traded against training compute under the Villalobos-Atkinson model, and price decline reflects parameter growth
    Sections 2.1 and 4 assume inference scaling follows this tradeoff and that a 9x price decline maps to 9x effective parameters; the paper flags inference algorithmic progress as an open question.
  • domain assumption Parameter count proxies training compute for empirical comparison
    Section 5 uses parameter count because training compute is unknown, and notes this is very rough and unreliable for overtrained or distilled models.
  • standard math Wald's stopping theorem for sequential probability ratio tests under stationary ergodic processes
    Appendix A uses it to derive Eq. 23; cited to Franken and Lisek 1982.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Meek Models Shall Inherit the Earth." pith.science (2026). https://pith.science/paper/IPVMEIOB

@misc{pith2026250707931,
  author       = {Pith},
  title        = {Pith review of: Meek Models Shall Inherit the Earth},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IPVMEIOB}},
  note         = {Machine review of arXiv:2507.07931}
}
read the original abstract

The past decade has seen incredible scaling of AI systems by a few companies, leading to inequality in AI model performance. This paper argues that, contrary to prevailing intuition, the diminishing returns to compute scaling will lead to a convergence of AI model capabilities. In other words, meek models (those with limited computation budget) shall inherit the earth, approaching the performance level of the best models overall. We develop a model illustrating that under a fixed-distribution next-token objective, the marginal capability returns to raw compute shrink substantially. Given current scaling practices, we argue that these diminishing returns are strong enough that even companies that can scale their models exponentially faster than other organizations will eventually have little advantage in capabilities. As part of our argument, we give several reasons that proxies like training loss differences capture important capability measures using evidence from benchmark data and theoretical performance models. In addition, we analyze empirical data on the capability difference of AI models over time. Finally, in light of the increasing ability of meek models, we argue that AI strategy and policy require reexamination, and we outline the areas this shift will affect.

Figures

Figures reproduced from arXiv: 2507.07931 by the authors.

Figure 1
Figure 1. Graph of loss Difference between a model with 3.6x yearly compute scaling and a meek [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Difference in MMLU performance be￾tween SOTA and meek model. 2 and 5 bench￾mark performance identify difference in capabil￾ity in tasks which involve multiple (2 to 5) correct MMLU answers. The difference in height is a di￾rect result of the 80% sigmoid fit ceiling; a higher ceiling would yield a different height relationship. We speculate that most model capabilities are monotonic functions of loss. Some models’ ca… view at source ↗
Figure 4
Figure 4. Graph of loss difference in inference vs training performance. The inference difference is [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Graph depicting the difference in MMLU-Pro score between the model with the maximum [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Loss difference between SOTA vs meek models with different levels of SOTA compute investment growth gi (3.6 is the default) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Expected number of tokens needed to differentiate text based on loss using SPRT. Bayesian Slowdown = 1 corresponds to an ideal discriminator [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 10
Figure 10. Figure 10: Model investment growth trajectories vs loss-difference. Surprisingly, large exponential [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 23 canonical work pages

  1. [1]

    Compute trends across three eras of machine learning

    Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, July 2022. doi:10.1109/ijcnn55064.2022.9891914. URL http://dx.doi.org/10.1109/IJCNN55064.2022.9891914

    arXiv 2022

  2. [2]

    Language models are few-shot learners

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877--1901, 2020

    work page 1901

  3. [3]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Martin, Sharan Narang, Lianwen Yao, Zhiyuan Zhang, and Peter Stenetorp. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [4]

    Introducing gemini: Google’s next-generation language model

    Google Research. Introducing gemini: Google’s next-generation language model. https://research.google.com/gemini, 2023

    work page 2023

  5. [5]

    The rising costs of training frontier ai models, 2024

    Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, and David Owen. The rising costs of training frontier ai models, 2024

    work page 2024

  6. [6]

    Scaling ai

    A Lohn. Scaling ai. Technical report, Technical report, Center for Security and Emerging Technology, 2023

    work page 2023

  7. [7]

    The Race to Efficiency: A New Perspective on AI Scaling Laws

    Chien-Ping Lu. The race to efficiency: A new perspective on ai scaling laws. arXiv preprint arXiv:2501.02156, 2025

    work page Pith review arXiv 2025

  8. [8]

    The Great AI Reckoning

    Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F. Manso. Deep learning’s diminishing returns, September 2021. URL https://spectrum.ieee.org/deep-learning-computational-cost. IEEE Spectrum, “The Great AI Reckoning” special report, accessed 11 May 2025

    work page 2021

Show all 35 references
  1. [9]

    Governing ai beyond the pretraining frontier

    Nicholas A Caputo. Governing ai beyond the pretraining frontier. arXiv preprint arXiv:2502.15719, 2025

    2025 arXiv

  2. [10]

    Tesla A 100 server total cost of ownership analysis, September 2021

    Chuan Li. Tesla A 100 server total cost of ownership analysis, September 2021. URL https://lambda.ai/blog/tesla-a100-server-total-cost-of-ownership. Lambda Deep Learning Blog

    2021

  3. [12]

    Reconciling kaplan and chinchilla scaling laws

    Tim Pearce and Jinyeop Song. Reconciling kaplan and chinchilla scaling laws. arXiv preprint arXiv:2406.12907, 2024

    2024 arXiv

  4. [13]

    Trends in gpu price-performance, 2022

    Marius Hobbhahn and Tamay Besiroglu. Trends in gpu price-performance, 2022. URL https://epoch.ai/blog/trends-in-gpu-price-performance. Accessed: 2025-05-12

    2022

  5. [14]

    40 years of microprocessor trend data

    Karl Rupp. 40 years of microprocessor trend data. https://www.karlrupp.net/2015/06/40-years-of-microprocessor-trend-data/, June 2015. Accessed: 2025-05-09

    2015

  6. [15]

    Algorithmic progress in language models, 2024

    Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carl Guo, David Atkinson, Neil Thompson, and Jaime Sevilla. Algorithmic progress in language models, 2024. URL https://arxiv.org/abs/2403.05812

    2024 arXiv

  7. [16]

    How much did gpt-2 cost? https://umatechnology.org/how-much-did-gpt-2-cost/, January 2025

    UMATechnology . How much did gpt-2 cost? https://umatechnology.org/how-much-did-gpt-2-cost/, January 2025. Accessed: 2025-05-09

    2025

  8. [17]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361

    2020 arXiv

  9. [18]

    How far can reasoning models scale?, May 2025

    Josh You. How far can reasoning models scale?, May 2025. URL https://epoch.ai/gradient-updates/how-far-can-reasoning-models-scale. Newsletter article

    2025

  10. [19]

    Trading off compute in training and inference, 2023

    Pablo Villalobos and David Atkinson. Trading off compute in training and inference, 2023. URL https://epochai.org/blog/trading-off-compute-in-training-and-inference. Accessed: 2024-07-24

    2023

  11. [20]

    Scaling vision transformers, 2022

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, 2022. URL https://arxiv.org/abs/2106.04560

    2022 arXiv

  12. [21]

    How predictable is language model benchmark performance?, 2024

    David Owen. How predictable is language model benchmark performance?, 2024. URL https://arxiv.org/abs/2401.04757

    2024 arXiv

  13. [22]

    Mapping global dynamics of benchmark creation and saturation in artificial intelligence

    Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Jan Brauner, and Matthias Samwald. Mapping global dynamics of benchmark creation and saturation in artificial intelligence. Nature Communications, 13 0 (1): 0 6793, 2022

    2022

  14. [23]

    The direct approach, 2023

    Matthew Barnett and Tamay Besiroglu. The direct approach, 2023. URL https://epochai.org/blog/the-direct-approach. Accessed: 2024-09-20

    2023

  15. [24]

    Daniel Jurafsky and James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. 3rd edition, Jan 2025. URL https://web.stanford.edu/ jurafsky/slp3/ed3book_Jan25.pdf. On...

    2025

  16. [25]

    Alisa: Accelerating large language model inference via sparsity-aware kv caching

    Youpeng Zhao, Di Wu, and Jun Wang. Alisa: Accelerating large language model inference via sparsity-aware kv caching. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 1005--1017. IEEE, 2024

    2024

  17. [26]

    Accelerating large language model decoding with speculative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    2023 arXiv

  18. [27]

    Llm inference prices have fallen rapidly but unequally across tasks, 2025

    Ben Cottier, Ben Snodin, David Owen, and Tom Adamczewski. Llm inference prices have fallen rapidly but unequally across tasks, 2025. URL https://epoch.ai/data-insights/llm-inference-price-trends. Accessed: 2025-05-21

    2025

  19. [28]

    Llm leaderboard - compare gpt-4o, llama 3, mistral, gemini & other models, 2025

    Artificial Analysis . Llm leaderboard - compare gpt-4o, llama 3, mistral, gemini & other models, 2025. URL https://artificialanalysis.ai/leaderboards/models. Accessed: 2025-05-20

    2025

  20. [29]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022 b

    2022 arXiv

  21. [30]

    Data on notable ai models, 6 2024

    Epoch AI . Data on notable ai models, 6 2024. URL https://epoch.ai/data/notable-ai-models. Accessed: 2025-05-21

    2024

  22. [31]

    Michaud, Ziming Liu, Uzay Girit, and Max Tegmark

    Eric J. Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling, 2024. URL https://arxiv.org/abs/2303.13506

    2024 arXiv

  23. [32]

    Andy L. Jones. Scaling scaling laws with board games, 2021. URL https://arxiv.org/abs/2104.03113

    2021 arXiv

  24. [33]

    Scaling laws for a multi-agent reinforcement learning model

    Oren Neumann and Claudius Gros. Scaling laws for a multi-agent reinforcement learning model. arXiv preprint arXiv:2210.00849, 2022

    2022 arXiv

  25. [34]

    Can ai scaling continue through 2030?, 2024

    Jaime Sevilla, Tamay Besiroglu, Ben Cottier, Josh You, Edu Roldán, Pablo Villalobos, and Ege Erdil. Can ai scaling continue through 2030?, 2024. URL https://epoch.ai/blog/can-ai-scaling-continue-through-2030. Accessed: 2025-05-12

    2024

  26. [35]

    Robert D. Nowak. Lecture 9: Sequential testing. https://nowak.ece.wisc.edu/ece830/ece830_fall11_lecture9.pdf, 2011. ECE 830: Statistical Signal Processing, Fall 2011, University of Wisconsin–Madison

    2011

  27. [36]

    Franken and B

    P. Franken and B. Lisek. On wald's identity for dependent variables. Z. Wahrscheinlichkeitstheorie verw Gebiete, 60: 0 143--150, June 1982. doi:10.1007/BF00531818. URL https://doi.org/10.1007/BF00531818

    1982 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.