The Race to Efficiency: A New Perspective on AI Scaling Laws

Chien-Ping Lu

REVIEW 4 major objections 5 minor 2 cited by

The Race to Efficiency: A New Perspective on AI Scaling Laws

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper derives a relative-loss equation showing that if hardware and algorithmic efficiency keeps doubling, AI training loss can keep falling exponentially.

desk verdict A correct but modest re-parameterization of scaling laws into time; useful as a planning heuristic, not a discovery. read the letter →

arxiv 2501.02156 v3 pith:GADPWYC7 submitted 2025-01-04 cs.LG cs.AIcs.PF

Chien-Ping Lu This is my paper · ORCID

classification cs.LGcs.AIcs.PF

keywords AIscalinglawsrelative-lossequationefficiencydoublingcompute-optimaltraininglossdiminishingreturnslargelanguagemodelstime-aware

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Classical AI scaling laws say training loss falls only as a power of compute, which makes further progress look impossibly expensive; this paper argues that those laws are missing a time axis. It introduces a relative-loss equation in which an annual efficiency-doubling rate $\gamma$ feeds new compute into the power law, so that when efficiency compounds, the loss decays exponentially at rate $\kappa\gamma$. With the small $\kappa$ typical of large language models, sustained efficiency gains turn a 'diminishing returns' story into a 'race to efficiency' that can keep progress brisk for years. The payoff is a concrete planning tool: it turns questions about giant GPU fleets and research investment into a single measurable number, the efficiency-doubling rate.

What carries the argument

The central object is the relative-loss equation $R(t) = (1 + (2^{\gamma t}-1)/(\gamma \ln 2 \times 1\text{ yr}))^{-\kappa}$, which extends the static power law by replacing the one-shot compute budget with an integral of time-varying efficiency $E(t) = E_0 2^{\gamma t}$ times a constant mean-field power $P_0$. This equation is what turns a snapshot into a compounding process: each year's efficiency gains add to cumulative compute, and because the added compute is exponential, the loss falls exponentially at rate $\kappa\gamma$. Two modeling choices carry the argument: the one-year baseline $C_0 = E_0 P_0 \times 1\text{ yr}$, and the mean-field power assumption, which the paper explicitly treats as an upper bound on realizable progress. A supporting mechanism is the definition of logical compute as dense, full-precision $6ND$ FLOPs, which keeps $\kappa$ invariant to sparsity and low-precision optimizations and lets those gains appear instead as higher efficiency.

What would settle it

Take several consecutive large-model training runs from one lab with reported losses, compute, power, and training time. Fit the relative-loss equation to each run and estimate the implied $\gamma$; if $\gamma$ is not roughly constant across generations, or if the observed losses deviate systematically from exponential decay at rate $\kappa\gamma$, the framework's central premise fails.

Watch

Extended reading notes

Core claim

Starting from the classical relation $L \propto C^{-\kappa}$ and assuming cumulative compute grows as $C(t) = C_0 + E_0 P_0 (2^{\gamma t}-1)/(\gamma \ln 2)$, the paper derives the relative-loss equation $R(t) = [1 + (2^{\gamma t}-1)/(\gamma \ln 2 \times 1\text{ yr})]^{-\kappa}$. The paper's central discovery is the long-time behavior $R(t) \propto 2^{-\kappa\gamma t}$: the training-loss curve becomes exponential in time, with the rate set by the product of the scaling exponent and the efficiency-doubling rate. That means a small $\kappa \approx 0.05$ does not doom progress as long as $\gamma$ is large enough; the paper's illustrations put $\gamma=2$ (doubling twice per year) in reach of today's reported hardware and algorithmic gains. When $\gamma=0$, the equation reduces to the static law $L(t) = L_0(1+t/1\text{ yr})^{-\kappa}$, and the paper shows that reducing loss from $1.0$ to $0.68$ nats/token then demands roughly a $3000\times$ compute increase, about 3,000 years on one 100,000-GPU cluster or a fleet near 300 million GPUs, which is why it frames efficiency gains as a necessity, not an optional extra.

Load-bearing premise

The load-bearing premise is that all hardware, software, and algorithmic gains can be summarized by a single annual efficiency-doubling rate, and that each doubling adds usable compute without changing the underlying relationship between loss and compute or the optimality of training.

Editorial extensions

If this is right

If the relative-loss equation is right, a flat efficiency rate ($\gamma=0$) pushes any substantial loss reduction beyond practical reach: the paper's worked example needs a $3000\times$ compute increase to move from $1.0$ to $0.68$ nats/token.
At $\gamma \ge 2$ (efficiency doubling every six months), a small $\kappa$ still yields exponential loss decay, compressing the time-to-target from roughly 20 years at $\gamma=0.5$ to well under a decade.
The framework gives organizations a measurable planning target: set an explicit efficiency-doubling rate and check whether hardware refreshes, algorithm changes, and data-pipeline improvements actually compound at that rate.
The case studies suggest that a smaller fleet with a high $\gamma$ (the 'Turtle') can match a larger fleet with a lower $\gamma$ (the 'Hare') over a five-year horizon, shifting the investment question from raw GPUs to sustained innovation.
Because the mean-field power assumption is an upper bound, any real fluctuation in power or suboptimal allocation will slow progress below the predicted curve.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the paper treats $\gamma$ as an exogenous constant, but nothing guarantees it stays constant; if optimization headroom shrinks over time, the predicted exponential decay would level off into an S-shaped curve, and the framework would need a time-dependent $\gamma(t)$ to remain predictive.
Editorial inference: the claim that $\kappa$ is invariant across model generations is a strong empirical assumption that can be tested by fitting the relative-loss equation to successive compute-optimal training runs from the same lab and checking whether the implied $\gamma$ and $\kappa$ are stable.
Editorial inference: the definition of logical compute invites a practical measurement program: compute 'efficiency' as logical FLOPs divided by realized GPU-hours on a fixed hardware baseline, then estimate $\gamma$ from the growth rate of that ratio across published training runs.
Editorial inference: the framework implies a direct investment comparison: one year of efficiency doubling at rate $\gamma$ is worth a $2^{\gamma}$-fold increase in compute, so a decision to front-load GPUs can be weighed against an R&D program that raises $\gamma$.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper does one thing and does it cleanly: it substitutes an exponential efficiency schedule into the standard scaling law L ∝ C^{-κ} and integrates to get a closed-form relative-loss equation. The algebra in Section 3.4 is correct, and the γ→0 limit is handled properly. That is genuinely new in form, though it is a re-parameterization rather than a new empirical result.

What deserves credit: the author is unusually honest. Section 7 explicitly states that the equation assumes compute-optimal usage, that validation is missing, and that the framework is a best-case trajectory. The sensitivity analysis and the Turtle/Hare case studies are useful for thinking about front-loaded GPUs versus efficiency gains. The paper does not hide its own caveats.

The soft spots are real, and the stress-test note lands. The load-bearing assumption is that κ stays fixed as efficiency accumulates, but Appendix A only shows that κ = α/(α+β) in a dense, full-precision setting; it does not address MoE, precision changes, or data-quality shifts. The papers the author cites on sparsity and precision show the effective exponent changes with the optimization strategy. So the quantitative outputs in Table 2 are illustrative, not trustworthy forecasts. The mean-field upper-bound claim also does not hold: fluctuating power could in principle be scheduled when efficiency is higher, so constant power is not necessarily optimal. That is a minor flaw, but it undermines the claim of an optimistic baseline. The γ=0 limit overstates the contrast: continuing to train a fixed model on fixed hardware is not compute-optimal scaling, so the 3,000-year figure is a strawman rather than a real implication of classical scaling laws.

The paper is what it says it is: a conceptual extension. It gives labs a back-of-the-envelope tool for thinking about efficiency-doubling rates, but it does not advance the empirical science of scaling. The citation pattern is fine; the author engages with the relevant literature and does not hide the lack of data fitting.

For whom? Practitioners and researchers who want a simple way to frame efficiency-vs-loss tradeoffs over time. Not a major result, but a legitimate planning perspective.

Recommendation: yes, send it to peer review—as a position or conceptual paper. A serious referee could push the author to validate the κ invariance assumption and temper the mean-field claim. With revision, it could be a useful contribution to the scaling-laws discussion.

Referee Report

4 major / 5 minor

Summary. The paper extends classical AI scaling laws to include time-dependent efficiency gains. It introduces an efficiency-doubling rate γ and derives a relative-loss equation R(t) = [1 + (2^{γt} - 1)/(γ ln 2 × 1 yr)]^{-κ}, arguing that with γ > 0 the training loss decays exponentially at rate κγ. The main body provides the derivation, analyzes limiting and asymptotic behaviors, performs a sensitivity analysis, and presents thought experiments contrasting front-loaded GPU investment with sustained efficiency improvements, including a case study involving DeepSeek-V3 and Llama 3. The paper does not fit any parameters to data; it is a conceptual framework built on the classical power-law scaling assumption L ∝ C^{-κ}.

Significance. If the underlying invariance assumptions held, the framework would provide a clean and intuitive way to think about how hardware and algorithmic progress can offset diminishing returns in AI scaling. The algebraic derivation of Eq. (3.4) is straightforward and correct given its premises, and the connection to a Moore's-Law-style doubling rate is pedagogically appealing. The paper is also transparent about several limitations in Section 7, and the inclusion of a worked case study helps ground the discussion. However, the contribution is primarily a reformulation rather than a new empirical result; its quantitative predictions depend on unvalidated assumptions about the constancy of the scaling exponent κ across different efficiency gains and about the applicability of scaling laws to cumulative cross-project compute. As a perspective piece the paper has merit, but as a predictive framework it requires substantial additional support or a much more careful statement of its domain of validity.

major comments (4)

[Section 3.4, Eq. (3.4), and Appendix A] The derivation assumes a single invariant exponent κ applies to all compute accumulated through hardware, software, and algorithmic efficiency gains. Appendix A's invariance argument covers only the dense, full-precision loss form L(N,D)=A N^{-α}+B D^{-β}+E0 under compute-optimal allocation; it does not address Mixture-of-Experts architectures, reduced-precision training, or data-quality improvements. The paper itself cites Refs. [14] and [15], which show that effective scaling exponents change with sparsity and precision. Consequently, the quantitative predictions in Section 4.2 and Table 2 are conditional on an unvalidated invariance that may be empirically false. The authors should either provide supporting evidence that κ is invariant across the relevant generations or reframe the claims as a model under an explicit and prominently stated invariance assumption.
[Section 4.1 and Section 5.1] The γ=0 limit is interpreted as 'running the same hardware and software for an additional time t' with L(t)=L0(1+t/1yr)^{-κ}. Classical scaling laws apply to compute-optimal training runs in which model size, dataset size, and compute are balanced; continuing to train a fixed model on a fixed dataset does not reduce loss as a power law. Thus the 3,000-year and 3,000×-compute figures in Section 5.1 are not consequences of classical scaling laws, and the stark contrast used to motivate the necessity of γ>0 is overstated. The paper's own Section 7 acknowledges the compute-optimal assumption, but the interpretation in Section 4.1 appears to contradict it.
[Section 3.3] The claim that the mean-field assumption P(τ)≈P0 represents 'an upper bound on performance' is not correct. For a fixed total energy budget and a monotonically increasing efficiency E(τ), concentrating power at later times yields more cumulative compute than constant power, so constant power is a lower bound on achievable compute, not an upper bound. This error affects the 'optimistic baseline' interpretation in Section 3.3 and the practical significance discussion in Section 3.4. The derivation of the relative-loss equation does not depend on this claim, but the surrounding interpretation does.
[Table 2] The 'Unfold in Space' row is internally inconsistent with the text. The table lists L0=0.68, R(t)=1.00, and time ~1 yr for a scenario targeting L=0.68, which implies no training is needed to reach the target. Section 5.1, however, describes starting from L0=1.0 and requiring a 3,000× compute increase to reach L=0.68 within one year. The row should read L0=1.00, R(t)=0.68, and time ~1 yr.

minor comments (5)

[Section 5.2] The description of Turtle as 'γ = 3.0 (tripling annually)' is a misstatement: γ=3 means three doublings per year, i.e., an 8× annual increase, not 'tripling annually.' Similarly, describing Hare's γ=2.0 as 'a still-robust annual doubling rate' is imprecise since γ=2 corresponds to doubling twice per year.
[Table 3] The 'Logical Compute (PFLOPs)' entries appear to be off by several orders of magnitude. For DeepSeek-V3, 6×671B×14.8T ≈ 5.96×10^25 FLOPs = 5.96×10^10 PFLOPs, not 5.95×10^15 PFLOPs; similarly for Llama 3. The relative efficiency ratio of 17× is unaffected, but the absolute units should be corrected to avoid confusion.
[Section 1, Eq. (1.1)] The relative-loss equation is presented in the introduction before κ and γ are defined. Consider either defining these variables in the introduction or moving the equation to Section 3, where the notation is formally introduced.
[Section 4.4] The target range is written as 'y ∈ [0.5, 09]'; this appears to be a typo and should likely read '[0.5, 0.9]'.
[Section 5.1 and Table 2 footnote] The footnote says that 'to reduce loss below 0.68 with γ=0 requires exponentially more GPUs,' but the table's target is exactly 0.68. The wording should be clarified to indicate whether the target is '≤0.68' or 'below 0.68,' and the phrase 'exponentially more' is imprecise for a power-law relationship.

Circularity Check

0 steps flagged · score 0.0 of 10

Derivation is a transparent algebraic extension of external scaling laws; no fitted parameter is relabeled as a prediction.

full rationale

No load-bearing circular step is present. The relative-loss equation is obtained in Section 3.4 by starting from the externally established classical scaling relation L ∝ C^{-κ}, introducing an explicit definition E(t)=E0·2^{γt}, forming C(t)=C0+∫E(τ)P(τ)dτ under the mean-field assumption, and then performing the integral to arrive at R(t)=[1+(2^{γt}-1)/(γ ln2 · 1yr)]^{-κ}. This is an algebraic consequence of stated assumptions, not an output that is equivalent to an input by construction; κ and γ are not fitted to the relative-loss curve, and the time-to-target values in Table 2 are conditional inversions of the defining equation rather than empirical confirmations. The paper explicitly labels these as illustrative scenarios and, in Section 7, acknowledges that the equation assumes compute-optimal usage and calls for external empirical validation. Citations are to external scaling-law work, not self-citations carrying the argument. The Appendix's invariance claim is an assumption about the behavior of κ under optimizations, not a circular reuse of the paper's own conclusion. Thus the derivation is self-contained conditional mathematics, and no circularity score above 0 is warranted.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The model's predictions are driven by k and g, both supplied from outside the paper. No parameters are fitted to data in this work, so the ledger is short; the main burden is the assumption that the static scaling law remains valid over multi-year horizons.

free parameters (3)

Scaling exponent k = 0.048 (used in examples; from Kaplan et al., Hoffmann et al.)
The time-to-target calculations depend on the exponent from classical scaling laws; the paper uses 0.048 as typical but does not re-fit it.
Efficiency-doubling rate g = 0, 0.5, 2, 3 in scenarios
Free parameter chosen by the user; the paper's central message is that higher g sustains progress, but g is not derived or measured.
Baseline loss L0 = 1.0 nats/token in the illustrative example
Scenario assumption; affects absolute time-to-target numbers.

assumptions (5)

domain assumption Classical scaling law L proportional to C^(-k) holds under compute-optimal training (Kaplan et al. 2020, Hoffmann et al. 2022).
Used as the starting point for the derivation in Section 3.4; the exponent is taken from prior empirical fits.
ad hoc to paper Efficiency E(t) grows exactly exponentially as E0 2^(gt) with constant rate g.
Equation (3.1); the paper models continuous exponential improvement, ignoring discrete jumps and saturations.
domain assumption Power usage P(t) is constant at mean value P0, described as an upper bound.
Section 3.3 mean-field assumption; the direction of the bound is not rigorously justified.
domain assumption Compute-optimal usage is maintained at all times; each unit of efficiency directly reduces loss as predicted by the static law.
Stated in Section 7 Limitations; the equation is a best-case trajectory.
ad hoc to paper The scaling exponent k is invariant to sparsity, precision, and architecture optimizations (logical compute invariance).
Appendix A; required so that efficiency gains appear only in g rather than changing k.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Race to Efficiency: A New Perspective on AI Scaling Laws." pith.science (2026). https://pith.science/paper/GADPWYC7

@misc{pith2026250102156,
  author       = {Pith},
  title        = {Pith review of: The Race to Efficiency: A New Perspective on AI Scaling Laws},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GADPWYC7}},
  note         = {Machine review of arXiv:2501.02156}
}

read the original abstract

As large-scale AI models expand, training becomes costlier and sustaining progress grows harder. Classical scaling laws (e.g., Kaplan et al. (2020), Hoffmann et al. (2022)) predict training loss from a static compute budget yet neglect time and efficiency, prompting the question: how can we balance ballooning GPU fleets with rapidly improving hardware and algorithms? We introduce the relative-loss equation, a time- and efficiency-aware framework that extends classical AI scaling laws. Our model shows that, without ongoing efficiency gains, advanced performance could demand millennia of training or unrealistically large GPU fleets. However, near-exponential progress remains achievable if the "efficiency-doubling rate" parallels Moore's Law. By formalizing this race to efficiency, we offer a quantitative roadmap for balancing front-loaded GPU investments with incremental improvements across the AI stack. Empirical trends suggest that sustained efficiency gains can push AI scaling well into the coming decade, providing a new perspective on the diminishing returns inherent in classical scaling.

Figures

Figures reproduced from arXiv: 2501.02156 by the authors.

**Figure 2.** Sensitivity to baseline perturbations. The horizontal axis shows τ in years, with τ = −1 yr representing a scenario where the baseline effectively vanishes. Even under large deviations, higher γ values preserve robust predictions for time-to-target. If y is a target relative loss (say, R(t(τ )) = y), then near τ = 0, the time-to-target t(τ ) is approximately 1/(γ ln 2), regardless of y. This implies a consistent fi… view at source ↗

**Figure 3.** Time horizons vs. efficiency-doubling rate. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance
cs.LG 2025-05 conditional novelty 6.0 of 10

MoEQuant combines expert-balanced self-sampled calibration data with affinity-weighted quantization error to improve low-bit quantization of MoE LLMs.
Meek Models Shall Inherit the Earth
cs.AI 2025-07 conditional novelty 5.0 of 10

Under fixed-distribution neural scaling laws, the capability gap between state-of-the-art and low-compute AI models shrinks over time toward zero.

Reference graph

Works this paper leans on

26 extracted references · 13 canonical work pages · cited by 2 Pith papers

[14]

Unified scaling laws for routed language models,

A. Clark, D. de Las Casas, A. Guy, A. Mensch, M. Paganini, J. Hoffmann, B. Damoc, B. Hechtman, T. Cai, S. Borgeaud et al. , “Unified scaling laws for routed language models,” in Proceedings of the International Conference on Machine Learning . PMLR, 2022, pp. 4057–4086. [Online]. Available: https://proceedings.mlr.press/v162/clark22a.html

work page 2022
[1]

AI with reasoning power will be less predictable, Ilya Sutskever says,

Reuters, “AI with reasoning power will be less predictable, Ilya Sutskever says,” Reuters Technology, 2024. [Online]. Available: https://www.reuters.com/technology/artificial- intelligence/ai-with-reasoning-power-will-be-less-predictable-ilya-sutskever-says-2024-12-14/

work page 2024
[2]

OpenAI is reportedly struggling to improve its next big AI model. It’s a warning for the entire AI industry

Business Insider, “OpenAI is reportedly struggling to improve its next big AI model. It’s a warning for the entire AI industry.” Business Insider , 2024. [Online]. Available: https: //www.businessinsider.com/openai-orion-model-scaling-law-silicon-valley-chatgpt-2024-11

work page 2024
[3]

Has AI progress really slowed down?

Time Magazine, “Has AI progress really slowed down?” Time, 2024. [Online]. Available: https://time.com/7178328/is-ai-progress-slowing-down/

work page arXiv 2024
[4]

NVIDIA boss Jensen Huang predicts computing power will in- crease a ’millionfold’ in a decade,

Business Insider, “NVIDIA boss Jensen Huang predicts computing power will in- crease a ’millionfold’ in a decade,” Business Insider , 2024. [Online]. Avail- able: https://www.businessinsider.com/nvidia-jensen-huang-predicts-increase-computing- power-ai-scaling-2024-11

work page 2024
[5]

Microsoft defies the AI plateau, pushing boundaries with new scaling laws,

Benzinga, “Microsoft defies the AI plateau, pushing boundaries with new scaling laws,” Benzinga, 2024. [Online]. Available: https://www.benzinga.com/24/11/42088290/microsoft- defies-the-ai-plateau-pushing-boundaries-with-new-scaling-laws

work page 2024
[6]

There’s no evidence scaling laws have begun to stop, former Google CEO claims AI systems will be 100 times more powerful,

W. Central, “There’s no evidence scaling laws have begun to stop, former Google CEO claims AI systems will be 100 times more powerful,” Windows Central , 2024. [Online]. Available: https://www.windowscentral.com/software-apps/theres-no-evidence-scaling-laws- have-begun-to-stop-former-google-ceo-claims-ai-systems-will-be-100-times-more-powerful

work page 2024
[7]

Cramming more components onto integrated circuits,

G. E. Moore, “Cramming more components onto integrated circuits,” Electronics, vol. 38, pp. 114–117, 1965

work page 1965

Show all 26 references

[8]

Design of ion-implanted MOSFETs with very small gate lengths,

R. H. Dennard, E. Gaensly, and F. Spreen, “Design of ion-implanted MOSFETs with very small gate lengths,” IEEE transactions on electron devices , vol. 21, no. 1, pp. 637–644, 1974

1974
[9]

Scaling laws for neural language models,

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020. [Online]. Available: https://arxiv.org/abs/2001.08361

2001 arXiv
[10]

Training compute-optimal large language models,

J. Hoffmann, S. Borgeaud, A. Mensch, T. Cai, E. Rutherford, D. de Las Casas, L. M. Hendricks, J. Welbl, A. Clark, J. Bewersdorf et al. , “Training compute-optimal large language models,” arXiv preprint arXiv:2203.15556 , 2022. [Online]. Available: https://arxiv.org/abs/2203.15556

2022 arXiv
[11]

Language models are few-shot learners,

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, 2020. [Online]. Available: https://arxiv.org/abs/2005.14165

2005 arXiv
[12]

Beyond Chinchilla-optimal: Accounting for inference-time in language model scaling,

N. Sardana, J. Portes, S. Doubov, and J. Frankle, “Beyond Chinchilla-optimal: Accounting for inference-time in language model scaling,” arXiv preprint arXiv:2211.02011 , 2024. [Online]. Available: https://arxiv.org/abs/2401.00448 17

2024 arXiv
[13]

Scaling LLM test-time compute optimally can be more effective than scaling model parameters,

C. Snell, J. Lee, K. Xu, and A. Kumar, “Scaling LLM test-time compute optimally can be more effective than scaling model parameters,” arXiv preprint arXiv:2408.03314 , 2023. [Online]. Available: https://arxiv.org/abs/2408.03314

2023 arXiv
[16]

Upcycling large language models into mixture of experts,

E. He, A. Khattar, R. Prenger, V. Korthikanti, Z. Yan, T. Liu, S. Fan, A. Aithal, M. Shoeybi, and B. Catanzaro, “Upcycling large language models into mixture of experts,” arXiv preprint arXiv:2410.07524, 2024. [Online]. Available: https://arxiv.org/abs/2410.07524

2024 arXiv
[17]

Llama 3 meets MoE: Efficient upcycling,

A. Vavre, E. He, D. Liu, Z. Yan, J. Yang, N. Tajbakhsh, and A. Aithal, “Llama 3 meets MoE: Efficient upcycling,” arXiv preprint arXiv:2412.09552v1 , 2024. [Online]. Available: https://arxiv.org/abs/2412.09952v1

2024
[18]

DeepSeek-V3: An ultra-large open-source AI model,

DeepSeek-AI, “DeepSeek-V3: An ultra-large open-source AI model,” arXiv preprint arXiv:2412.19437, 2024. [Online]. Available: https://arxiv.org/abs/2412.19437

2024 arXiv
[19]

The llama 3 herd of models,

A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 . [Online]. Available: https://arxiv.org/abs/2407.21783

arXiv
[20]

Can innovation curb AI’s hunger for power?

NVIDIA Blog, “Can innovation curb AI’s hunger for power?” https://blogs.nvidia.com/blog/ ai-energy-efficiency/, 2024, accessed: December 29, 2024

2024
[21]

[Online]

NVIDIA, NVIDIA Puts Grace Blackwell on Every Desk and at Every AI Developer’s Fin- gertips, 2025. [Online]. Available: https://nvidianews.nvidia.com/news/nvidia-puts-grace- blackwell-on-every-desk-and-at-every-ai-developers-fingertips

2025
[22]

R. N. Foster, Innovation: The Attacker’s Advantage . New York, NY: Summit Books, 1986

1986
[24]

Sparse upcycling: Training mixture-of-experts from dense checkpoints,

A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. Riquelme, B. Mustafa, J. Ainslie, Y. Tay, M. Dehghani, and N. Houlsby, “Sparse upcycling: Training mixture-of-experts from dense checkpoints,” arXiv preprint arXiv:2212.05055 , 2022. [Online]. Available: https://arxiv.org/abs/2212.05055

2022 arXiv
[25]

Available: https://arxiv.org/abs/2408.00724

[Online]. Available: https://arxiv.org/abs/2408.00724

arXiv
[26]

Quiet-STaR: Language models can teach themselves to think before speaking,

E. Zelikman, Y. Wu, and N. D. Goodman, “Quiet-STaR: Language models can teach themselves to think before speaking,” arXiv preprint arXiv:2403.09629 , 2024. [Online]. Available: https://arxiv.org/abs/2403.09629 A Invariance of Logical Compute This appendix explains why we defin...

2024 arXiv
[27]

STaR: Bootstrapping reasoning with reasoning,

E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman, “STaR: Bootstrapping reasoning with reasoning,” arXiv preprint arXiv:2203.14465 , 2022. [Online]. Available: https: //arxiv.org/abs/2203.14465 18

2022 arXiv
[2024]

Available: https://arxiv.org/abs/2411.04330

[Online]. Available: https://arxiv.org/abs/2411.04330

arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [14]

Unified scaling laws for routed language models,

A. Clark, D. de Las Casas, A. Guy, A. Mensch, M. Paganini, J. Hoffmann, B. Damoc, B. Hechtman, T. Cai, S. Borgeaud et al. , “Unified scaling laws for routed language models,” in Proceedings of the International Conference on Machine Learning . PMLR, 2022, pp. 4057–4086. [Online]. Available: https://proceedings.mlr.press/v162/clark22a.html

work page 2022

[2] [1]

AI with reasoning power will be less predictable, Ilya Sutskever says,

Reuters, “AI with reasoning power will be less predictable, Ilya Sutskever says,” Reuters Technology, 2024. [Online]. Available: https://www.reuters.com/technology/artificial- intelligence/ai-with-reasoning-power-will-be-less-predictable-ilya-sutskever-says-2024-12-14/

work page 2024

[3] [2]

OpenAI is reportedly struggling to improve its next big AI model. It’s a warning for the entire AI industry

Business Insider, “OpenAI is reportedly struggling to improve its next big AI model. It’s a warning for the entire AI industry.” Business Insider , 2024. [Online]. Available: https: //www.businessinsider.com/openai-orion-model-scaling-law-silicon-valley-chatgpt-2024-11

work page 2024

[4] [3]

Has AI progress really slowed down?

Time Magazine, “Has AI progress really slowed down?” Time, 2024. [Online]. Available: https://time.com/7178328/is-ai-progress-slowing-down/

work page arXiv 2024

[5] [4]

NVIDIA boss Jensen Huang predicts computing power will in- crease a ’millionfold’ in a decade,

Business Insider, “NVIDIA boss Jensen Huang predicts computing power will in- crease a ’millionfold’ in a decade,” Business Insider , 2024. [Online]. Avail- able: https://www.businessinsider.com/nvidia-jensen-huang-predicts-increase-computing- power-ai-scaling-2024-11

work page 2024

[6] [5]

Microsoft defies the AI plateau, pushing boundaries with new scaling laws,

Benzinga, “Microsoft defies the AI plateau, pushing boundaries with new scaling laws,” Benzinga, 2024. [Online]. Available: https://www.benzinga.com/24/11/42088290/microsoft- defies-the-ai-plateau-pushing-boundaries-with-new-scaling-laws

work page 2024

[7] [6]

There’s no evidence scaling laws have begun to stop, former Google CEO claims AI systems will be 100 times more powerful,

W. Central, “There’s no evidence scaling laws have begun to stop, former Google CEO claims AI systems will be 100 times more powerful,” Windows Central , 2024. [Online]. Available: https://www.windowscentral.com/software-apps/theres-no-evidence-scaling-laws- have-begun-to-stop-former-google-ceo-claims-ai-systems-will-be-100-times-more-powerful

work page 2024

[8] [7]

Cramming more components onto integrated circuits,

G. E. Moore, “Cramming more components onto integrated circuits,” Electronics, vol. 38, pp. 114–117, 1965

work page 1965