Pith. sign in

REVIEW 4 major objections 6 minor 52 references

Rethinking LLM Advancement: Compute-Dependent and Independent Paths to Progress

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that LLM gains split into compute-dependent and compute-independent algorithmic advances, and that compute-independent advances keep producing measurable progress even when hardware is restricted.

desk verdict A genuinely useful compute-dependent/independent framework with a solid literature synthesis, but the empirical validation as reported has an internal contradiction in Table 3 that needs fixing before the experimental claims can be trusted. read the letter →

arxiv 2505.04075 v2 pith:OGMYVV2H submitted 2025-05-07 cs.LG cs.AI

Jack Sanderson , Teddy Foley , Spencer Guo , Anqi Qu , Henry Josephson This is my paper · ORCID
classification cs.LGcs.AI
keywords compute-equivalentgaincompute-dependentadvancementscompute-independentalgorithmicprogressLLMgovernancehardwarecontrolsscalinglawsnanoGPT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that algorithmic progress in large language models comes in two distinct modes: compute-dependent advances, whose benefits materialize mainly when training compute is large, and compute-independent advances, which improve efficiency at every compute scale. The authors quantify each advance with compute-equivalent gain (CEG), the factor by which an improved algorithm reduces the compute needed to reach a baseline performance level. In nanoGPT experiments at 50M and 110M parameters, compute-independent methods such as layer normalization, rotary position embeddings, and FlashAttention improved efficiency at both scales, with the three combined reaching CEG up to $3.5\times$; compute-dependent methods (multi-query attention, sparse attention) hurt at the smaller scale and approached parity at the larger scale. The policy conclusion is that hardware restrictions can slow frontier development but cannot prevent algorithmic capability gains, so governance should also target algorithmic research. This matters because export controls and compute thresholds are currently the main regulatory levers for frontier AI.

What carries the argument

The central object is compute-equivalent gain (CEG), defined as $C_b/C_e$, the ratio of compute (FLOPs) a baseline model needs to reach a target loss to the compute an improved model needs to reach the same loss. The classification scheme around it is the load-bearing mechanism: an algorithm is compute-independent if it beats the status quo by a similar amount at baseline and increased compute, and compute-dependent if its advantage is absent or negative at baseline but grows with compute. The experiments operationalize CEG by fixing FLOPs per step for each model size and counting steps to a fixed validation-loss target, so a CEG above 1 means fewer FLOPs for the same loss.

What would settle it

Run the same fixed-loss CEG protocol on a compute-dependent algorithm such as sparse attention across a wider range of model sizes, from 100M to 1B parameters or beyond; if its CEG stays below 1 at the larger sizes, or if a compute-independent algorithm such as RoPE loses its advantage above some scale, the classification and the governance conclusion would fail.

Watch

Extended reading notes

Core claim

The discovery is a way to separate two types of algorithmic progress and evidence that the separation holds in small models. Advances are classified by comparing their performance gain over a predecessor at baseline and at increased compute: if the gain is similar at both, it is compute-independent; if it is small or negative at baseline and large at higher compute, it is compute-dependent. Using CEG as the common yardstick, the paper estimates that compute-dependent advances are historically the largest (transformer $20$--$50\times$, mixture-of-experts $7$--$11\times$, sparse attention $4.8$--$7\times$), while compute-independent advances are smaller but universal (RoPE $1.7\times$, layer normalization $1.67\times$, FlashAttention formally $1\times$ with a $2$--$4\times$ practical speedup). The experiments reproduce this pattern: at 50M parameters, LayerNorm and RoPE give CEGs of $1.836\times$ and $1.870\times$ and a combined $3.483\times$, while multi-query attention and sparse attention give $0.673\times$ and $0.515\times$; at 110M parameters, the compute-dependent methods rise to $0.931\times$ and $0.964\times$, consistent with benefits that emerge as compute grows.

Load-bearing premise

The load-bearing premise is that the compute-dependence pattern observed at 50M and 110M parameters also holds at the much larger scales of real frontier models, which the paper does not test.

Editorial extensions

If this is right

  • Hardware export controls should be expected to slow frontier LLM progress rather than stop it, because compute-independent algorithms keep improving models within any fixed compute budget.
  • The largest per-advance gains remain tied to high compute, so restricting large-scale training would still suppress the most significant capability jumps.
  • If compute-independent results transfer across scale, small models become a cheaper testbed for algorithmic search, which should make automated discovery of such advances more effective.
  • Compute thresholds used as governance tripwires become less reliable, since algorithmic gains can push a model's capability across a threshold without crossing the corresponding compute threshold.
  • Algorithmic gains can combine super-additively, as the combined LayerNorm+RoPE+FlashAttention CEG of $3.5\times$ exceeds the individual effects, so capability assessments should test bundles of advances rather than single changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the paper's framework is to apply CEG to data-quality and inference-time improvements; the authors list these as out of scope, but the same ratio would reveal whether their compute-independence classification holds beyond pretraining.
  • The paper's observation that its compute-dependent examples all alter the attention mechanism implies a prediction it does not prove: the next very large CEGs will come from attention-cost changes such as linear or recurrent attention, while non-attention components yield smaller gains.
  • If the scale-transfer assumption is correct, monitoring published small-scale efficiency gains could serve as a leading indicator for frontier progress, because compute-independent advances should propagate upward at little extra cost.
  • The combined-gain result raises the possibility that the paper's single-algorithm CEG estimates understate real progress; frontier models bundle many advances, so their joint CEG could be larger than the sum of the parts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a framework that classifies algorithmic advancements in LLM development as compute-dependent (benefits emerge primarily at high compute) or compute-independent (benefits appear across scales), and uses Compute-Equivalent Gain (CEG) to quantify these effects. It presents case studies for several well-known algorithms (Transformer, Sparse Attention, MoE, MQA as dependent; RoPE, FlashAttention, LayerNorm as independent) and reports small-scale nanoGPT experiments at two model sizes (~50M and ~110M parameters) trained for 50,000 iterations. Based on these experiments, it claims that compute-independent advances yield performance gains across scales (e.g., combined CEG up to 3.5x), while compute-dependent advances are detrimental at small scale but improve toward baseline as scale increases. The paper concludes that hardware controls can slow but not halt algorithmic progress, and recommends that AI governance incorporate algorithmic research oversight.

Significance. The compute-dependent/independent distinction is a clear and potentially policy-relevant conceptual contribution. The case studies synthesize CEG estimates from the literature, and the authors make their code available, which supports reproducibility. If the experimental results were robust, the framework would offer a practical tool for forecasting AI progress and assessing governance interventions. However, the quantitative validation is currently fragile: it relies on a small scale range, single runs without error bars, and contains an apparent internal inconsistency in the reported CEG values. The policy conclusions are plausible and align with prior literature (e.g., Gupta et al. 2024), but they are not yet strongly supported by the empirical evidence presented here.

major comments (4)
  1. [Experiments, Table 3 and CEG definition] There is an internal inconsistency between the stated CEG definition and the values in Table 3 for the full-model rows. The paper defines primary CEG as S_base / S_alg, where S_alg is the first step at which the algorithm reaches L_target, the baseline's validation loss at its 50,000th iteration. Therefore, if an algorithm's validation loss ever falls to or below L_target within those 50,000 steps, its CEG must be at least 1.0. Table 3 reports full-model MQA and Sparse Attention with minimum validation losses of 3.535 and 3.418, both below the baseline's minimum of 3.542, yet the table lists CEGs of 0.931 and 0.964. Since these minima are direct evidence that the algorithms reached L_target, the auxiliary rule (which is reserved for algorithms that fail to reach L_target) does not apply. Either the min-loss column, the CEG column, or the stated measurement definition is wrong. These two rows are the key evidence for the claim that compute-dependent advances move from detrimental at small scale to 'on par with the baseline' at larger scale, so the quantitative validation is not trustworthy as reported.
  2. [Experiments, all reported results] All CEG values come from a single training run per configuration, with no seeds or error bars. Because CEG is derived from the first step at which the loss crosses L_target, a single noisy low-loss spike can produce a spuriously high CEG, and the absence of repeated runs makes it impossible to assess whether the reported compact-model values (e.g., 0.515 for Sparse Attention) or the combined 3.5x gain are reliably different from noise. The authors should rerun each configuration with multiple seeds (or at least report variance/confidence intervals) and make the raw loss curves available so that the step-to-target computations can be verified.
  3. [Methodology, 'Our Framework' and Results] The experimental validation is at risk of circularity because the classification of an algorithm as compute-dependent is defined by where its benefits materialize. Observing low small-scale CEG for MQA and Sparse Attention is thus partly a restatement of the classification rather than an independent confirmation. The case studies in Table 2 provide some independent grounding, but the experimental test would be substantially stronger if the classifications were pre-registered or if the experiments covered a wider, pre-specified range of scales to test the predicted scaling trend. As it stands, the results are consistent with the framework but do not strongly discriminate it from alternative explanations.
  4. [Implications for Policymakers and Limitations] The central policy conclusion—that hardware restrictions are insufficient to prevent all capability gains—assumes that the compute-dependence behavior observed at 50M–110M parameters persists at frontier scales. The paper's own Limitations section concedes that the modest scale and lack of full convergence 'could influence the precise CEG values reported and the observed performance characteristics of some algorithms.' Without larger-scale evidence or a theoretical argument explaining why the observed trend should extrapolate, the empirical basis for the policy claim is thin. The authors should either soften the policy statements or provide additional evidence, such as experiments at a few more scales or a connection to established scaling-law frameworks.
minor comments (6)
  1. [Methodology, after Eq. (2)] The phrase 'fit will within our framework' appears to be a typo; it should read 'fit well within our framework.'
  2. [Figure 1] The x-axis of Figure 1 is labeled up to 40 (thousands of iterations), but the text in the Experiments section states that training ran for 50,000 iterations. Please clarify whether the curves are truncated or the axis is mislabeled.
  3. [Table 2] For MQA, the CEG column lists encoder and decoder speedups rather than a single CEG value; the table would be clearer if it noted explicitly that these are inference-time speedups and distinguished them from the FLOP-based CEG defined in Eq. (2).
  4. [References] The reference 'Andrej 2025' is incomplete; the name should be given in full (Andrej Karpathy) and the GitHub repository URL should be included, as done in the abstract.
  5. [Discussion] The hypothesis that 'the attention mechanism is the primary compute consumer' is speculative given the small set of algorithms tested; it should be framed as a conjecture to be tested in future work.
  6. [Abstract and Conclusion] The phrase 'combined CEG up to 3.5x' is derived from a single run without error bars; the wording should be qualified to reflect the limited evidence, e.g., 'in our single-run experiments, the combined algorithms achieved CEG ≈ 3.5x.'

Circularity Check

2 steps flagged · score 4.0 of 10

Compute-dependent 'validation' restates its own definition; Table 3 contradicts the stated CEG rule.

  1. self definitional [Methodology: 'Our Framework: Compute-Dependence and Compute-Equivalent Gain'; Experimental Results: 'Compute-Dependent Advancements']
    "Alternatively, if an algorithm provides small benefits (or worsens performance) at baseline compute, but gives large benefits with increased compute, we classify it as a compute-dependent advance. ... This behavior—poor or neutral performance at smaller scales improving towards neutrality or potential benefit with increased compute—is characteristic of compute-dependent advancements as defined in our framework."

    The definition of a compute-dependent advance is precisely the pattern that the experiments are said to confirm: small or negative benefit at baseline compute and larger benefit when compute increases. Reporting that MQA and sparse attention move from CEG < 1 at 50M parameters to CEG approximately 1 at 110M parameters is therefore a restatement of the classification rule rather than an independent test. The classifications were made a priori from prior literature, so this is partial rather than total circularity; nevertheless, the paper presents the match as experimental validation of the framework.

  2. other [Experiments: 'Primary CEG' definition and Table 3]
    "The Primary CEG was then calculated as S_base / S_alg. Consequently, a CEG>1.0 indicates the algorithm reached L_target in fewer steps (i.e., with less compute) than the baseline, signifying improved efficiency. ... Table 3: Multi-Query Attention 3.846 0.673 3.535 0.931; Sparse Attention 3.886 0.515 3.418 0.964; None 3.799 1.000 3.542 1.000."

    With S_base equal to 50,000 steps, any algorithm reaching L_target at or before 50,000 steps has CEG = S_base / S_alg >= 1. The table lists full-model MQA min validation loss 3.535 and sparse attention min validation loss 3.418, both below the baseline's 3.542, so both algorithms necessarily reached L_target within the run, forcing CEG >= 1 under the primary rule. The auxiliary rule applies only when the algorithm fails to reach L_target, and a minimum below the baseline's minimum is direct evidence that it did reach it. Therefore the reported CEGs of 0.931 and 0.964 cannot be produced by either stated formula, meaning the numbers used to validate the compute-dependent trend are not derived from the paper's own metric.

full rationale

The paper's compute-dependent versus compute-independent distinction is not itself circular: the algorithms are classified from prior literature, and the Table 2 CEG estimates are computed from external benchmarks and FLOPs rather than from the paper's own experimental fits. The main circularity lies in the experimental validation of the compute-dependent category, where the observed pattern is identical to the definitional pattern and therefore provides confirmation only in a tautological sense. There is no load-bearing self-citation chain or imported-uniqueness argument. Separately, although this is not circularity per se, Table 3's full-model CEG values for MQA and sparse attention contradict the paper's stated primary CEG definition, so the specific empirical numbers used to illustrate the compute-dependent trend are not trustworthy. The compute-independent results, such as LayerNorm and RoPE showing CEG > 1 at small scale, retain independent content, and the authors' limitations section candidly acknowledges that modest scale and lack of convergence could affect the reported values. Overall, the central framework has independent grounding, but one load-bearing validation step reduces to its own definition, yielding a partial circularity score of 4.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central quantitative claims rest on the chosen L_target reference, the loss-as-capability proxy, the fixed-iteration protocol, and the assumption that 50-110M parameter behavior extrapolates to frontier scales. The CEG metric itself is reused from Davidson et al. (2023).

free parameters (1)
  • Baseline target validation loss L_target = 3.799 (compact), 3.542 (full)
    Chosen as the baseline model's loss after 50,000 steps; all CEG values depend on this reference point, and using another target would change them.
assumptions (4)
  • domain assumption Compute cost is proportional to active parameters per step multiplied by training steps (Eq. 1).
    Underlies every CEG estimate; ignores data differences, memory bandwidth, and wall-clock effects; authors acknowledge this in the framework section.
  • domain assumption Validation cross-entropy loss is an adequate performance measure for capability gain.
    CEG is defined through validation loss; downstream task capability is not measured.
  • domain assumption Results from 50M and 110M parameter models transfer to frontier-scale LLMs.
    The policy implications depend on this extrapolation; the Limitations section concedes the scale gap.
  • domain assumption Comparing all models at 50,000 iterations, without training to convergence, yields comparable CEG estimates.
    The experimental protocol fixes iteration count rather than convergence; the authors note this could affect CEG values.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking LLM Advancement: Compute-Dependent and Independent Paths to Progress." pith.science (2026). https://pith.science/paper/OGMYVV2H

@misc{pith2026250504075,
  author       = {Pith},
  title        = {Pith review of: Rethinking LLM Advancement: Compute-Dependent and Independent Paths to Progress},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OGMYVV2H}},
  note         = {Machine review of arXiv:2505.04075}
}
abstract

Regulatory efforts to govern large language model (LLM) development have predominantly focused on restricting access to high-performance computational resources. This study evaluates the efficacy of such measures by examining whether LLM capabilities can advance through algorithmic innovation in compute-constrained environments. We propose a novel framework distinguishing compute-dependent innovations--which yield disproportionate benefits at high compute--from compute-independent innovations, which improve efficiency across compute scales. The impact is quantified using Compute-Equivalent Gain (CEG). Experimental validation with nanoGPT models confirms that compute-independent advancements yield significant performance gains (e.g., with combined CEG up to $3.5\times$) across the tested scales. In contrast, compute-dependent advancements were detrimental to performance at smaller experimental scales, but showed improved CEG (on par with the baseline) as model size increased, a trend consistent with their definition of yielding primary benefits at higher compute. Crucially, these findings indicate that restrictions on computational hardware, while potentially slowing LLM progress, are insufficient to prevent all capability gains driven by algorithmic advancements. We argue that effective AI oversight must therefore incorporate mechanisms for understanding, anticipating, and potentially guiding algorithmic research, moving beyond a singular focus on hardware. The proposed framework also serves as an analytical tool for forecasting AI progress.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 17 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    AI, M. 2024. Introducing Llama 3.1: Our Most Capable Models to Date. https://ai.meta.com/blog/meta-llama-3-1/

    work page 2024

  4. [4]

    Amodei, D. 2025. On DeepSeek and Export Controls . Accessed: March 13, 2025

    work page 2025

  5. [5]

    Andrej. 2025. karpathy/ nanoGPT

    work page 2025

  6. [6]

    L.; Kiros, J

    Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016. Layer Normalization. arXiv:1607.06450

    arXiv 2016

  7. [7]

    Child, R.; Gray, S.; Radford, A.; and Sutskever, I. 2019. Generating Long Sequences with Sparse Transformers . arXiv:1904.10509

    arXiv 2019

  8. [8]

    Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H. W.; Sutton, C.; Gehrmann, S.; Schuh, P.; Shi, K.; Tsvyashchenko, S.; Maynez, J.; Rao, A.; Barnes, P.; Tay, Y.; Shazeer, N.; Prabhakaran, V.; Reif, E.; Du, N.; Hutchinson, B.; Pope, R.; Bradbury, J.; Austin, J.; Isard, M.; Gur-Ari , G.; Yin, P.; Duke, T.; Levsk...

    arXiv 2022

Show all 52 references
  1. [9]

    X.; Gao, H.; Chen, D.; Li, J.; Zeng, W.; Yu, X.; Wu, Y.; Xie, Z.; Li, Y

    Dai, D.; Deng, C.; Zhao, C.; Xu, R. X.; Gao, H.; Chen, D.; Li, J.; Zeng, W.; Yu, X.; Wu, Y.; Xie, Z.; Li, Y. K.; Huang, P.; Luo, F.; Ruan, C.; Sui, Z.; and Liang, W. 2024. DeepSeekMoE : Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models . arXiv:2401.06066

    2024 arXiv

  2. [10]

    Dao, T. 2023. FlashAttention-2 : Faster Attention with Better Parallelism and Work Partitioning . arXiv:2307.08691

    2023 arXiv

  3. [11]

    Y.; Ermon, S.; Rudra, A.; and R \'e , C

    Dao, T.; Fu, D. Y.; Ermon, S.; Rudra, A.; and R \'e , C. 2022. FlashAttention : Fast and Memory-Efficient Exact Attention with IO-Awareness . arXiv:2205.14135

    2022 arXiv

  4. [12]

    Davidson, T.; Denain, J.-S.; Villalobos, P.; and Bas, G. 2023. AI Capabilities Can Be Significantly Improved without Expensive Retraining. arXiv:2312.07413

    2023 arXiv

  5. [13]

    DeepSeek-AI ; Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; Zhang, X.; Yu, X.; Wu, Y.; Wu, Z. F.; Gou, Z.; Shao, Z.; Li, Z.; Gao, Z.; Liu, A.; Xue, B.; Wang, B.; Wu, B.; Feng, B.; Lu, C.; Zhao, C.; Deng, C.; Zhang, C.; Ruan, C.; ...

    2025 arXiv

  6. [14]

    DeepSeek-AI ; Liu, A.; Feng, B.; Xue, B.; Wang, B.; Wu, B.; Lu, C.; Zhao, C.; Deng, C.; Zhang, C.; Ruan, C.; Dai, D.; Guo, D.; Yang, D.; Chen, D.; Ji, D.; Li, E.; Lin, F.; Dai, F.; Luo, F.; Hao, G.; Chen, G.; Li, G.; Zhang, H.; Bao, H.; Xu, H.; Wang, H.; Zhang, H.; Ding, H.; X...

    2024 arXiv

  7. [15]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding . arXiv:1810.04805

    2019 arXiv

  8. [16]

    L.; Zhang, C.; Xu, Y.; Shang, N.; Xu, J.; Yang, F.; and Yang, M

    Ding, Y.; Zhang, L. L.; Zhang, C.; Xu, Y.; Shang, N.; Xu, J.; Yang, F.; and Yang, M. 2024. LongRoPE : Extending LLM Context Window Beyond 2 Million Tokens . arXiv:2402.13753

    2024 arXiv

  9. [17]

    Droppo, J.; and Elibol, O. 2021. Scaling Laws for Acoustic Models . arXiv:2106.09488

    2021 arXiv

  10. [18]

    Erdil, E.; and Besiroglu, T. 2023. Algorithmic progress in computer vision. arXiv:2212.05153

    2023 arXiv

  11. [19]

    Fedus, W.; Zoph, B.; and Shazeer, N. 2022. Switch Transformers : Scaling to Trillion Parameter Models with Simple and Efficient Sparsity . arXiv:2101.03961

    2022 arXiv

  12. [20]

    Fist, T.; and Grunewald, E. 2023. Preventing AI Chip Smuggling to China: A Working Paper. Working paper, Center for a New American Security. Center for a New American Security Technology and National Security Program

    2023

  13. [21]

    Gao, L.; Biderman, S.; Black, S.; Golding, L.; Hoppe, T.; Foster, C.; Phang, J.; He, H.; Thite, A.; Nabeshima, N.; Presser, S.; and Leahy, C. 2020. The P ile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv preprint arXiv:2101.00027

    2020 arXiv

  14. [22]

    Gupta, R.; Walker, L.; and Reddie, A. W. 2024. Whack-a- Chip : The Futility of Hardware-Centric Export Controls . arXiv:2411.14425

    2024 arXiv

  15. [23]

    Hernandez, D.; and Brown, T. 2020. Measuring the Algorithmic Efficiency of Neural Networks

    2020

  16. [24]

    C.; Atkinson, D.; Thompson, N.; and Sevilla, J

    Ho, A.; Besiroglu, T.; Erdil, E.; Owen, D.; Rahman, R.; Guo, Z. C.; Atkinson, D.; Thompson, N.; and Sevilla, J. 2024. Algorithmic Progress in Language Models. arXiv:2403.05812

    2024 arXiv

  17. [25]

    A.; Jordan, M

    Jacobs, R. A.; Jordan, M. I.; Nowlan, S. J.; and Hinton, G. E. 1991. Adaptive Mixtures of Local Experts . Neural Computation, 3(1): 79--87

    1991

  18. [26]

    Jia, Z.; Maggioni, M.; Staiger, B.; and Scarpazza, D. P. 2018. Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking . arXiv:1804.06826

    2018 arXiv

  19. [27]

    B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; and Amodei, D

    Kaplan, J.; McCandlish, S.; Henighan, T.; Brown, T. B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; and Amodei, D. 2020. Scaling Laws for Neural Language Models. arXiv:2001.08361

    2020 arXiv

  20. [28]

    Karnofsky, H. 2024. A Sketch of Potential Tripwire Capabilities for AI. Carnegie California

    2024

  21. [29]

    Lepikhin, D.; Lee, H.; Xu, Y.; Chen, D.; Firat, O.; Huang, Y.; Krikun, M.; Shazeer, N.; and Chen, Z. 2020. GShard : Scaling Giant Models with Conditional Computation and Automatic Sharding . arXiv:2006.16668

    2020 arXiv

  22. [30]

    B.; Zi, Y.; Muennighoff, N.; Kocetkov, D.; Mou, C.; Marone, M.; Akiki, C.; Li, J.; Chim, J.; Liu, Q.; Zheltonozhskii, E.; Zhuo, T

    Li, R.; Allal, L. B.; Zi, Y.; Muennighoff, N.; Kocetkov, D.; Mou, C.; Marone, M.; Akiki, C.; Li, J.; Chim, J.; Liu, Q.; Zheltonozhskii, E.; Zhuo, T. Y.; Wang, T.; Dehaene, O.; Davaadorj, M.; Lamy-Poirier , J.; Monteiro, J.; Shliazhko, O.; Gontier, N.; Meade, N.; Zebaze, A.; Ye...

    2023 arXiv

  23. [31]

    Lohn, A. 2023. Scaling AI : Cost and Performance of AI at the Leading Edge . Technical report, Center for Security and Emerging Technology

    2023

  24. [32]

    R.; Cojocaru, R.; Farooq, M.; Campesan, G.; Djilali, Y

    Malartic, Q.; Chowdhury, N. R.; Cojocaru, R.; Farooq, M.; Campesan, G.; Djilali, Y. A. D.; Narayan, S.; Singh, A.; Velikanov, M.; Boussaha, B. E. A.; Al-Yafeai , M.; Alobeidli, H.; Qadi, L. A.; Seddik, M. E. A.; Fedyanin, K.; Alami, R.; and Hacid, H. 2024. Falcon2- 11B Technic...

    2024 arXiv

  25. [33]

    Nawrot, P.; Li, R.; Huang, R.; Ruder, S.; Marchisio, K.; and Ponti, E. M. 2025. The Sparse Frontier : Sparse Attention Trade-offs in Transformer LLMs . arXiv:2504.17768

    2025 arXiv

  26. [34]

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. Training language model...

    2022

  27. [35]

    Patel, D. 2025. Jeff Dean & Noam Shazeer – 25 years at Google: from PageRank to AGI. Podcast. Two of Gemini's co-leads on Google's path to AGI

    2025

  28. [36]

    A.; and Lewis, M

    Press, O.; Smith, N. A.; and Lewis, M. 2022. Train Short , Test Long : Attention with Linear Biases Enables Input Length Extrapolation . arXiv:2108.12409

    2022 arXiv

  29. [37]

    Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019. Language Models are Unsupervised Multitask Learners . OpenAI Blog

    2019

  30. [38]

    Schmid, P.; Sanseviero, O.; Bartolome, A.; von Werra , L.; Vila, D.; Srivastav, V.; Sun, M.; and Cuenca, P. 2025. Llama 3.1 - 405B , 70B & 8B with Multilinguality and Long Context. https://huggingface.co/blog/llama31

    2025

  31. [39]

    Schneider, J. 2024. Deepseek: The Quiet Giant Leading China 's AI Race . https://www.chinatalk.media/p/deepseek-ceo-interview-with-chinas

    2024

  32. [40]

    Sevilla, J.; Besiroglu, T.; Cottier, B.; You, J.; Roldán, E.; Villalobos, P.; and Erdil, E. 2024. Can AI Scaling Continue Through 2030? Accessed: 2025-02-14

    2024

  33. [41]

    Sevilla, J.; Heim, L.; Ho, A.; Besiroglu, T.; Hobbhahn, M.; and Villalobos, P. 2022. Compute Trends Across Three Eras of Machine Learning. In 2022 International Joint Conference on Neural Networks (IJCNN), 1--8

    2022

  34. [42]

    Sevilla, J.; and Roldán, E. 2024. Training Compute of Frontier AI Models Grows by 4-5x per Year. Accessed: 2025-03-03

    2024

  35. [43]

    Shah, J.; Bikshandi, G.; Zhang, Y.; Thakkar, V.; Ramani, P.; and Dao, T. 2024. FlashAttention-3 : Fast and Accurate Attention with Asynchrony and Low-precision . arXiv:2407.08608

    2024 arXiv

  36. [44]

    Shazeer, N. 2019. Fast Transformer Decoding : One Write-Head Is All You Need . arXiv:1911.02150

    2019 arXiv

  37. [45]

    Shazeer, N.; Mirhoseini, A.; Maziarz, K.; Davis, A.; Le, Q.; Hinton, G.; and Dean, J. 2017. Outrageously Large Neural Networks : The Sparsely-Gated Mixture-of-Experts Layer . arXiv:1701.06538

    2017 arXiv

  38. [46]

    Su, J.; Lu, Y.; Pan, S.; Murtadha, A.; Wen, B.; and Liu, Y. 2023. RoFormer : Enhanced Transformer with Rotary Position Embedding . arXiv:2104.09864

    2023 arXiv

  39. [47]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention Is All You Need. In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., Advances in Neural Informati...

    2017

  40. [48]

    Wang, L.; Gao, H.; Zhao, C.; Sun, X.; and Dai, D. 2024. Auxiliary- Loss-Free Load Balancing Strategy for Mixture-of-Experts . arXiv:2408.15664

    2024 arXiv

  41. [49]

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E.; Le, Q.; and Zhou, D. 2023. Chain-of- Thought Prompting Elicits Reasoning in Large Language Models . arXiv:2201.11903

    2023 arXiv

  42. [50]

    X.; Wang, L.; Xiao, Z.; Wang, Y.; Ruan, C.; Zhang, M.; Liang, W.; and Zeng, W

    Yuan, J.; Gao, H.; Dai, D.; Luo, J.; Zhao, L.; Zhang, Z.; Xie, Z.; Wei, Y. X.; Wang, L.; Xiao, Z.; Wang, Y.; Ruan, C.; Zhang, M.; Liang, W.; and Zeng, W. 2025. Native Sparse Attention : Hardware-Aligned and Natively Trainable Sparse Attention . arXiv:2502.11089

    2025 arXiv

  43. [51]

    Zhang, B.; and Sennrich, R. 2019. Root Mean Square Layer Normalization . arXiv:1910.07467

    2019 arXiv

  44. [52]

    Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; Du, Y.; Yang, C.; Chen, Y.; Chen, Z.; Jiang, J.; Ren, R.; Li, Y.; Tang, X.; Liu, Z.; Liu, P.; Nie, J.-Y.; and Wen, J.-R. 2024. A Survey of Large Language Models . arXiv:2303.18223

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.