Pith. sign in

REVIEW 3 major objections 5 minor 91 references

From 2006 to 2025, dense FP16 compute in NVIDIA datacenter GPUs doubled every 1.44 years and FP32 every 1.69 years, driven by Tensor Cores and architectural specialization.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 07:30 UTC pith:EP4GVBYQ

load-bearing objection Useful dataset and honest regressions, but the headline FP32 doubling time is really TF32 tensor throughput, and that should be fixed before publication. the 3 major comments →

arxiv 2601.20115 v3 pith:EP4GVBYQ submitted 2026-01-27 cs.AR cs.AI

How Much Progress Has There Been in NVIDIA Datacenter GPUs?

Emanuele Del Sozzo , Martin Fleming , Kenneth Flamm , Neil Thompson This is my paper
classification cs.AR cs.AI
keywords NVIDIA GPUsprogress trendsgrowth ratesdoubling timesTensor Coresmemory bandwidthHBMexport control
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Across 101 NVIDIA datacenter GPUs released from 2006 through 2025, the paper fits exponential growth curves and finds that low-precision AI compute has been improving far faster than any other measured hardware property. Dense FP16 performance doubles every 1.44 years (61.9% annual growth) and dense FP32 every 1.69 years (50.9%), while off-chip memory size and bandwidth double every 3.32-3.53 years, release price every 5.09 years, and power draw every ~16 years. The mechanism the authors identify is architectural specialization—FP16 units, Tensor Cores, and optional sparsity—rather than general chip scaling, and they argue this is why per-GPU progress has kept pace with AI's exploding compute demand. If these rates are right, memory bandwidth and export policy, not raw FLOPs, are the emerging constraints on AI compute. The paper's export-control analysis puts the regulated performance gap at 23.6x under the 2025 rules, falling to 3.54x after the December 2025 decision to permit H200 exports.

Core claim

The paper's central claim is that the pace of NVIDIA datacenter GPU progress is set by how NVIDIA chooses to spend silicon on precision-specific tensor math. Pooling CUDA Cores and Tensor Cores into a single 'peak performance at a given precision' series—taking the Tensor Core value whenever present—the authors estimate CAGRs of 61.9% for FP16 and 50.9% for FP32 without sparsity, and 72.0%/60.3% with sparsity. They read the FP64 result (20.1% to 39.9% depending on FP64 core count) as deliberate deprioritization of high precision. Memory grows slower: HBM-only bandwidth at 24.8%, GDDR-only at 12.7%; price at 14.6% and TDP at 4.4% across the full lineup. The paper also converts its performance

What carries the argument

The load-bearing object is the pooled peak-TFLOPS series: for every GPU, the paper takes the maximum theoretical throughput at FP16/FP32/FP64 and, once Tensor Cores exist, uses the Tensor Core number for that precision instead of the CUDA Core number, because the two engines share die area and their FLOPs do not add. This pooling rule converts the introduction of Volta's FP16 Tensor Cores (2017) and Ampere's FP32/TF32 Tensor Cores (2020) into visible step changes and is the stated reason the paper's FP16/FP32 growth rates exceed those of studies that keep CUDA and Tensor series separate by 26-36 percentage points. Support machinery: log-linear exponential regressions with delta-method standa

Load-bearing premise

The headline doubling times rest on treating NVIDIA's specialized matrix-multiply units (Tensor Cores) as the GPU's FP16/FP32 engine as soon as they exist; if the older general-purpose CUDA Cores are tracked separately, the paper's own comparison shows growth falling by 26-36 percentage points.

What would settle it

Re-fit the paper's regressions with one change: whenever both exist, use the CUDA Core TFLOPS instead of the Tensor Core TFLOPS for FP16/FP32. The paper acknowledges this lowers CAGR by 26-36 percentage points, so if that series is the correct one the 1.44/1.69 year doubling times collapse. The data to do this are the same 101 GPUs' spec-sheet values.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Top-of-line NVIDIA GPUs roughly quadruple FP16 throughput every three years, which is fast enough to absorb much of AI's reported compute-demand growth without relying on cluster expansion alone.
  • Because compute grows faster than off-chip bandwidth (FP16-compute-per-bandwidth CAGR 33.0%), memory-bound workloads will hit a bandwidth wall before peak FLOPs become the constraint.
  • FP64 is slipping below the general trend: high-FP64 chips grow at 39.9% but the low-FP64 class at 20.1%, and the latest flagship has nearly none, so double-precision scientific computing will not enjoy the same curve.
  • HBM is the technology doing the memory work: HBM-only bandwidth grows at 24.8% per year versus 12.7% for GDDR-only, so progress in memory depends on continued HBM supply and density.
  • The export-control arithmetic says policy choices, not engineering, currently define the frontier that restricted countries can reach: the TPP gap is 23.6x under strict rules and 3.54x after permitting H200 sales.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The doubling times are a statement about NVIDIA's product mix: once Tensor Cores exist, they become the measured engine, so the headline rates describe specialization strategy, not general-purpose silicon progress. A testable corollary is that workstation and consumer NVIDIA GPUs with weak Tensor Core provisions should show markedly flatter FP16/FP32 curves over the same years.
  • The export-control gap is computed in TPP, which omits memory bandwidth and capacity even though the paper finds memory growing slower than compute. A regulatory or analytical metric that includes HBM bandwidth would likely produce a larger effective gap for real AI workloads.
  • The paper itself doubts the trends will persist, citing diminishing returns from bitwidth reduction and multi-die packaging; a forward-looking test is to track whether post-2025 releases keep FP16 doubling near 1.4 years or stretch toward 2+ years.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper compiles a dataset of 101 NVIDIA datacenter GPUs (2007–2025) and fits exponential trends to peak FP16/FP32/FP64 performance, off-chip memory size/bandwidth, release price, and TDP, plus per-bandwidth, per-dollar, and per-watt ratios. The headline results are doubling times of about 1.44 y for FP16 and 1.69 y for FP32 (without sparsity), slower growth for FP64 (2.06–3.79 y), and even slower growth for memory, price, and TDP. The paper also compares its rates with Huang's law and Epoch AI, and quantifies the impact of US export controls as a performance gap that drops from 23.6× to 3.54× under the recent H200 allowance.

Significance. If the results hold, the paper provides a careful, apples-to-apples characterization of NVIDIA datacenter GPU progress using a transparently documented dataset and regression methodology with delta-method confidence intervals. Its explicit comparison with Epoch AI and Huang's law, and its direct arithmetic application of export-control thresholds, are useful contributions. The main caveat is that the headline FP32 rate is built on a nonstandard definition that mixes TF32 with FP32, and the all-datacenter TDP trend has very low explanatory power. The paper also makes a defensible but contestable modeling choice to pool CUDA Core and Tensor Core performance, which drives a large part of the difference from prior work.

major comments (3)
  1. [§III-B1, §IV-B1, Insight 3, Abstract] The FP32 series from 2020 onward uses Tensor Core TF32 peak values, not IEEE FP32, but the abstract, Insight 3, and Figure 4 present this as 'FP32 dense operations' doubling every 1.69 years. Footnote 6 acknowledges 'we make no distinction between FP32 and TF32 data types,' and §IV-A2 says 'FP32 (or, rather, TF32),' but the headline claims are unqualified. Since TF32 has a 10-bit explicit mantissa versus FP32's 23 bits, this is not a like-for-like FP32 metric. The difference is material: Table IV shows Epoch AI's separate CUDA FP32 CAGR is 28%, 23 ppts below the paper's 51%. The paper should either relabel the series as 'peak 32-bit tensor (TF32)' throughout, including the abstract and Insight 3, or present a separate pure-CUDA FP32 trend as a robustness check. As it stands, the central claim overstates the growth rate of IEEE FP32 dense computation.
  2. [§IV-B1, Table III, Abstract] For all datacenter GPUs, the TDP exponential fit has R² = 0.15 (Table III), yet the paper presents a '4.4% CAGR' and '16.16y DT' as a headline result, and the abstract says power 'roughly doubled every 15 years.' With R² = 0.15, the exponential model explains almost none of the variation, so this DT is not a meaningful trend. The top-performer TDP fit (R² = 0.73) supports a slow-growth claim, but the all-datacenter claim should be either removed, restricted to top performers, or reported with an explicit caveat that the data show no stable exponential trend.
  3. [§IV-A, §IV-B, §VI-A] The pooling of CUDA Core and Tensor Core performance into a single peak-performance series is a load-bearing modeling choice. The paper justifies it by saying the units 'compete for the same space on the chip,' which is a reasonable view of architectural trade-offs, but it is not the only reasonable view, and it is the main reason the FP16/FP32 growth rates are 26–36 ppts higher than Epoch AI's. The manuscript should state more prominently in the abstract and conclusions that these are 'peak available performance, taking the faster specialized unit when present' rather than a homogeneous precision-specific compute rate. The current wording in figures and text, e.g., 'FP16 without Sparsity - CAGR: 61.9%,' implies a single coherent FP16 unit that does not exist across the full time series.
minor comments (5)
  1. [Abstract/Introduction] The abstract reports FP16 and FP32 doubling times of 1.43 and 1.67 years, while the introduction and conclusion report 1.44 and 1.69 years. Please reconcile the numbers and the rounding convention.
  2. [§I, Introduction] There are several typos: 'the the creation' in the first sentence, 'thede facto' (missing space), and 'thethe' constructions. A careful proofread is needed.
  3. [§III-B2] The selection of 'top-performing GPUs per year' is based on 'the average of the peak performance in FP16, FP32, and FP64.' Please specify whether this is an arithmetic or geometric mean, since the choice affects the selected set given the very different magnitudes of FP16 vs FP64 throughput.
  4. [Table II] The legend for Table II uses the same checkmark symbol for 'collected for all GPUs' and 'collected for some GPUs,' which is confusing. Please use distinct symbols, e.g., ✓ and ◐.
  5. [Figures 2 and 4] The figures use 'FP32' labels on points and trend lines that, after 2020, are actually Tensor Core TF32 values. Even if the text clarifies this, the figures should carry a note such as 'FP32 includes TF32 tensor throughput' to prevent misreading.

Circularity Check

0 steps flagged

No significant circularity: the growth rates are empirical OLS fits to published GPU specifications, not derivations from the claimed doubling times; the FP32/TF32 pooling is a disclosed measurement convention, not a circular reduction.

full rationale

The paper's central results are log-linear OLS fits (Eq. 1) to publicly documented peak TFLOPS, memory, price, and TDP values for 101 NVIDIA datacenter GPUs. The reported CAGRs and doubling times are transformations of the estimated time coefficient, so they are outputs of the data, not inputs. The export-control section applies regulatory TPP thresholds (Eq. 2) to the same specification data; the 23.6x and 3.54x gaps are arithmetic ratios of peak TPP under different policy regimes, not predictions derived from the fitted curves. The one modeling choice that a reader could question — pooling CUDA Core and Tensor Core throughput into a single 'FP32' series and including TF32 tensor values from Ampere onward — is explicitly disclosed: 'we make no distinction between FP32 and TF32 data types and refer to both as FP32 for simplicity' (§III-B1) and 'we replace CUDA Cores with Tensor Cores in the performance metrics as soon as the latter units become available' (§IV-B). This choice defines the measured series, but the fitted growth rate remains an empirical output of that series; it affects labeling and validity, but it is not a circular reduction in which the conclusion is assumed in the construction of the input. The cited prior work by one of the authors (Flamm, 'Measuring Moore's Law' [19]) supplies a similar methodology and a historical comparison; it is not used as the sole justification for any estimate, and the regression and delta-method machinery is standard and independently checkable. No self-citation chain forces the conclusions. The paper's own limitations (peak theoretical values, price imputation, boost-clock assumptions) concern measurement accuracy and scope, not circular structure. Therefore there is no significant circularity.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or conceptual entities. Its central claims rest on the exponential-growth functional form, the use of theoretical peak specifications, the FP16≈FP32 pre-2016 approximation, the CUDA/Tensor pooling choice, and the regulatory TPP definitions. The only hand-chosen numeric parameter is the FP64 CUDA-core-per-SM threshold of 8.

free parameters (1)
  • FP64 CUDA Cores per SM threshold = 8
    Manual binning threshold splitting FP64 trends into two groups; changing this threshold changes the estimated FP64 CAGRs (20.1% vs 39.9%).
axioms (5)
  • domain assumption GPU progress follows a constant exponential growth model Y = α e^(βt) over 2007–2025
    Used to estimate CAGR/DT (Eq. 1, §III-C); a misspecified functional form would bias all trend estimates.
  • domain assumption Theoretical peak TFLOPS from vendor documentation (boost clock) is a valid comparable measure of GPU progress
    Real workloads and sustained clock behavior are not used; the paper acknowledges boost clocks are not sustainable (§VI-B).
  • ad hoc to paper Pre-2016 FP16 performance equals FP32 peak performance (software emulation)
    Needed to extend the FP16 series before Pascal hardware support; actual emulated FP16 throughput is unmeasured (§III-B1).
  • ad hoc to paper CUDA Core and Tensor Core performance can be pooled, taking Tensor Core peak when available, as competing resources on the same chip
    This pooling drives the fast FP16/FP32 growth rates; Epoch AI separates the two and finds substantially lower rates (§IV-B, §VI-A).
  • domain assumption TPP formula and regulatory thresholds from BIS rules are applied as given
    The export-control gap calculations depend on interpreting ECCN 3A090 and the TPP=2·MAC TOPS·b definition (§V-A, Eq. 2).

pith-pipeline@v1.3.0-alltime-deepseek · 35147 in / 12001 out tokens · 127308 ms · 2026-08-03T07:30:36.008748+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of How Much Progress Has There Been in NVIDIA Datacenter GPUs?." pith.science (2026). https://pith.science/paper/EP4GVBYQ

@misc{pith2026260120115,
  author       = {Pith},
  title        = {Pith review of: How Much Progress Has There Been in NVIDIA Datacenter GPUs?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EP4GVBYQ}},
  note         = {Machine review of arXiv:2601.20115}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

As the role of modern Graphics Processing Units (GPUs) becomes increasingly essential for several computing tasks, analyzing their past and current progress is paramount for determining future constraints on scientific research. This is particularly compelling in the Artificial Intelligence (AI) domain, where rapid technological advancements and fierce global competition have led the United States to recently implement export control regulations limiting international access to advanced AI chips. Consequently, this paper examines technical progress in NVIDIA datacenter GPUs from the mid-2000s through 2025. Our main results identify doubling times of 1.43 and 1.67 years for FP16 and FP32 dense operations, while FP64 doubling times range from 2.05 to 3.79 years. Off-chip memory size and bandwidth have grown at slower rates than computing performance, doubling every 3.29 to 3.41 years, whereas the release prices and power consumption roughly doubled every 5.03 and 15 years, respectively. Moreover, our cross-vendor comparison of the top-performing GPUs per year shows that NVIDIA's performance advantage is narrowing, but not enough to compel a major market shift. Finally, we quantify the potential implications of current U.S. export control regulations and the consequent performance gaps, which the recently proposed policy changes could shrink from 23.6X to 3.54X.

Figures

Figures reproduced from arXiv: 2601.20115 by Emanuele Del Sozzo, Kenneth Flamm, Martin Fleming, Neil Thompson.

Figure 1
Figure 1. Figure 1: Compound Annual Growth Rates (CAGRs) with 90 percent confidence intervals and Doubling Times (DTs) for [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Top-performing GPUs per year: Computing performance, off-chip memory size and bandwidth, release price, and Thermal Design Power (TDP) scaling normalized to Tesla C870. For 64-bit Floating Point (FP64), the values are normalized to Fermi C2090 (2011). The Figure also reports the estimated Compound Annual Growth Rates (CAGRs) based on exponential trends derived from the selected GPUs (solid lines) and indic… view at source ↗
Figure 3
Figure 3. Figure 3: Top-performing GPUs per year: Exponential scaling trends for the technical improvements per memory bandwidth (left column), per dollar (central column), and per watt (right column). CAGR and 2.66y DT) and bandwidth (29.3% CAGR and 2.70y DT). Price at launch, the cost of purchasing GPUs has been consistently increasing over time, at a 22.9% CAGR from 2007 to 2025 (3.36y DT). Finally, the TDP of the selected… view at source ↗
Figure 4
Figure 4. Figure 4: All datacenter GPUs: Computing performance, off-chip memory size and bandwidth, release price, and Thermal Design Power (TDP) scaling normalized to Tesla C870. For 64-bit Floating Point (FP64), the values are normalized to Tesla M1060 (2008) or Fermi C2050 (2011), depending on the number of FP64 CUDA Cores per Streaming Multiprocessor (SM). The Figure also reports the estimated Compound Annual Growth Rates… view at source ↗
Figure 5
Figure 5. Figure 5: All datacenter GPUs: Exponential scaling trends for the technical improvements per memory bandwidth (left column), per dollar (central column), and per watt (right column). We apply horizontal jitter to prevent overplotting. the performance boost that Tensor Cores and sparsity support provide. The Tensor Core contribution is particularly evident in the FP32 per-memory bandwidth trends (CAGRs of 23.9% witho… view at source ↗
Figure 6
Figure 6. Figure 6: Impact of the United States export control regulations from 2022 and 2025 on the GPUs in our dataset. We first show [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

91 extracted references · 1 canonical work pages

  1. [1]

    TOP500 List of June 2025,

    TOP500, “TOP500 List of June 2025,” https://top500.org/lists/top500/2025/06/, 2024, accessed: 2025-11- 13

    2025

  2. [2]

    The rise of GPU computing in science,

    EMBL, “The rise of GPU computing in science,” https://www.embl.org /news/science/the-rise-gpu-computing-science/, 2018

    2018

  3. [3]

    The Evolution Of The GPU: How It Became The Heart Of AI And ML,

    AceCloud, “The Evolution Of The GPU: How It Became The Heart Of AI And ML,” https://acecloud.ai/resources/blog/the-evolution-of-gpu/, 2023

    2023

  4. [4]

    Implementation of Additional Export Controls: Certain Advanced Computing and Semiconductor Manufac- turing Items; Supercomputer and Semiconductor End Use; Entity List Modification,

    Bureau of Industry and Security, “Implementation of Additional Export Controls: Certain Advanced Computing and Semiconductor Manufac- turing Items; Supercomputer and Semiconductor End Use; Entity List Modification,” https://www.govinfo.gov/content/pkg/FR-2022-10-13/pd f/2022-21658.pdf, 2022, accessed: 2025-02-03

    2022

  5. [5]

    Implementation of Additional Export Controls: Certain Advanced Computing Items; Supercomputer and Semiconductor End Use; Updates and Corrections,

    ——, “Implementation of Additional Export Controls: Certain Advanced Computing Items; Supercomputer and Semiconductor End Use; Updates and Corrections,” https://www.govinfo.gov/content/pkg/FR-2023-10-2 5/pdf/2023-23055.pdf, 2023, accessed: 2025-02-03

    2023

  6. [6]

    Foreign-Produced Direct Product Rule Additions, and Refine- ments to Controls for Advanced Computing and Semiconductor Manu- facturing Items,

    ——, “Foreign-Produced Direct Product Rule Additions, and Refine- ments to Controls for Advanced Computing and Semiconductor Manu- facturing Items,” https://public-inspection.federalregister.gov/2024-282 70.pdf, 2024, accessed: 2025-02-03

    2024

  7. [7]

    Additions and Modifications to the Entity List; Removals from the Validated End-User (VEU) Program,

    ——, “Additions and Modifications to the Entity List; Removals from the Validated End-User (VEU) Program,” https://public-inspection.fede ralregister.gov/2024-28267.pdf, 2024, accessed: 2025-02-03

    2024

  8. [8]

    Framework for Artificial Intelligence Diffusion,

    ——, “Framework for Artificial Intelligence Diffusion,” https://www. govinfo.gov/content/pkg/FR-2025-01-15/pdf/2025-00636.pdf, 2025, accessed: 2025-02-03

    2025

  9. [9]

    Evolution of the Graphics Processing Unit (GPU),

    W. J. Dally, S. W. Keckler, and D. B. Kirk, “Evolution of the Graphics Processing Unit (GPU),” IEEE Micro, vol. 41, no. 6, pp. 42–51, 2021

    2021

  10. [10]

    Training compute growth is driven by larger clusters, longer training, and better hardware,

    L. Emberson and D. Owen, “Training compute growth is driven by larger clusters, longer training, and better hardware,” 2025, accessed: 2025-12-16. [Online]. Available: https://epoch.ai/data-insights/training -compute-decomposition

    2025

  11. [11]

    Compute trends across three eras of machine learning,

    J. Sevilla, L. Heim, A. Ho, T. Besiroglu, M. Hobbhahn, and P. Villalobos, “Compute trends across three eras of machine learning,” in 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, Jul. 2022, p. 1–8. [Online]. Available: http: //dx.doi.org/10.1109/IJCNN55064.2022.9891914

    arXiv 2022

  12. [12]

    Llms now accept longer inputs, and the best models can use them more effectively,

    G. Burnham and T. Adamczewski, “Llms now accept longer inputs, and the best models can use them more effectively,” https://epoch.ai/data-i nsights/context-windows, 2025, accessed: 2026-01-09

    2025

  13. [13]

    NVIDIA Tesla P100,

    NVIDIA, “NVIDIA Tesla P100,” https://images.nvidia.com/content/pd f/tesla/whitepaper/pascal-architecture-whitepaper.pdf, 2016, accessed: 2024-12-17

    2016

  14. [14]

    Cramming more components onto integrated circuits, reprinted from electronics, volume 38, number 8, april 19, 1965, pp.114 ff

    G. E. Moore, “Cramming more components onto integrated circuits, reprinted from electronics, volume 38, number 8, april 19, 1965, pp.114 ff.” IEEE Solid-State Circuits Society Newsletter, vol. 11, no. 3, pp. 33–35, 2006

    1965

  15. [15]

    Progress in digital integrated electronics,

    G. E. Moore et al., “Progress in digital integrated electronics,” in Electron devices meeting, vol. 21. Washington, DC, 1975, pp. 11– 13

    1975

  16. [16]

    The End of Moore’s Law: A New Begin- ning for Information Technology,

    T. N. Theis and H.-S. P. Wong, “The End of Moore’s Law: A New Begin- ning for Information Technology,”Computing in Science & Engineering, vol. 19, no. 2, pp. 41–50, 2017

    2017

  17. [17]

    There’s plenty of room at the Top: What will drive computer performance after Moore’s law?

    C. E. Leiserson, N. C. Thompson, J. S. Emer, B. C. Kuszmaul, B. W. Lampson, D. Sanchez, and T. B. Schardl, “There’s plenty of room at the Top: What will drive computer performance after Moore’s law?” Science, vol. 368, no. 6495, p. eaam9744, 2020

    2020

  18. [18]

    Move over, Moore’s law. Make way for Huang’s law [Spectral Lines],

    T. S. Perry, “Move over, Moore’s law. Make way for Huang’s law [Spectral Lines],” IEEE Spectrum, vol. 55, no. 5, pp. 7–7, 2018

    2018

  19. [19]

    Measuring moore’s law: evidence from price, cost, and quality indexes,

    K. Flamm, “Measuring moore’s law: evidence from price, cost, and quality indexes,” in Measuring and Accounting for Innovation in the 21st Century. University of Chicago Press, 2021

    2021

  20. [20]

    Intel Xeon 6978P,

    TechPowerUp, “Intel Xeon 6978P,” https://www.techpowerup.com/cp u-specs/xeon-6978p.c4279, 2025, accessed: 2026-01-07

    2025

  21. [21]

    AMD Ryzen Threadripper PRO 9995WX,

    ——, “AMD Ryzen Threadripper PRO 9995WX,” https://www.techpo werup.com/cpu-specs/ryzen-threadripper-pro-9995wx.c4163, 2025, accessed: 2026-01-07

    2025

  22. [22]

    Inside NVIDIA Blackwell Ultra: The Chip Powering the AI Factory Era,

    NVIDIA, “Inside NVIDIA Blackwell Ultra: The Chip Powering the AI Factory Era,” https://developer.nvidia.com/blog/inside-nvidia-black well-ultra-the-chip-powering-the-ai-factory-era/, 2025, accessed: 2025-11-30

    2025

  23. [23]

    NVDA: NVIDIA Corp - Stock Price, Quote and News,

    CNBC, “NVDA: NVIDIA Corp - Stock Price, Quote and News,” https: //www.cnbc.com/quotes/NVDA, 2026, accessed: 2026-01-12

    2026

  24. [24]

    Heeding Huang’s Law: Video Shows How Engineers Keep the Speedups Coming,

    NVIDIA, “Heeding Huang’s Law: Video Shows How Engineers Keep the Speedups Coming,” https://blogs.nvidia.com/blog/huangs-law-dally -hot-chips/, 2023, accessed: 2024-12-17

    2023

  25. [25]

    Trends in machine learning hardware,

    M. Hobbhahn, L. Heim, and G. Aydos, “Trends in machine learning hardware,” 2023, accessed: 2025-12-08. [Online]. Available: https://epoch.ai/blog/trends-in-machine-learning-hardware

    2023

  26. [26]

    Data on machine learning hardware

    Epoch AI, “Data on machine learning hardware”,” 10 2024, accessed: 2026-01-17. [Online]. Available: https://epoch.ai/data/machine-learnin g-hardware

    2024

  27. [27]

    NVIDIA BlueField Platform,

    NVIDIA, “NVIDIA BlueField Platform,” https://www.nvidia.com/en-us/ networking/products/data-processing-unit/, 2025, accessed: 2025-12-16

    2025

  28. [28]

    Accelerating Data Center AI with the NVIDIA Converged Ac- celerator Developer Kit,

    ——, “Accelerating Data Center AI with the NVIDIA Converged Ac- celerator Developer Kit,” https://developer.nvidia.com/blog/accelerating -data-center-ai-with-the-nvidia-converged-accelerator-developer-kit/, 2021, accessed: 2025-12-15

    2021

  29. [29]

    Converged Accelerators,

    ——, “Converged Accelerators,” https://www.nvidia.com/content/dam /en-zz/Solutions/gtcf21/converged-accelerator/pdf/datasheet.pdf, 2023, accessed: 2025-12-15

    2023

  30. [30]

    The evolution of GDDR: From GDDR1 to GDDR7,

    Micron, “The evolution of GDDR: From GDDR1 to GDDR7,” https: //www.micron.com/about/blog/memory/dram/the-evolution-of-gddr-fro m-gddr1-to-gddr7, 2024, accessed: 2025-01-29

    2024

  31. [31]

    NVIDIA GTC 2025 - Built For Reasoning, Vera Rubin, Kyber, CPO, Dynamo Inference, Jensen Math, Feynman,

    SemiAnalysis, “NVIDIA GTC 2025 - Built For Reasoning, Vera Rubin, Kyber, CPO, Dynamo Inference, Jensen Math, Feynman,” https://news letter.semianalysis.com/p/nvidia-gtc-2025-built-for-reasoning-vera-rub JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 21 in-kyber-cpo-dynamo-inference-jensen-math-feynman, 2025, accessed: 2025-12-01

    2025

  32. [32]

    Nvidia gh200 grace hopper superchip,

    NVIDIA, “Nvidia gh200 grace hopper superchip,” https://www.nvid ia.com/en-us/data-center/grace-hopper-superchip/, 2022, accessed: 2025-12-05

    2022

  33. [33]

    Nvidia gb200 nvl72,

    ——, “Nvidia gb200 nvl72,” https://www.nvidia.com/en-us/data-center/ gb200-nvl72/, 2024, accessed: 2025-12-05

    2024

  34. [34]

    Tesla K80 GPU Accelerator,

    NVIDIA, “Tesla K80 GPU Accelerator,” https://www.nvidia.com/conte nt/dam/en-zz/Solutions/Data-Center/tesla-product-literature/Tesla-K80 -BoardSpec-07317-001-v05.pdf, 2015, accessed: 2025-11-30

    2015

  35. [35]

    J. P. Researcg, https://www.jonpeddie.com/news/pc-aib-shipments-follo w-seasonality-show-nominal-increase-for-q424/, 2025

    2025

  36. [36]

    Bloomberg, https://www.bloomberg.com/news/articles/2025-02-25/ab out-nvidia-hopper-blackwell-chips-and-why-amd-intel-struggle-to-c ompete, 2025

    2025

  37. [37]

    Ampere Generation,

    TechPowerUp, “Ampere Generation,” https://www.techpowerup.com/ gpu-specs/?architecture=Ampere&sort=generation, 2024, accessed: 2024-12-17

    2024

  38. [38]

    GeForce RTX 30 Series,

    NVIDIA, “GeForce RTX 30 Series,” https://www.nvidia.com/en-us/ge force/graphics-cards/30-series/, 2021, accessed: 2024-12-17

    2021

  39. [39]

    NVIDIA A100 Tensor Core GPU,

    ——, “NVIDIA A100 Tensor Core GPU,” https://www.nvidia.com/en-u s/data-center/a100/, 2020, accessed: 2024-12-17

    2020

  40. [40]

    TechPowerUp, https://www.techpowerup.com, 2024, accessed: 2024- 12-17

    2024

  41. [41]

    VideoCardz, https://videocardz.com, 2024, accessed: 2024-12-17

    2024

  42. [42]

    NVIDIA A100 Tensor Core GPU Architecture,

    NVIDIA, “NVIDIA A100 Tensor Core GPU Architecture,” https://imag es.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-arc hitecture-whitepaper.pdf, 2020, accessed: 2025-01-07

    2020

  43. [43]

    NVIDIA H100 Tensor Core GPU Architecture,

    ——, “NVIDIA H100 Tensor Core GPU Architecture,” https://resour ces.nvidia.com/en-us-tensor-core/gtc22-whitepaper-hopper, 2023, accessed: 2024-12-17

    2023

  44. [44]

    NVIDIA HGX Platform,

    NVIDIA, “NVIDIA HGX Platform,” https://www.nvidia.com/en-us/da ta-center/hgx/, 2025, accessed: 2025-11-30

    2025

  45. [45]

    NVIDIA Ampere GA102 GPU Architecture,

    NVIDIA, “NVIDIA Ampere GA102 GPU Architecture,” https://www. nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-architecture-white paper-v2.pdf, 2021, accessed: 2025-01-10

    2021

  46. [46]

    Structured Sparsity in the NVIDIA Ampere Architecture and Applications in Search Engines,

    ——, “Structured Sparsity in the NVIDIA Ampere Architecture and Applications in Search Engines,” https://developer.nvidia.com/blog/stru ctured-sparsity-in-the-nvidia-ampere-architecture-and-applications-in-s earch-engines/, 2023, accessed: 2025-01-07

    2023

  47. [47]

    Llm.int8(): 8-bit matrix multiplication for transformers at scale,

    T. Dettmers, M. Lewis, Y . Belkada, and L. Zettlemoyer, “Llm.int8(): 8-bit matrix multiplication for transformers at scale,” 2022. [Online]. Available: https://arxiv.org/abs/2208.07339

    Pith/arXiv arXiv 2022

  48. [48]

    Deep learning with limited numerical precision,

    S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep learning with limited numerical precision,” 2015. [Online]. Available: https://arxiv.org/abs/1502.02551

    Pith/arXiv arXiv 2015

  49. [49]

    Training deep neural networks with low precision multiplications,

    M. Courbariaux, Y . Bengio, and J.-P. David, “Training deep neural networks with low precision multiplications,” 2015. [Online]. Available: https://arxiv.org/abs/1412.7024

    Pith/arXiv arXiv 2015

  50. [50]

    NVIDIA Blackwell Architecture Technical Brief,

    NVIDIA, “NVIDIA Blackwell Architecture Technical Brief,” https://re sources.nvidia.com/en-us-blackwell-architecture/blackwell-architectur e-technical-brief, 2025, accessed: 2025-12-10

    2025

  51. [51]

    J. L. Hennessy and D. A. Patterson, Computer Architecture, Sixth Edition: A Quantitative Approach, 6th ed. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2017

    2017

  52. [52]

    statsmodels: Econometric and statistical modeling with python,

    S. Seabold and J. Perktold, “statsmodels: Econometric and statistical modeling with python,” in 9th Python in Science Conference, 2010

    2010

  53. [53]

    A primer on visualizations for comparing populations, including the issue of overlapping confidence intervals,

    T. Wright, M. Klein, and J. Wieczorek, “A primer on visualizations for comparing populations, including the issue of overlapping confidence intervals,” The American Statistician, vol. 73, no. 2, pp. 165–178, 2019

    2019

  54. [54]

    How to use cuda core and tensor core simultaneously?

    NVIDIA, “How to use cuda core and tensor core simultaneously?” https: //forums.developer.nvidia.com/t/how-to-use-cuda-core-and-tensor-cor e-simultaneously/223881, 2022, accessed: 2025-01-07

    2022

  55. [55]

    Scaling the bandwidth wall: challenges in and avenues for cmp scaling,

    B. M. Rogers, A. Krishna, G. B. Bell, K. Vu, X. Jiang, and Y . Solihin, “Scaling the bandwidth wall: challenges in and avenues for cmp scaling,” SIGARCH Comput. Archit. News, vol. 37, no. 3, p. 371–382, Jun

  56. [56]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” 2020. [Online]. Available: https://arxiv.org/abs/1909.08053

    Pith/arXiv arXiv 2020

  57. [57]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” 2020. [Online]. Available: https://arxiv.org/abs/2001.08361

    Pith/arXiv arXiv 2020

  58. [58]

    Relativistic bondi accretion for stiff equations of state,

    C. B. Richards, T. W. Baumgarte, and S. L. Shapiro, “Relativistic bondi accretion for stiff equations of state,” Monthly Notices of the Royal Astronomical Society, vol. 502, no. 2, p. 3003–3011, Jan. 2021. [Online]. Available: http://dx.doi.org/10.1093/mnras/stab161

    work page doi:10.1093/mnras/stab161 2021

  59. [59]

    Allison and E

    G. Allison and E. Schmidt, Is China beating the US to AI supremacy? Harvard Kennedy School, Belfer Center for Science and International Affairs, 2020

    2020

  60. [60]

    America, china, and the struggle for ai supremacy,

    E. H. Christie, C. Buts, and C. Du Bois, “America, china, and the struggle for ai supremacy,” in 24th Annual International Conference on Economics and Security, V olos,Greece, 2021

    2021

  61. [61]

    The AI Superpower Showdown,

    Medium, “The AI Superpower Showdown,” https://medium.com/@mc raddock/inside-the-us-china-race-for-technological-supremacy-52cb5 c3df063, 2025, accessed: 2025-02-08

    2025

  62. [62]

    Nvidia’s New China AI Chips Circumvent US Restric- tions — H20 Faster Than H100 — Huawei Ascend 910B,

    SemiAnalysis, “Nvidia’s New China AI Chips Circumvent US Restric- tions — H20 Faster Than H100 — Huawei Ascend 910B,” https://news letter.semianalysis.com/p/nvidias-new-china-ai-chips-circumvent, 2023

    2023

  63. [63]

    Trump greenlights Nvidia H200 AI chip sales to China if U.S. gets 25% cut, says Xi responded positively,

    CNBC, “Trump greenlights Nvidia H200 AI chip sales to China if U.S. gets 25% cut, says Xi responded positively,” https://www.cnbc.com/2 025/12/08/trump-nvidia-h200-sales-china.html, 2025

    2025

  64. [64]

    US to allow Nvidia H200 chip shipments to China, Trump says,

    Reuters, “US to allow Nvidia H200 chip shipments to China, Trump says,” https://www.reuters.com/world/china/us-open-up-exports-nvidi a-h200-chips-china-semafor-reports-2025-12-08/, 2025

    2025

  65. [65]

    US Curbs HBM Exports to China—More for the Rest of Us,

    Morgan, Timothy P., “US Curbs HBM Exports to China—More for the Rest of Us,” https://www.nextplatform.com/2024/12/02/us-curbs-hbm-e xports-to-china-more-for-the-rest-of-us/, 2024, accessed 2025-05-22

    2024

  66. [66]

    Hwang and E

    T. Hwang and E. S. Weinstein, Decoupling in strategic technologies: From satellites to artificial intelligence. Center for Security and Emerging Technology, 2022

    2022

  67. [67]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Li et al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024

    Pith/arXiv arXiv 2024

  68. [68]

    Deepseek llm: Scaling open-source language models with longtermism,

    X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu et al., “Deepseek llm: Scaling open-source language models with longtermism,” arXiv preprint arXiv:2401.02954, 2024

    Pith/arXiv arXiv 2024

  69. [69]

    Mapping China’s HBM Advances,

    Wang, Ray and Ottinger, Lily, “Mapping China’s HBM Advances,” ht tps://www.chinatalk.media/p/mapping-chinas-hbm-advancement, 2025, accessed 2025-05-22

    2025

  70. [70]

    YMTC expected to outproduce Micron in NAND flash while SK hynix cuts output,

    Min-gyu, Hwang, “YMTC expected to outproduce Micron in NAND flash while SK hynix cuts output,” https://biz.chosun.com/en/en-it/ 2025/05/16/I7XUWDAC5JACLPTZC4AP725LFU/, 2025, accessed 2025-05-22

    2025

  71. [71]

    Why china isn’t about to leap ahead of the west on compute,

    V . Blablov ´a and R. Rahman, “Why china isn’t about to leap ahead of the west on compute,” 2025, accessed: 2025-12-16. [Online]. Available: https://epoch.ai/gradient-updates/why-china-isnt-about-to-leap-ahead-o f-the-west-on-compute

    2025

  72. [72]

    China’s AI Chip Deficit: Why Huawei Can’t Catch Nvidia and U.S. Export Controls Should Remain,

    Council on Foreign Relations, “China’s AI Chip Deficit: Why Huawei Can’t Catch Nvidia and U.S. Export Controls Should Remain,” https: //www.cfr.org/article/chinas-ai-chip-deficit-why-huawei-cant-catch-nvi dia-and-us-export-controls-should-remain, 2025, accessed: 2025-12-16

    2025

  73. [73]

    J. R. Harris, Industrial Espionage and Technology Transfer: Britain and France in the 18th Century. Routledge, 1998

    1998

  74. [74]

    D. J. Jeremy, Transatlantic Industrial Revolution: The Diffusion of Textile Technologies between Britain and America, 1790-1830s. MIT Press, 1981

    1981

  75. [75]

    Industrial policy on the frontier: lessons from the first two industrial revolutions,

    Kelly, Morgan and O’Rourke, Kevin H, “Industrial policy on the frontier: lessons from the first two industrial revolutions,” 2023

    2023

  76. [76]

    Catching up, forging ahead, and falling behind,

    M. Abramovitz, “Catching up, forging ahead, and falling behind,” The journal of economic history, vol. 46, no. 2, pp. 385–406, 1986

    1986

  77. [77]

    D. C. Mowery and N. Rosenberg, Paths of innovation: Technological change in 20th-century America. Cambridge University Press, 1999

    1999

  78. [78]

    The gifts of athena: Historical origins of the knowledge economy,

    J. Mokyr, “The gifts of athena: Historical origins of the knowledge economy,” in The gifts of Athena. princeton university press, 2011

    2011

  79. [79]

    The entrepreneurial state: Debunking public vs. private sector myths,

    M. Mazzucato, “The entrepreneurial state: Debunking public vs. private sector myths,” 2013

    2013

  80. [80]

    Fleming, Breakthrough: A Growth Revolution

    M. Fleming, Breakthrough: A Growth Revolution. Business Expert Press, 2022

    2022

Showing first 80 references.