REVIEW 32 cited by
The rising costs of training frontier AI models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
The rising costs of training frontier AI models
read the original abstract
The costs of training frontier AI models have grown dramatically in recent years, but there is limited public data on the magnitude and growth of these expenses. This paper develops a detailed cost model to address this gap, estimating training costs using three approaches that account for hardware, energy, cloud rental, and staff expenses. The analysis reveals that the amortized cost to train the most compute-intensive models has grown precipitously at a rate of 2.4x per year since 2016 (90% CI: 2.0x to 2.9x). For key frontier models, such as GPT-4 and Gemini, the most significant expenses are AI accelerator chips and staff costs, each costing tens of millions of dollars. Other notable costs include server components (15-22%), cluster-level interconnect (9-13%), and energy consumption (2-6%). If the trend of growing development costs continues, the largest training runs will cost more than a billion dollars by 2027, meaning that only the most well-funded organizations will be able to finance frontier AI models.
Forward citations
Cited by 32 Pith papers
-
Khondo: A Multimodal Benchmark for Document Packet Splitting of Bangla Forms
A new Bangla–English benchmark shows vision-language models can cluster pages of shuffled government-form packets but cannot reliably reconstruct their original page order.
-
BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension
BaFCo introduces the first fine-grained Bangla form benchmark (26 entity types, relationships) and demonstrates that current MLLMs struggle especially with granular layout localization.
-
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
FlexServe introduces recallable secure memory and NPU to enable cooperative secure LLM inference on mobile devices, reporting 10.05X TTFT speedup over a basic TrustZone strawman.
-
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
FlexServe decouples access and management of secure resources in TrustZone to enable efficient LLM inference on mobiles, reporting 10.05X TTFT speedup over basic strawman designs and 2.44X over optimized ones.
-
zkComposer: Decomposing Proof Construction to Scale zkML
zkComposer decomposes monolithic zkML proofs into parallel sub-proofs linked by shared boundary commitments, yielding up to 6.84× lower prover time on GPT-2 without new cryptographic primitives.
-
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
Page-granular Flex-Mem and switchable Flex-NPU cut TrustZone LLM TTFT by ~10× vs a CMA strawman and ~2.4× vs a pipelined secure-NPU strawman on RK3588.
-
Opti-Q: A Constraint-Based Optimization Framework for Multi-LLM Question Planning
Per-question database-style plan search over multi-LLM DAGs improves QA quality under budgets by ~58% (MMLU-Pro) and ~41% (SimpleQA) versus reimplemented baselines.
-
AI Sovereignty: A Qualitative Model of Strategic Competition as AI Becomes an Instrument of National Power
The authors introduce definitions and a qualitative model of AI sovereignty that identifies multi-scale contributors and leverage points nations can target through kinetic and non-kinetic actions to influence AI-drive...
-
Zero knowledge verification for frontier AI training is possible
Proposes zkVM-based protocol for verifiable frontier AI pre-training with committed specs, network observations, Merkle commitments, and FP precompiles, estimating 36-month POC at single-digit overhead.
-
Validity Threats for Foundation Model Research
Maps common low-compute research strategies for foundation models onto statistical, internal, external, and construct validity threats via a causal-inference lens.
-
An Asymptotic Theory of Chain-of-Thought in In-Context Learning
Exact RMT-derived formula for CoT generalization error in linear ICL reveals phase transition between exponential/polynomial improvement, saturation, and overthinking regimes depending on depth, pretraining, and conte...
-
Spectral structural distortion reveals redundant neurons in neural networks
A graph-signal-processing-based pruning framework scores neurons by spectral structural importance between layer input/output graphs and removes low-scoring ones before a final recovery fine-tune.
-
Rescaled Asynchronous SGD: Optimal Distributed Optimization under Data and System Heterogeneity
Rescaled ASGD recovers convergence to the true global objective by rescaling worker stepsizes proportional to computation times, matching the known time lower bound in the leading term under non-convex smoothness and ...
-
Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification
Cross-model semantic disagreement adds an epistemic uncertainty term that improves total uncertainty estimation over self-consistency alone, helping flag confident errors in LLMs.
-
Switching Efficiency: A Novel Framework for Dissecting AI Data Center Network Efficiency
Introduces Switching Efficiency (η) decomposed into data, routing efficiency, and port utilization factors to analyze and improve communication bottlenecks in AI data center networks for LLM training.
-
When Do We Need LLMs? A Diagnostic for Language-Driven Bandits
Lightweight numerical bandits on text embeddings match or exceed LLM accuracy in contextual bandits at a fraction of the cost, with an embedding-based diagnostic to choose between them.
-
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
FlexServe achieves up to 10x faster time-to-first-token for secure LLM inference on mobile devices by using flexible resource isolation in TrustZone compared to standard approaches.
-
Optimal Resource Allocation for ML Model Training and Deployment under Concept Drift
Optimal training uses a single front-loaded burst when concept durations are DMRL, and back-loading when they are IMRL; deployment schedules are treated as quasi-convex optimization problems.
-
Unraveling Syntax: Language Modeling and the Substructure of Grammars
Language-modeling loss decomposes linearly over the sub-grammars of a probabilistic context-free grammar, and models learn these sub-grammars in parallel rather than in stages.
-
Spectral structural distortion reveals redundant neurons in neural networks
A graph-spectral importance score based on layer-wise structural distortion between pre- and post-activation neuron graphs identifies removable neurons for iterative pruning without intermediate updates, followed by r...
-
Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP
FCP shards sequences at block level with flexible P2P communication and bin-packing to achieve near-linear scaling up to 256 GPUs and 1.13x-2.21x higher attention MFU in foundation model pre-training.
-
Who Prices Cognitive Labor in the Age of Agents? Compute-Anchored Wages
AI agents convert compute capital into cognitive labor units, so on substitutable tasks the competitive human wage is bounded above by relative productivity times compute intensity times the rental rate of compute.
-
Physics Priors Offer Useful Accuracy-Carbon Trade-Offs in Spatio-Temporal Forecasting
Stronger physics priors in neural networks for spatio-temporal shear flow forecasting yield substantially lower training carbon footprints than weak or no priors, though inference savings are less consistent.
-
The Human-AI Substitution Principle: When will you be replaced by AI in your organization?
AI replaces a human role whenever its risk-adjusted cost is lower; the paper packages this comparison with hierarchy and risk to derive conditional organizational predictions.
-
The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model
A scaling law model derived from roofline analysis and a speedup-based efficiency factor predicts training energy for BERT models across GPU parallelism configurations.
-
From Cradle to Cloud: A Life Cycle Review of AI's Environmental Footprint
A review of AI sustainability studies finds inconsistent life cycle definitions and predominant reliance on coarse CO2e proxies, with limited coverage of water, materials, and multi-impact assessments.
-
Beware of GeeksBearing Gifts: Building True EU Frontier AI Sovereignty
Authors propose a five-pillar sovereignty framework mapped to a five-layer AI stack with 26 components to identify gaps in EU frontier AI policy.
-
Revisiting Training Scale: An Empirical Study of Token Count, Power Consumption, and Parameter Efficiency
The paper's finding that training efficiency declines monotonically with token count is guaranteed by its efficiency metric, which divides by token count and power consumption.
-
Who Prices Cognitive Labor in the Age of Agents? Compute-Anchored Wages
AI agents are a compute-to-cognitive-labor technology, bounding the competitive wage for substitutable human cognitive tasks by the rental rate of compute capital.
-
The End of the Foundation Model Era: Open-Weight Models, Sovereign AI, and Inference as Infrastructure
Open-weight models have ended the foundation model era by eliminating pre-training as a durable moat and enabling sovereign AI control through direct access to model weights.
-
LLMOrbit: A Circular Taxonomy of Large Language Models -From Scaling Walls to Agentic AI Systems
A survey taxonomy of LLMs identifies three scaling crises and six efficiency paradigms while tracing the shift from generation to tool-using agents.
-
Towards EnergyGPT: A Large Language Model Specialized for the Energy Sector
Fine-tuned LLaMA 3.1-8B variants for the energy sector outperform the base model on domain QA benchmarks, with LoRA delivering similar gains at lower training cost.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.