Pith. sign in

REVIEW 4 major objections 5 minor 8 cited by

Densing Law of LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The maximum capability density of open-source LLMs grows exponentially, doubling about every 3.3 months.

desk verdict A useful new metric and a plausible-sounding trend, but the doubling time rests on an unvalidated 250x extrapolation of a scaling law fitted to 0.8B-parameter models, so the paper needs major revision before its central claim can be trusted. read the letter →

arxiv 2412.04315 v2 pith:CAMLNABW submitted 2024-12-05 cs.AI cs.CL

Chaojun Xiao , Jie Cai , Weilin Zhao , Guoyang Zeng , Biyuan Lin , Jie Zhou , Zhi Zheng , Xu Han
show 2 more authors
Zhiyuan Liu Maosong Sun
This is my paper · ORCID
classification cs.AIcs.CL
keywords capabilitydensityDensinglaweffectiveparametersizelargelanguagemodelsLLMscalinglawsmodelefficiencybenchmarkevaluationexponentialgrowth
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that LLM progress can be quantified not only by what a model can do but by what it does per parameter, and that this quantity follows a predictable exponential trend. It introduces capability density, defined as the ratio of a model's effective parameter size to its actual parameter size, where the effective size is the parameter count a reference LLM would need to match the target's benchmark scores. Using 29 open-source base models and five common benchmarks, it reports that the maximum density has grown as $\ln \rho_{\max} = A t + B$ with $A \approx 0.007$, a doubling roughly every 3.3 months. On this account, a model with about half the parameters of today's best will match its performance in about one quarter, turning model efficiency into a forecastable quantity.

What carries the argument

The load-bearing object is the capability-density metric $\rho(M) = \hat{N}(S_M)/N_M$ and the calibration chain behind it. Effective parameter size $\hat{N}(S_M)$ is the inverse of a two-step scaling function: first a power law maps parameter count $N$ and training tokens $D$ to conditional language-modeling loss on test items, then a sigmoid maps that loss to downstream benchmark performance. The chain converts any model's public benchmark scores into a parameter-equivalent size, making models of different scales comparable on a single efficiency axis; the envelope fit over release dates of this ratio is what produces the Densing Law.

What would settle it

Compute the effective parameter size of a held-out large model from a different family (for example, a 100B-plus model trained on non-reference data) using its measured scores on the five benchmarks, and compare it with the performance predicted by the calibrated curves. If the error is much larger than the residual scatter of the fits in the paper's Figure 2, or if refitting the two curves without the reference checkpoints changes the fitted slope $A$ enough to push the doubling time outside the two-to-six-month range, the universal calibration and the 3.3-month doubling time fail.

Watch

Extended reading notes

Core claim

The paper's central claim is that there exists a measurable quantity, capability density, that has been increasing exponentially for open-source base LLMs. Density is defined as $\rho(M) = \hat{N}(S_M)/N_M$, where $N_M$ is the model's actual parameter count and $\hat{N}(S_M)$ is the effective parameter size: the number of parameters a reference model trained on $D_0 = 10^{12}$ tokens would need to reach the same downstream score $S_M$. The effective size is obtained by composing two fitted maps, a power-law loss curve $L = aN^{-\alpha} + bD^{-\beta}$ and a sigmoid performance curve $S = c/(1+e^{-\gamma(L-l)}) + d$, calibrated on small reference models and intermediate checkpoints. On MMLU, BBH, MATH, HumanEval, and MBPP, the maximum density of released models fits $\ln \rho_{\max} = A t + B$ with $A \approx 0.007$ (about 0.0073 in the detailed fit), i.e. a doubling time of about 95 days.

Load-bearing premise

The whole measurement stands on the assumption that the loss-to-performance and loss-to-parameters curves fitted on small reference models and one family's checkpoints apply universally to every model family and every scale, including models hundreds of times larger.

Editorial extensions

If this is right

  • Inference cost for a fixed performance level falls exponentially, with the paper's API price data showing a halving roughly every 2.6 months.
  • Density growth compounded with chip-density growth means the largest effective model that fits on a given chip grows far faster than hardware improvements alone would suggest.
  • After ChatGPT's release the fitted density-growth slope rose from about 0.0048 to 0.0073 per day, a 50 percent acceleration.
  • Existing pruning and distillation outputs mostly show lower density than their source models, so compression is not automatically a density improvement.
  • Model development should target density rather than raw performance, because an over-large model's capability may be matched by a smaller model released months later at a fraction of the cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper does not make: if the doubling time persists, a multi-month training run is a depreciating asset, and training decisions should discount future density gains.
  • A neighbouring application: re-base density on inference FLOPs rather than parameters, which would let the law be tested against reasoning models that trade extra test-time compute for accuracy.
  • A measurement caution: as models saturate the five current benchmarks, measured density growth will flatten even if real capability growth continues, so keeping the law testable requires continually harder evaluation sets.
  • An application to closed models: applying the same effective-parameter mapping to public benchmark scores would turn density into a price-performance index for API offerings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces a metric, capability density, defined as the ratio between a model's effective parameter size and its actual parameter size. The effective parameter size is computed by inverting a two-step scaling law: Eq. (2) predicts conditional language-modeling loss from parameter count N and training tokens D, fitted on six small reference models (0.005B–0.8B parameters) trained on the MiniCPM-3 corpus; Eq. (3) maps loss to downstream benchmark performance using MiniCPM-3 models and checkpoints. Applying this calibration to 29 open-source base LLMs, the paper reports that the maximum capability density follows ln(ρmax) = 0.0073 t + B (Eq. 5), corresponding to a doubling every ~95 days, and draws several corollaries about inference cost, Moore's law, and training efficiency.

Significance. The concept of a unified effectiveness–efficiency metric is useful and timely, and the paper is clearly structured. If the Densing Law could be validated with a calibration that demonstrably extrapolates across scales and model families, the result would be practically important for planning model training and deployment. The authors deserve credit for framing the density idea, for using public benchmarks and models, and for explicitly discussing benchmark contamination and other limitations. However, the quantitative law as stated is not yet supported by the evidence in the manuscript: the central claim rests on an unvalidated extrapolation of the loss-scaling law and on a small envelope fit, as detailed below.

major comments (4)
  1. [§2.2 and §2.4, Eq. (2) and Eq. (4), Table 1] The scaling law in Eq. (2) is fitted on reference models with parameter counts at most 0.8B and training tokens at most 60×N ≈ 48B, yet Eq. (4) inverts this law at D0 = 1T and assigns effective parameter sizes of tens to hundreds of billions to models on the 2024 envelope. This is a 20× extrapolation in D and a 250× extrapolation in N beyond the fitted range, and the paper gives no check that the additive power-law surface holds there. Since N_hat enters the numerator of every density value, any uncontrolled deviation in the loss surface biases the density values and, because the bias grows as measured losses decrease over time, can create a spurious exponential trend in Eq. (5). A minimally necessary addition is to validate Eq. (2) against the conditional losses of the MiniCPM-3 large checkpoints already used to fit Eq. (3), or against other large open models.
  2. [§3.2, Eq. (3), Figure 2(b)] The loss-to-performance curve is fit exclusively on MiniCPM-3 models and checkpoints, and the same family supplies some of the highest-density points that determine the envelope (e.g., MiniCPM-1-2.4B and MiniCPM-3-4B in Figure 1). The mapping from loss to benchmark performance is treated as universal across architectures and pretraining-data distributions, but no held-out validation across families is provided. A concrete test would be to fit Eq. (3) on all evaluated models except one family (e.g., Llama-3) and check whether predicted performance for the held-out family is unbiased; without such a test, the density of non-MiniCPM models is potentially miscalibrated, and the envelope is partly constructed from the same family that defines the ruler.
  3. [§3.1, Figure 1] The density calculations mix benchmark scores taken from each model's technical report, which differ in evaluation harnesses, prompt templates, few-shot exemplars, and contamination-cleaning procedures. Because N_hat is a steeply nonlinear function of S (the sigmoid in Eq. (3) is near its midpoint for many models), small protocol differences can change density by a large factor. The paper itself acknowledges contamination issues in Section 5, but the values in Figure 1 are still computed from heterogeneous reported scores. To support a quantitative claim of a 3.3-month doubling time, the authors should recompute all scores in a single harness (for example, OpenCompass, which they already use) for all 29 models, and state which version of each benchmark was used.
  4. [§3.3, Eq. (5)] The Densing Law is fit to the upper envelope of density versus time, with roughly one envelope point per several months in Figure 1 and no explicit list of which models are used as envelope points. The paper reports R2=0.912 but not the number of points, the standard error of A, or a sensitivity analysis. Removing a single envelope model could plausibly change the doubling time by months. Please report the envelope points, the uncertainty in A, a leave-one-out analysis, and a comparison against a null model with release dates permuted.
minor comments (5)
  1. [§3.4, Corollary 3] The comparison of slopes before and after ChatGPT (0.0048 vs. 0.0073) is not accompanied by a statistical test, and the two regressions use different time spans and model subsets; the claim that growth accelerated by 50% should either be tested or softened.
  2. [§2.2 and §3.2] The fitted parameters of Eqs. (2) and (3) are not reported; without them, the density values are not reproducible. Please include a table with a, α, b, β for Eq. (2) and c, γ, l, d for Eq. (3) for each benchmark.
  3. [Abstract and §1] The manuscript uses both "capability density" and "capacity density" for the same concept; please choose one term and use it consistently throughout.
  4. [§2.2, Reference [OpenAI, 2023]] The text cites OpenAI (2023) for GPT-4o, but that reference is the GPT-4 technical report; please correct the citation or use the appropriate GPT-4o technical report.
  5. [Figure 2(a)] Figure 2(a) is drawn as a function of compute 6ND, while Eq. (2) is a function of N and D separately; the paper should explain how the plotted curves follow from the fitted function, since the same loss can correspond to different (N,D) combinations.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the Densing Law is a fitted trend over independently benchmarked open models, not a quantity reduced by the calibration curves by construction.

full rationale

The paper's derivation chain is: fit Eq. 2 (loss vs. N and D) on six small reference models from Table 1; fit Eq. 3 (loss vs. downstream score) on MiniCPM-3 checkpoints; invert the composition via Eq. 4 at D0=1T to assign each evaluated model an effective parameter size; then fit Eq. 5 to the envelope of these densities over time. None of these equations defines the exponential law in terms of itself. The densities are measurements transformed by fixed calibration curves, and the slope A is fitted afterward to values that include models outside the MiniCPM family, such as Llama, Phi, Gemma, and Mistral. The fact that some MiniCPM models appear both in the calibration set and in Figure 1 does not force the exponential trend, because least-squares fitting does not make any single model's density equal to a fitted parameter, and the envelope also contains independent models. The real weaknesses are extrapolation and external validity: Eq. 2 is fit at N up to 0.8B and D up to about 48B tokens but is inverted at D0=1T for models up to Llama-3.1-405B, and the sigmoid ceiling in Eq. 3 can inflate effective parameter sizes for very high-scoring models. These are serious correctness risks, but they are not circular reductions, so they do not raise the circularity score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces a new metric (capacity density) rather than a new physical entity. Its free parameters are the fitted scaling coefficients and the hand-chosen reference token count; the key axioms are the assumed functional forms and the comparability of reported scores.

free parameters (4)
  • a, alpha, b, beta (Eq. 2) = not reported
    Power-law coefficients mapping parameter count N and training tokens D to conditional loss L. Fitted to small reference models trained on the MiniCPM-3 corpus; used to invert loss to effective parameter size.
  • c, gamma, l, d (Eq. 3) = not reported
    Sigmoid coefficients mapping loss to benchmark performance. Fitted to MiniCPM-3 checkpoints and small open models; used to invert performance to loss.
  • A, B (Eq. 5) = A ~ 0.0073, B not reported
    Slope and intercept of the linear fit between log maximum density and time. The slope determines the claimed doubling time of about 95 days.
  • D0 (reference training tokens) = 1T tokens (chosen by hand)
    The default training data size used to define the effective parameter size. Different D0 values would change every density value and could alter the fitted trend.
assumptions (5)
  • domain assumption Conditional loss on downstream tasks follows the power law L = aN^{-alpha} + bD^{-beta} (Eq. 2).
    Assumed without testing alternative functional forms; the small-model fits are extrapolated to much larger models.
  • domain assumption Downstream performance is a sigmoid function of loss, S = c/(1+e^{-gamma(L-l)}) + d (Eq. 3).
    Assumed to hold uniformly across model families, despite the loss-performance curve being fitted mainly on MiniCPM-3 models.
  • domain assumption Benchmark scores reported in different technical reports are comparable.
    The paper prioritizes official reported scores (Section 3.1) and does not re-evaluate all 29 models under one protocol.
  • domain assumption The five chosen benchmarks are representative of overall LLM capability.
    The density trend depends on this selection; the paper acknowledges different benchmarks give different growth rates.
  • domain assumption GPT-4o-generated reasoning steps adequately represent task outputs for loss computation.
    Loss is computed on reasoning chains produced by GPT-4o for all test instances (Section 2.2), which introduces dependence on an external model's output distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Densing Law of LLMs." pith.science (2026). https://pith.science/paper/CAMLNABW

@misc{pith2026241204315,
  author       = {Pith},
  title        = {Pith review of: Densing Law of LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CAMLNABW}},
  note         = {Machine review of arXiv:2412.04315}
}
read the original abstract

Large Language Models (LLMs) have emerged as a milestone in artificial intelligence, and their performance can improve as the model size increases. However, this scaling brings great challenges to training and inference efficiency, particularly for deploying LLMs in resource-constrained environments, and the scaling trend is becoming increasingly unsustainable. This paper introduces the concept of ``\textit{capacity density}'' as a new metric to evaluate the quality of the LLMs across different scales and describes the trend of LLMs in terms of both effectiveness and efficiency. To calculate the capacity density of a given target LLM, we first introduce a set of reference models and develop a scaling law to predict the downstream performance of these reference models based on their parameter sizes. We then define the \textit{effective parameter size} of the target LLM as the parameter size required by a reference model to achieve equivalent performance, and formalize the capacity density as the ratio of the effective parameter size to the actual parameter size of the target LLM. Capacity density provides a unified framework for assessing both model effectiveness and efficiency. Our further analysis of recent open-source base LLMs reveals an empirical law (the densing law)that the capacity density of LLMs grows exponentially over time. More specifically, using some widely used benchmarks for evaluation, the capacity density of LLMs doubles approximately every three months. The law provides new perspectives to guide future LLM development, emphasizing the importance of improving capacity density to achieve optimal results with minimal computational overhead.

Figures

Figures reproduced from arXiv: 2412.04315 by the authors.

Figure 1
Figure 1. presents the capability density of popular LLMs, measured by their performance on 5 widely-used benchmarks. A trend is fitted between maximum capability density and release date, revealing that A ≈ 0.007 with R2 ≈ 0.93. This indicates the maximum capability density of LLMs doubles approximately every 3.3 months1 . That means, around three months, it is possible to achieve performance comparable to current state-of-t… view at source ↗
Figure 2
Figure 2. The results for loss estimation and performance estimation. Here, the lines are fitted curves. X-axis in (a) refers to the pre-training compute, which is approximated by Compute = 6ND. Triangles in (b) are larger models for prediction. a widely-used edge-size model, to train the small models. As for the model architecture, we use grouped query attention (Ainslie et al., 2023), gated feedforward layers with SiLU as t… view at source ↗
Figure 3
Figure 3. Prices of LLMs that can outperform GPT-3.5. The line connects the cheapest models. To better illustrate the decreasing trend in in￾ference costs for LLMs, we present the API pricing of LLMs that have achieved superior performance to GPT-3.5 since its release in Fig￾ure 3. From the figure, we can observe that the prices of LLMs exhibit an exponential decline. Specifically, in December 2022, GPT-3.5 cost $20 for one m… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Density evaluated using MMLU. Two trend lines represent the growth of LLMs’ density before and after the release of ChatGPT. as smartphones and PCs, with low power consumption. This convergence of algorithmic efficiency and hardware capability is paving the way for mor…
Figure 5
Figure 5. Figure 5: Comparison between compressed mod￾els and their larger counterparts. The results show that only the Gemma-2-9B model has a higher density than the original model, whereas all other compressed models have lower densities compared to their original counterparts. Intuitiv…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Predicting LLM Reasoning Performance with Small Proxy Model

    cs.LG 2025-09 conditional novelty 6.0 of 10

    rBridge uses a small proxy model's confidence-weighted likelihood of a frontier model's reasoning traces to predict and rank large-model reasoning performance across scales.

  2. A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A skill-graph random-walk model gives closed-form accuracy-versus-compute formulas for four reasoning strategies and connects them to training scaling.

  3. Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Reflective Verification fuses a target LLM's normal and reflection-prompted logits to accept semantically correct draft tokens, increasing accepted draft length and decoding speed by 5-15%.

  4. Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Ultra-FineWeb is a fastText-filtered pretraining corpus whose seed samples were chosen by a cheap 'efficient verification' step, and 1.2B models trained on it outperform models trained on FineWeb and FineWeb-edu on av...

  5. MiniCPM4: Ultra-Efficient LLMs on End Devices

    cs.CL 2025-06 conditional novelty 5.0 of 10

    MiniCPM4-8B reportedly matches Qwen3-8B on standard benchmarks while using about 22% of the training tokens, and achieves large long-context speedups on edge devices.

  6. Know What, Know Why: Semantic Hazard Communication for Intelligent V2X Systems

    eess.SP 2025-09 reject novelty 4.0 of 10

    SEE-V2X transmits scene-level hazard descriptions from road cameras to cars for an augmented-reality view; the claimed traffic-efficiency gain is based on a simulation with no equations, error bars, or code.

  7. Semantic Retention and Extreme Compression in LLMs: Can We Have Both?

    cs.CL 2025-05 conditional novelty 4.0 of 10

    A 25% pruning plus 4-bit quantization configuration retains roughly 20% more benchmark performance than 3-bit quantization alone at matching theoretical compression rates, across two LLMs.

  8. Language Games as the Pathway to Artificial Superhuman Intelligence

    cs.AI 2025-01 conditional novelty 4.0 of 10

    A position paper arguing that open-ended language games with fluid roles, varied rewards, and evolving rules can drive expanded data reproduction and thus a path to artificial superhuman intelligence.

Reference graph

Works this paper leans on

58 extracted references · 13 canonical work pages · cited by 8 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  3. [3]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 4895--4901, 2023

    work page 2023

  4. [4]

    The falcon series of open language models

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, M \'e rouane Debbah, \'E tienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311.16867, 2023

    arXiv 2023

  5. [5]

    Welcome to llmflation – llm inference cost is going down fast

    Guido Appenzeller. Welcome to llmflation – llm inference cost is going down fast. Blog, 2024. URL https://a16z.com/llmflation-llm-inference-cost/

    work page 2024

  6. [6]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  7. [7]

    Stable lm 2 1.6 b technical report

    Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. Stable lm 2 1.6 b technical report. arXiv preprint arXiv:2402.17834, 2024

    arXiv 2024

  8. [8]

    Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Du...

    arXiv 2021

Show all 58 references
  1. [9]

    Bradley C. A. Brown, Jordan Juravsky, Ryan Saul Ehrlich, Ronald Clark, Quoc V. Le, Christopher R \' e , and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. CoRR, abs/2407.21787, 2024. doi:10.48550/ARXIV.2407.21787. URL https://doi.o...

    2024 doi arXiv

  2. [10]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jef...

    2020

  3. [11]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  4. [12]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  5. [13]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    2023 arXiv

  6. [14]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 0 16344--16359, 2022

    2022

  7. [15]

    Training on the test task confounds evaluation and emergence

    Ricardo Dominguez-Olmedo, Florian E Dorner, and Moritz Hardt. Training on the test task confounds evaluation and emergence. arXiv preprint arXiv:2407.07890, 2024

    2024 arXiv

  8. [16]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  9. [17]

    Textbooks are all you need

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C \'e sar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023

    2023 arXiv

  10. [18]

    Apple intelligence foundation language models

    Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung - Cheng Chiu, David Qiu, et al. Apple intelligence foundation language models. CoRR, abs/2407.21075, 2024. doi:10.48550/ARXIV.2407.21075. URL https://doi.org/10.48550/a...

    2024 doi

  11. [19]

    Pre-trained models: Past, present and future

    Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji - Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and J...

    2021

  12. [20]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020

    2020

  13. [21]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track ...

    2021

  14. [22]

    Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish...

    2010 arXiv

  15. [23]

    Trends in machine learning hardware, 2023

    Marius Hobbhahn, Lennart Heim, and Gökçe Aydos. Trends in machine learning hardware, 2023. URL https://epoch.ai/blog/trends-in-machine-learning-hardware. Accessed: 2024-12-05

    2023

  16. [24]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 doi arXiv

  17. [25]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. CoRR, abs/2404.06395, 2024. doi:10.48550/ARXIV.2404.06395. U...

    2024 doi arXiv

  18. [26]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  19. [27]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020. URL https://arxiv.org/abs/2001.08361

    2001 arXiv

  20. [28]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, p...

    2023

  21. [29]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing...

    2020

  22. [30]

    Textbooks are all you need ii: phi-1.5 technical report

    Yuanzhi Li, S \'e bastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023

    2023 arXiv

  23. [31]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  24. [32]

    Deja vu: Contextual sparsity for efficient llms at inference time

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning, pp.\ 22137--22176. P...

    2023

  25. [33]

    Llm-pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36: 0 21702--21720, 2023

    2023

  26. [34]

    Cramming more components onto integrated circuits

    Gordon E Moore. Cramming more components onto integrated circuits. Electronics, 1965

    1965

  27. [35]

    Compact language models via pruning and knowledge distillation

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. arXiv preprint arXiv:2407.14679, 2024

    2024 arXiv

  28. [36]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023

    2023 arXiv

  29. [37]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. Technical Report, 2024 a . URL https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

    2024

  30. [38]

    Gpt-4o mini: advancing cost-efficient intelligence

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence. Technical Report, 2024 b . URL https://openai.com/index/learning-to-reason-with-llms/

    2024

  31. [39]

    Opencompass: A universal evaluation platform for foundation models

    OpenCompass. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass, 2023

    2023

  32. [40]

    Pre-trained models for natural language processing: A survey

    Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. CoRR, abs/2003.08271, 2020

    2003 arXiv

  33. [41]

    Scaling language models: Methods, analysis & insights from training gopher

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021

    2021 arXiv

  34. [42]

    Beyond chinchilla-optimal: Accounting for inference in language model scaling laws

    Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 202...

    2024

  35. [43]

    Scaling LLM test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.03314, 2024. doi:10.48550/ARXIV.2408.03314. URL https://doi.org/10.48550/arXiv.2408.03314

    2024 doi arXiv

  36. [44]

    Powerinfer: Fast large language model serving with a consumer-grade gpu

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. Powerinfer: Fast large language model serving with a consumer-grade gpu. arXiv preprint arXiv:2312.12456, 2023

    2023 arXiv

  37. [45]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  38. [46]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational L...

    2023

  39. [47]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  40. [48]

    Introducing mpt-30b: Raising the bar for open-source foundation models, 2023

    MosaicML NLP Team. Introducing mpt-30b: Raising the bar for open-source foundation models, 2023. URL www.mosaicml.com/blog/mpt-30b. Accessed: 2023-06-22

    2023

  41. [49]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie - Anne Lachaux, Timoth \' e e Lacroix, Baptiste Rozi \` e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur \' e lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient fo...

    2023 arXiv

  42. [50]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  43. [51]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022 a

    2022

  44. [52]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022 b

    2022

  45. [53]

    Skywork: A more open bilingual foundation model

    Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei L \"u , Rui Hu, et al. Skywork: A more open bilingual foundation model. arXiv preprint arXiv:2310.19341, 2023

    2023 arXiv

  46. [54]

    A survey on knowledge distillation of large language models

    Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116, 2024

    2024 arXiv

  47. [55]

    Powerinfer-2: Fast large language model inference on a smartphone

    Zhenliang Xue, Yixin Song, Zeyu Mi, Le Chen, Yubin Xia, and Haibo Chen. Powerinfer-2: Fast large language model inference on a smartphone. CoRR, abs/2406.06282, 2024. doi:10.48550/ARXIV.2406.06282. URL https://doi.org/10.48550/arXiv.2406.06282

    2024 doi arXiv

  48. [56]

    Survey on knowledge distillation for large language models: Methods, evaluation, and application

    Chuanpeng Yang, Yao Zhu, Wang Lu, Yidong Wang, Qian Chen, Chenlong Gao, Bingjie Yan, and Yiqiang Chen. Survey on knowledge distillation for large language models: Methods, evaluation, and application. ACM Transactions on Intelligent Systems and Technology, 2024

    2024

  49. [57]

    Xing, and Hao Zhang

    Longfei Yun, Yonghao Zhuang, Yao Fu, Eric P. Xing, and Hao Zhang. Toward inference-optimal mixture-of-expert large language models. CoRR, abs/2404.02852, 2024. doi:10.48550/ARXIV.2404.02852. URL https://doi.org/10.48550/arXiv.2404.02852

    2024 doi arXiv

  50. [58]

    Tinyllama: An open-source small language model

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.