{"id":"0e1f79cf-17d3-46c8-a0a1-49bb0665d31f","arxiv_id":"2412.04315","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Maximum LLM capability per parameter, measured on five benchmarks, has grown exponentially, doubling about every three months.","lead":"This paper defines 'capacity density' as how much performance a model delivers per parameter, and reports that the maximum density of open-source LLMs doubles roughly every three months. The metric and the empirical trend could help developers decide when to train or deploy smaller models that match larger ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Densing Law depends on inverting a loss-scaling law fit to models ≤0.8B to compute effective sizes of hundreds of billions of parameters, an extrapolation the paper never validates.","rationale":"The reader correctly identifies the two-step calibration as the soft spot, and the most damaging part is the N→loss curve (Eq. 2) being fit at ≤0.8B but inverted at sizes up to hundreds of billions. This is not a matter of disagreement with consensus; it is an internal gap between the evidence shown and the regime in which the fitted function is used. The authors had larger checkpoints available for the f2 step but never report whether Eq. 2 predicts their losses, so the check is inexpensive and directly decisive. If the extrapolation fails, every density value changes, and because the bias scales with model strength, the fitted exponential envelope in Fig. 1 and the headline doubling time are both compromised. I therefore agree with the REJECT verdict: the Densing Law is plausible but not independently established. My concrete test would let the authors (or an independent group) settle the question without new large-scale training.","tokens_in":16400,"tokens_out":9479,"duration_ms":94533,"concrete_test":"Plot the measured conditional losses of the larger MiniCPM-3 checkpoints (0.5B to tens of billions, already used for the performance-estimation step in §3.2) against the predictions of Eq. 2 using their known N and D, with parameters fit only on the six small reference models from Table 1. If these mid/large points deviate by more than the small-model fit residuals, the inverse mapping of Eq. 4 is invalid. Then refit Eq. 2 including all checkpoint losses and recompute the density values and slope A in Eq. 5; report whether the doubling time (≈95 days) shifts by more than 30%. If it does, the Densing Law is an artifact of the extrapolation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation 2 is fitted on six reference models from Table 1, with N from 0.005B to 0.8B and D from 10×N to 60×N (maximum D ≈ 48B). Equation 4 inverts this curve using D0=1T to assign every evaluated model an effective parameter size N_hat. For high-performing models on the 2024 envelope (Phi-2, MiniCPM-3-4B, Gemma-2-9B), N_hat is in the tens of billions; for Llama-3.1-405B it is hundreds of billions. This is a 12×–250× extrapolation in N and a 20× extrapolation in D beyond the fitting range. The paper's only large-model validation is in the performance-estimation step (f2), which fits loss→performance using MiniCPM-3 checkpoints up to tens of billions; it does not test whether Eq. 2 predicts those checkpoints' losses from their N and D. If the true loss-scaling surface deviates from the additive power law outside the fitted range—e.g., because data quality or training recipes change the effective exponent—then N_hat is biased, and the bias grows as L_hat decreases (i.e., as models improve over time). Since the Densing Law's slope A is fit to the envelope of these density values, a time-growing bias in N_hat would produce a spurious exponential trend. Thus the central claim is not supported without evidence that Eq. 2 extrapolates to the regime in which it is inverted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a metric, capability density, defined as the ratio between a model's effective parameter size and its actual parameter size. The effective parameter size is computed by inverting a two-step scaling law: Eq. (2) predicts conditional language-modeling loss from parameter count N and training tokens D, fitted on six small reference models (0.005B–0.8B parameters) trained on the MiniCPM-3 corpus; Eq. (3) maps loss to downstream benchmark performance using MiniCPM-3 models and checkpoints. Applying this calibration to 29 open-source base LLMs, the paper reports that the maximum capability density follows ln(ρmax) = 0.0073 t + B (Eq. 5), corresponding to a doubling every ~95 days, and draws several corollaries about inference cost, Moore's law, and training efficiency.","tokens_in":16641,"tokens_out":9885,"duration_ms":94927,"significance":"The concept of a unified effectiveness–efficiency metric is useful and timely, and the paper is clearly structured. If the Densing Law could be validated with a calibration that demonstrably extrapolates across scales and model families, the result would be practically important for planning model training and deployment. The authors deserve credit for framing the density idea, for using public benchmarks and models, and for explicitly discussing benchmark contamination and other limitations. However, the quantitative law as stated is not yet supported by the evidence in the manuscript: the central claim rests on an unvalidated extrapolation of the loss-scaling law and on a small envelope fit, as detailed below.","major_comments":[{"comment":"The scaling law in Eq. (2) is fitted on reference models with parameter counts at most 0.8B and training tokens at most 60×N ≈ 48B, yet Eq. (4) inverts this law at D0 = 1T and assigns effective parameter sizes of tens to hundreds of billions to models on the 2024 envelope. This is a 20× extrapolation in D and a 250× extrapolation in N beyond the fitted range, and the paper gives no check that the additive power-law surface holds there. Since N_hat enters the numerator of every density value, any uncontrolled deviation in the loss surface biases the density values and, because the bias grows as measured losses decrease over time, can create a spurious exponential trend in Eq. (5). A minimally necessary addition is to validate Eq. (2) against the conditional losses of the MiniCPM-3 large checkpoints already used to fit Eq. (3), or against other large open models.","section":"§2.2 and §2.4, Eq. (2) and Eq. (4), Table 1"},{"comment":"The loss-to-performance curve is fit exclusively on MiniCPM-3 models and checkpoints, and the same family supplies some of the highest-density points that determine the envelope (e.g., MiniCPM-1-2.4B and MiniCPM-3-4B in Figure 1). The mapping from loss to benchmark performance is treated as universal across architectures and pretraining-data distributions, but no held-out validation across families is provided. A concrete test would be to fit Eq. (3) on all evaluated models except one family (e.g., Llama-3) and check whether predicted performance for the held-out family is unbiased; without such a test, the density of non-MiniCPM models is potentially miscalibrated, and the envelope is partly constructed from the same family that defines the ruler.","section":"§3.2, Eq. (3), Figure 2(b)"},{"comment":"The density calculations mix benchmark scores taken from each model's technical report, which differ in evaluation harnesses, prompt templates, few-shot exemplars, and contamination-cleaning procedures. Because N_hat is a steeply nonlinear function of S (the sigmoid in Eq. (3) is near its midpoint for many models), small protocol differences can change density by a large factor. The paper itself acknowledges contamination issues in Section 5, but the values in Figure 1 are still computed from heterogeneous reported scores. To support a quantitative claim of a 3.3-month doubling time, the authors should recompute all scores in a single harness (for example, OpenCompass, which they already use) for all 29 models, and state which version of each benchmark was used.","section":"§3.1, Figure 1"},{"comment":"The Densing Law is fit to the upper envelope of density versus time, with roughly one envelope point per several months in Figure 1 and no explicit list of which models are used as envelope points. The paper reports R2=0.912 but not the number of points, the standard error of A, or a sensitivity analysis. Removing a single envelope model could plausibly change the doubling time by months. Please report the envelope points, the uncertainty in A, a leave-one-out analysis, and a comparison against a null model with release dates permuted.","section":"§3.3, Eq. (5)"}],"minor_comments":[{"comment":"The comparison of slopes before and after ChatGPT (0.0048 vs. 0.0073) is not accompanied by a statistical test, and the two regressions use different time spans and model subsets; the claim that growth accelerated by 50% should either be tested or softened.","section":"§3.4, Corollary 3"},{"comment":"The fitted parameters of Eqs. (2) and (3) are not reported; without them, the density values are not reproducible. Please include a table with a, α, b, β for Eq. (2) and c, γ, l, d for Eq. (3) for each benchmark.","section":"§2.2 and §3.2"},{"comment":"The manuscript uses both \"capability density\" and \"capacity density\" for the same concept; please choose one term and use it consistently throughout.","section":"Abstract and §1"},{"comment":"The text cites OpenAI (2023) for GPT-4o, but that reference is the GPT-4 technical report; please correct the citation or use the appropriate GPT-4o technical report.","section":"§2.2, Reference [OpenAI, 2023]"},{"comment":"Figure 2(a) is drawn as a function of compute 6ND, while Eq. (2) is a function of N and D separately; the paper should explain how the plotted curves follow from the fitted function, since the same loss can correspond to different (N,D) combinations.","section":"Figure 2(a)"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical-law claim built on a custom metric, so the validation burden is high. The central issues are the unvalidated extrapolation of Eq. (2) and the family-specific calibration of Eq. (3); both are potentially fixable because the authors already have the MiniCPM-3 checkpoints and a unified evaluation tool. If the requested validations cannot be produced, the Densing Law should not be published as stated. I therefore view major revision as the appropriate path, rather than outright rejection, but the onus is on the authors to turn the metric from a family-relative artifact into a robust measurement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is good. Capacity density – effective parameter size divided by actual parameter size – is a genuinely useful way to talk about both effectiveness and efficiency at once, and the two-step fitting procedure is clearly laid out. The paper is readable and honest about some limitations, and the qualitative claim that models are getting denser over time is almost certainly true, consistent with the visible collapse in API prices for fixed quality.\n\nThe problem is the load-bearing extrapolation. Equation 2 is fit on reference models with at most 0.8B parameters and at most 48B training tokens. Then it is inverted at D0 = 1T to assign effective parameter sizes to hundreds of billions of parameters – a 12x to 250x extrapolation in N and a 20x extrapolation in D that is never validated. The paper does validate the loss-to-performance curve (f2) on larger MiniCPM checkpoints, but it never checks whether the loss-scaling curve (f1) predicts those checkpoints' losses from their N and D. If the additive power law bends outside the fitted range – because data quality, compute allocation, or training recipes change the effective exponents – then every density value is biased, and the bias grows as models improve. That can manufacture exactly the exponential envelope trend the paper reports. This is not a minor quibble; it is the difference between a supported law and a fitted artifact.\n\nThe MiniCPM circularity makes it worse. The reference models are trained on the MiniCPM corpus, the performance curve uses MiniCPM checkpoints, and MiniCPM-3-4B is one of the highest-density points on the envelope. The ruler is built from the same family being measured, so part of the trend is self-fulfilling. Scores taken from technical reports, no error bars on the slope, and no released code or data add to the fragility. The qualitative trend survives these concerns, but the specific doubling time of 3.3 months does not.\n\nWho should read this? Anyone working on model efficiency, scaling laws, or deployment cost forecasting. The metric deserves discussion, and the paper's corollaries are provocative. It is not ready as is, but it is worth a serious referee: the idea is novel, the presentation is clear, and the flaws are fixable. If the authors re-fit on an external reference family, validate the loss extrapolation against at least one large publicly available model, re-evaluate all models under one protocol, and give uncertainties, the result could be a solid empirical paper.\n\nMy recommendation: send it to peer review, but expect a major revision before it becomes publishable. Do not desk-reject; the central question is important enough to spend referee time on.","headline":"A useful new metric and a plausible-sounding trend, but the doubling time rests on an unvalidated 250x extrapolation of a scaling law fitted to 0.8B-parameter models, so the paper needs major revision before its central claim can be trusted.","tokens_in":17281,"tokens_out":2714,"would_cite":false,"duration_ms":29126,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The maximum capability density of open-source LLMs grows exponentially, doubling about every 3.3 months.","keywords":["capability density","Densing law","effective parameter size","large language models","LLM scaling laws","model efficiency","benchmark evaluation","exponential growth"],"falsifier":"Compute the effective parameter size of a held-out large model from a different family (for example, a 100B-plus model trained on non-reference data) using its measured scores on the five benchmarks, and compare it with the performance predicted by the calibrated curves. If the error is much larger than the residual scatter of the fits in the paper's Figure 2, or if refitting the two curves without the reference checkpoints changes the fitted slope $A$ enough to push the doubling time outside the two-to-six-month range, the universal calibration and the 3.3-month doubling time fail.","tokens_in":16103,"feed_emoji":"📈","tokens_out":12994,"duration_ms":111744,"temperature":0.7,"pith_summary":"This paper tries to establish that LLM progress can be quantified not only by what a model can do but by what it does per parameter, and that this quantity follows a predictable exponential trend. It introduces capability density, defined as the ratio of a model's effective parameter size to its actual parameter size, where the effective size is the parameter count a reference LLM would need to match the target's benchmark scores. Using 29 open-source base models and five common benchmarks, it reports that the maximum density has grown as $\\ln \\rho_{\\max} = A t + B$ with $A \\approx 0.007$, a doubling roughly every 3.3 months. On this account, a model with about half the parameters of today's best will match its performance in about one quarter, turning model efficiency into a forecastable quantity.","feed_headline":"LLM capability density doubles every 3.3 months","feed_subtitle":"A new metric turns benchmark scores into effective parameter counts, making model efficiency predictable.","key_machinery":"The load-bearing object is the capability-density metric $\\rho(M) = \\hat{N}(S_M)/N_M$ and the calibration chain behind it. Effective parameter size $\\hat{N}(S_M)$ is the inverse of a two-step scaling function: first a power law maps parameter count $N$ and training tokens $D$ to conditional language-modeling loss on test items, then a sigmoid maps that loss to downstream benchmark performance. The chain converts any model's public benchmark scores into a parameter-equivalent size, making models of different scales comparable on a single efficiency axis; the envelope fit over release dates of this ratio is what produces the Densing Law.","core_discovery":"The paper's central claim is that there exists a measurable quantity, capability density, that has been increasing exponentially for open-source base LLMs. Density is defined as $\\rho(M) = \\hat{N}(S_M)/N_M$, where $N_M$ is the model's actual parameter count and $\\hat{N}(S_M)$ is the effective parameter size: the number of parameters a reference model trained on $D_0 = 10^{12}$ tokens would need to reach the same downstream score $S_M$. The effective size is obtained by composing two fitted maps, a power-law loss curve $L = aN^{-\\alpha} + bD^{-\\beta}$ and a sigmoid performance curve $S = c/(1+e^{-\\gamma(L-l)}) + d$, calibrated on small reference models and intermediate checkpoints. On MMLU, BBH, MATH, HumanEval, and MBPP, the maximum density of released models fits $\\ln \\rho_{\\max} = A t + B$ with $A \\approx 0.007$ (about 0.0073 in the detailed fit), i.e. a doubling time of about 95 days.","pith_inferences":["An extension the paper does not make: if the doubling time persists, a multi-month training run is a depreciating asset, and training decisions should discount future density gains.","A neighbouring application: re-base density on inference FLOPs rather than parameters, which would let the law be tested against reasoning models that trade extra test-time compute for accuracy.","A measurement caution: as models saturate the five current benchmarks, measured density growth will flatten even if real capability growth continues, so keeping the law testable requires continually harder evaluation sets.","An application to closed models: applying the same effective-parameter mapping to public benchmark scores would turn density into a price-performance index for API offerings."],"forward_implications":["Inference cost for a fixed performance level falls exponentially, with the paper's API price data showing a halving roughly every 2.6 months.","Density growth compounded with chip-density growth means the largest effective model that fits on a given chip grows far faster than hardware improvements alone would suggest.","After ChatGPT's release the fitted density-growth slope rose from about 0.0048 to 0.0073 per day, a 50 percent acceleration.","Existing pruning and distillation outputs mostly show lower density than their source models, so compression is not automatically a density improvement.","Model development should target density rather than raw performance, because an over-large model's capability may be matched by a smaller model released months later at a fraction of the cost."],"supporting_citations":[{"why":"Supplies the power-law scaling-law form used in the loss-estimation step.","marker":"Kaplan et al., 2020"},{"why":"Establishes the loss-vs-parameters-and-data relationship that Eq. 2 extends to conditional downstream loss.","marker":"Hoffmann et al., 2022"},{"why":"Provides the training corpus and model family used to build the reference scaling models and checkpoints.","marker":"Hu et al., 2024"},{"why":"Inspires the two-step loss-to-performance estimation and contributes the Llama-3 models to the density trend.","marker":"Dubey et al., 2024"},{"why":"MMLU, one of the five benchmarks that define downstream performance.","marker":"Hendrycks et al., 2020"},{"why":"BBH, one of the five benchmarks that define downstream performance.","marker":"Suzgun et al., 2023"},{"why":"MATH, one of the five benchmarks that define downstream performance.","marker":"Hendrycks et al., 2021"},{"why":"HumanEval, one of the five benchmarks that define downstream performance.","marker":"Chen et al., 2021"},{"why":"MBPP, one of the five benchmarks that define downstream performance.","marker":"Austin et al., 2021"},{"why":"Supplies Llama-3.1-minitron-4B, the main compressed-model comparison in the compression corollary.","marker":"Muralidharan et al., 2024"}],"fun_headline_variants":["LLM capacity density doubles every 3 months","Densing law: LLM efficiency doubles every 95 days","New metric shows LLM density doubles quarterly","LLM densing: capability per parameter doubles ~3 months","Capacity density: LLMs double in efficiency every 3 months"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole measurement stands on the assumption that the loss-to-performance and loss-to-parameters curves fitted on small reference models and one family's checkpoints apply universally to every model family and every scale, including models hundreds of times larger.","fun_headline_variants_meta":{"raw":{"variants":["LLM capacity density doubles every 3 months","Densing law: LLM efficiency doubles every 95 days","New metric shows LLM density doubles quarterly","LLM densing: capability per parameter doubles ~3 months","Capacity density: LLMs double in efficiency every 3 months"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001033,"raw_usage":{"total_tokens":4393,"prompt_tokens":1032,"completion_tokens":3361,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":3282}},"tokens_in":648,"tokens_out":3361,"duration_ms":22080,"temperature":1.0,"reasoning_tokens":3282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:32:09.793829+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the effective parameter size of a held-out large model from a different family (for example, a 100B-plus model trained on non-reference data) using its measured scores on the five benchmarks, and compare it with the performance predicted by the calibrated curves. If the error is much larger than the residual scatter of the fits in the paper's Figure 2, or if refitting the two curves without the reference checkpoints changes the fitted slope $A$ enough to push the doubling time outside the two-to-six-month range, the universal calibration and the 3.3-month doubling time fail.","supporting_citations":[],"review_version":1}