Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that replacing the parameter-count term in the Chinchilla scaling law with the average number of active parameters over pre-training predicts final evaluation loss for both dense and sparsely pre-trained LLMs.

desk verdict Average-parameter-count scaling law is a genuinely new idea with independent matched-pair support, but the Nbar definition is under-specified and the derivation only covers the pruning phase. read the letter →

arxiv 2501.12486 v2 pith:RFQ2B3AL submitted 2025-01-21 cs.LG cs.CL

Tian Jin , Ahmed Imtiaz Humayun , Utku Evci , Suvinay Subramanian , Amir Yazdanbakhsh , Dan Alistarh , Gintare Karolina Dziugaite This is my paper · ORCID
classification cs.LGcs.CL
keywords sparsepre-trainingscalinglawslargelanguagemodelsneuralnetworkpruningaverageparametercountChinchillalawcompute-optimaltrainingiterativemagnitude
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sparse pre-training and dense pre-training are governed by the same scaling law if the model-size variable is the average number of active parameters over training rather than the final parameter count or the sparsity level. The paper modifies the Chinchilla scaling law to $L(\bar{N},D)=A/\bar{N}^\alpha+B/D^\beta+E$ and fits this single form to dense and sparse runs spanning 3 model sizes, 5 sparsity levels, and 2 training durations, with an average absolute prediction error of 0.016 in evaluation loss. It also reports a simple near-optimal schedule for sparse pre-training: start iterative magnitude pruning at 25% of total training compute and finish at 75%. If the claim holds, sparse pre-training matches dense final quality for the same compute budget while producing a smaller final model, shifting the practical payoff of sparsity from training cost to inference cost.

What carries the argument

The load-bearing object is the average active parameter count, $\bar{N}=\frac{1}{T}\sum_{k=1}^{T}N_k$, where $N_k$ is the number of parameters that remain active and receive gradient updates during the $k$-th pruning iteration. Substituting $\bar{N}$ for $N$ in the Chinchilla law makes dense pre-training the sparsity-zero special case and lets one set of fitted constants predict both paradigms. The supporting mechanism is the three-phase sparse pre-training schedule (dense training, iterative magnitude pruning, sparse recovery); under the compute-proportionality assumption, each iteration contributes a compute increment proportional to $N_k$, and the Taylor expansion shows the total loss reduction is proportional to $\bar{N}$.

What would settle it

A concrete check: on hardware that genuinely skips masked computation, pre-train a 1B-parameter model at 80% sparsity and its dense counterpart matched on average parameter count and compute; if the sparse model loses more than about 0.016 in evaluation loss at the same FLOP budget, the central equivalence fails. A second check: two schedules with the same $\bar{N}$ but pruning concentrated early versus late should produce identical final loss, and any systematic gap would show the average is not the effective model-size variable.

Watch

Extended reading notes

Core claim

The central discovery is that a sparse pre-training run behaves, for loss prediction, like a dense model whose size is the sparse run's average active parameter count. Concretely, the paper proposes the law $L(\bar{N},D)=A/\bar{N}^\alpha+B/D^\beta+E$, which keeps the Chinchilla functional form and reduces to it when sparsity is zero. Matched sparse-dense pairs with equal average parameter count and equal compute reach nearly identical final evaluation loss, and the match holds in a 1.14B-parameter validation and on downstream benchmarks. The law is derived by Taylor-expanding the loss-vs-compute curve, assuming log loss decays linearly with log compute and compute per step is proportional to active parameters, and summing the incremental loss reductions across pruning iterations.

Load-bearing premise

The load-bearing premise is that the compute needed to process a fixed number of tokens is proportional to the number of active parameters; the paper's implementation masks pruned parameters rather than skipping their computation, so actual compute may be up to $1/(1-\text{sparsity})$ times the effective compute.

Editorial extensions

If this is right

  • - Dense scaling prescriptions carry over to sparse pre-training: choose learning rate and batch size as for an equivalent dense model and predict loss by substituting average parameter count into the Chinchilla form.
  • - Sparse pre-training improves the three-way trade-off among training compute, evaluation loss, and final model size: same quality and compute as dense, smaller final model, lower inference cost.
  • - Lifetime-compute optimization with the unified law recommends training on less data than dense prescriptions; at 80% sparsity and 100T inference tokens the paper estimates a 31.4% additional lifetime compute saving.
  • - Up to a rate equal to the ratio of average to final parameter count, sparse pre-training compresses losslessly during training; the paper reports a 2x compression at 80% final sparsity.
  • - Downstream task performance matches between sparse and dense models paired on average parameter count, not just perplexity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • - Beyond the paper's explicit claims, the law predicts that any schedule with the same average active parameter count—gradual growth, layer dropout, or stagewise expansion—should fall on the same loss curve, making $\bar{N}$ a general statistic for dynamically sized pre-training.
  • - The compute-equivalence argument presumes hardware that actually skips masked computation; on current implementations that mask rather than skip, the practical FLOP saving may not materialize even though the effective-compute law holds.
  • - A testable extension is to fit the same law to progressive-growth runs; if they land on the same curve, the mean is truly the schedule statistic, and if they do not, the direction of parameter change carries extra information.
  • - The law also implies that very different pruning distributions with the same mean should give the same final loss; an experiment that violates this would force a higher-order schedule statistic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a unified scaling law for dense and sparse pre-training of large language models by replacing the final parameter count N in the Chinchilla law with the average active parameter count Nbar over the full pre-training run. The authors define a three-phase sparse pre-training schedule (dense training, iterative pruning, sparse recovery), sweep over schedule and hyperparameter configurations to identify a near-optimal allocation (approximately 25% dense, 50% pruning, 25% recovery), and fit the proposed law to 30 runs spanning five sparsity levels, three model sizes, and two training durations, reporting a mean absolute prediction error of 0.016. The central hypothesis is additionally tested with paired experiments in which dense models with width adjusted to match each sparse model's average active parameter count achieve similar final loss, including a 1.14B-parameter validation in Appendix C and a downstream task comparison in Appendix D. The paper also presents a theoretical derivation based on compute-scaling assumptions, and an analysis of lifetime compute suggesting sparse pre-training can reduce data and inference costs.

Significance. If the claim holds, the paper valuably unifies sparse and dense scaling laws, predicts evaluation loss across sparsity levels and schedules, and offers a simple practical prescription for sparse pre-training. The strongest evidence is the matched-pair design in Figure 1 and Appendix C, which provides an out-of-sample check independent of the parametric fit, and the downstream task agreement in Appendix D. The systematic search over 80 configurations and the 1.14B-parameter paired validation are concrete strengths that go beyond typical small-scale pruning studies. However, the central variable Nbar is under-specified, the theoretical derivation is heuristic and internally questionable, and the fitted parameters of the proposed law are not reported. These issues are substantial but appear addressable, so the contribution is significant yet in need of revision.

major comments (4)
  1. [§5.1, §3, Eq. (2)] Section 5.1 defines Nbar as an unweighted mean over the T pruning iterations, whereas Section 3 describes a three-phase schedule (dense training, iterative pruning, sparse recovery) and Eq. (2) uses the total token count D of the full run. The paper's own text in Section 1 calls Nbar the 'mean of per-step parameter count over pre-training', which is a different, full-run quantity. If the experiments and matched-pair construction use the full-run step-weighted average, then the derivation in Eq. (5), which sums only over pruning iterations, does not support the fitted quantity; if instead the Section 5.1 definition is used, the law ignores the dense and recovery phases, which account for roughly half of the compute in the recommended (25%, 50%, 25%) schedules. The two averages can differ by about 10% for an 80%-sparsity run, and both the matched-pair evidence and the fitted law depend on which average is used. The authors must state unambiguously how Nbar is computed in every experiment and align the definition with the derivation.
  2. [§5.2, Eqs. (3)–(5), Figure 2] The theoretical derivation in Section 5.2 rests on Assumption 1, L(C) = (A/C)^α, which implies that the final loss is a function of total compute only. Under this assumption, the sum of Taylor increments in Eq. (4) approximates an integral of -α A^α C^{-α-1} dC, which evaluates to A^α(C_T^{-α} - C_0^{-α}) and is independent of the sequence N_k. The conclusion that the total loss change is proportional to the sum of N_k is therefore not justified unless the coefficient C_{0:k}^{-α-1} is constant. The paper asserts this coefficient is 'essentially constant' after about 100 pruning iterations, but for the reported values α=0.041 and α=0.203 and compute ranges spanning orders of magnitude, the coefficient varies by well over an order of magnitude, and Figure 2 (right) actually shows two different fitted α values for the same model family. The derivation should be corrected to handle the path-dependence explicitly, or presented more modestly as an empirical motivation rather than a theoretical validation.
  3. [§5.3, Appendix B] Section 5.3 reports the fit of Eq. (2) to 30 runs with a mean absolute error of 0.016, but the fitted values of A, B, E, α, and β are not reported anywhere, and no confidence intervals or across-seed variability are given. Without the fitted constants, the proposed scaling law cannot be used by practitioners, and the lifetime-compute recommendations in Appendix B cannot be reproduced or checked against the fit. The authors should report the fitted parameters, ideally with bootstrap uncertainties as in Hoffmann et al. (2024), and state whether the 30-point fit is validated on held-out configurations beyond the paired out-of-sample test in Appendix C, which tests the matched-pair hypothesis but not the parametric form of Eq. (2).
  4. [§7, Limitation] The paper's Limitation explicitly states that the implementation masks pruned parameters rather than skipping computation, and that actual compute may be up to 1/(1-sparsity) times the effective compute. Nevertheless, the abstract and Section 7 state that sparse pre-training 'achieves the same final model quality as dense pre-training for equivalent compute budgets' without the 'effective compute' qualifier. Since compute proportionality in Assumption 2 is central to both the derivation and the equivalence claim, the authors should either qualify all such claims as effective-compute statements, or clearly separate the conceptual effective-compute results from the practical actual-FLOPs situation.
minor comments (6)
  1. [Figure 2 (right)] The axis label '1e 17' appears broken; the label should be formatted as a multiplier or as 'C_{0:k}^{-α-1}'.
  2. [Abstract, §6] The abstract says '80 unique pruning schedules' while the sweep in Section 6 covers 9 schedules times 2 durations times 4 sparsity levels, which counts 72 schedule-duration-sparsity combinations; the counting should be clarified.
  3. [Appendix B] The recommendation 'Chinchilla: ... 70B model for 4.26T tokens' should be verified against Hoffmann et al. (2024) and Sardana et al. (2024); the compute-optimal Chinchilla allocation for 70B is typically around 1.4T tokens, so if this number comes from Sardana et al., it should say so.
  4. [§5.2, Eq. (5)] The notation C_{0:k} is used in Eq. (5) but only defined in the preceding sentence; it should be defined inline at the equation.
  5. [§5.3] The paper does not provide the standard deviation or range of Nbar across the 30 fitted runs; adding a small table of configurations, Nbar values, and observed losses would improve reproducibility and make the reported MAE of 0.016 interpretable.
  6. [§5.2, Further validation] The description of the transition point around 1e19 FLOPs at 'the first 2.6% of training steps' does not make clear which of the two fitted α values is used in the derivation of Eq. (5); clarifying this would avoid confusion about the claimed constancy of C^{-α-1}.

Circularity Check

2 steps flagged · score 4.0 of 10

Central matched-pair evidence is independent, but the fitted scaling law's 'predictions' and the theoretical validation are in-sample, so the paper overstates confirmation while the core claim is not definitionally forced.

  1. fitted input called prediction [Section 5.3, 'Fitting method' and 'Results' (Figure 3)]
    "We fit our proposed scaling law (Equation (2)) using the final evaluation loss obtained from sparse pre-training experiments with these optimal configurations. ... producing 30 data points. ... We present the predicted model evaluation loss and the actual final evaluation loss in Figure 3."

    The same 30 final-evaluation-loss values are used both to estimate the free parameters A, B, E, alpha, and beta in Equation (2) and then to compute the 'predicted' losses shown in Figure 3. No held-out split, cross-validation, or independent scaling-law fit is reported for this figure, so the agreement is an in-sample training fit rather than an out-of-sample prediction. The paper's separate matched sparse/dense pairs in Figure 1 and Appendix C do provide independent support for the average-parameter-count hypothesis, but the fitted constants and the reported 0.016 average absolute error are not independently validated by Figure 3.

  2. fitted input called prediction [Section 5.2, 'Further validation' and Figure 2]
    "Our analysis predicts that sparse pre-training, despite its varying parameter count, should maintain the same log-linear relationship between loss and training compute as dense pre-training. Figure 2 (left) shows the relationship between the training loss of sparsely pretrained 410M models and effective training compute consumption. The data confirms that the log-linear relationship largely holds. ... A linear fit on the loss data before and after this point estimates the scaling parameter alpha to be approximately 0.041 and 0.203, respectively."

    The 'prediction' that the log-linear relationship holds is confirmed by fitting the very same 410M loss curves from which alpha is then estimated; the data are not held out, so this is calibration to the observed curves rather than a parameter-free test. These same fitted alpha values are subsequently used to argue that the coefficient C_{0:k-1}^{-alpha-1} in Equation (5) is stable, so the theoretical justification and its 'Further validation' rest on the same in-sample loss data.

full rationale

No self-definitional circularity is present: the central proposal that average parameter count, rather than final parameter count or sparsity, is the effective model-size variable is tested by matched sparse/dense runs (Figure 1, Appendix C, and downstream tasks in Appendix D) whose construction is independent of the fitted scaling-law constants. Those matched pairs are genuine external checks and prevent the core claim from reducing to a fit by construction. The main circularity concern is limited to the scaling-law validation: Section 5.3 fits Equation (2) to all 30 runs and then presents the same runs as 'predicted vs actual' evidence, and Section 5.2 fits alpha from the 410M loss curves and then uses those curves to 'confirm' the predicted log-linear form. These are in-sample confirmations, not predictions. The paper's self-citations to Frantar et al. (2024) for the choice P=100 and for effective-compute scaling are not load-bearing for the average-parameter-count claim and are externally checkable. The acknowledged Limitation that actual compute may be up to 1/(1-sparsity) times effective compute weakens the practical-equivalence conclusion but is not a circularity. The noted ambiguity in the definition of Nbar (unweighted mean over pruning iterations versus token-weighted full-run average) is a correctness risk in the derivation, not a circular reduction, so it is flagged here rather than scored as a circular step.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

The central scaling law adds no new physical entities; it transplants five fitted constants from Chinchilla and reinterprets N. The genuinely new content is the claim that Nbar, not N or final sparsity, enters the law, supported by matched dense/sparse pairs. The paper's own derivation adds empirically validated assumptions about compute proportionality, coefficient stability, and prune-step loss spikes, none of which hold as theorems.

free parameters (6)
  • A (prefactor on Nbar term)
    Fitted via L-BFGS on the 30 sparse/dense runs in Section 5.3; value not reported in the text.
  • B (prefactor on data term)
    Fitted via L-BFGS on the 30 runs; value not reported in the text.
  • E (irreducible loss)
    Fitted via L-BFGS on the 30 runs; value not reported in the text.
  • alpha (exponent on Nbar)
    Fitted via L-BFGS on the 30 runs; value not reported in the text.
  • beta (exponent on D)
    Fitted via L-BFGS on the 30 runs; value not reported in the text.
  • Empirical alpha values from Figure 2 = 0.041 and 0.203
    Piecewise linear fit of loss versus effective compute for 410M models, used to argue that the coefficient C^{-alpha-1} is stable; these values come from the same data the derivation is meant to explain.
assumptions (7)
  • domain assumption The Chinchilla functional form L(N,D)=A/N^alpha+B/D^beta+E is the correct dense baseline.
    Borrowed from Hoffmann et al. 2024 and invoked in Sections 3 and 5.1 without re-derivation.
  • domain assumption Assumption 1: log loss decays linearly with log compute for fixed parameter count in the compute-optimal regime.
    Section 5.2, based on Kaplan et al. 2020; the experiments train 10x/20x past Chinchilla-optimal, so the regime assumption is questionable.
  • domain assumption Assumption 2: total compute for a fixed number of tokens is proportional to the number of active parameters.
    Section 5.2, used to set delta_C_k proportional to N_k and to define effective compute; contradicted by the masked implementation acknowledged in the Limitation.
  • ad hoc to paper Loss spikes at pruning steps do not materially affect final pre-training loss.
    Section 5.2, asserted from empirical observation; needed to sum Taylor loss decrements across pruning iterations.
  • ad hoc to paper The coefficient C_{0:k}^{-alpha-1} is approximately constant after early training.
    Section 5.2 and Figure 2 right; verified only for 410M runs with fitted alpha values, not a general proof.
  • ad hoc to paper Nbar as defined over pruning iterations equals the per-step average active parameter count over the full schedule, including dense and recovery phases.
    Section 5.1 defines Nbar as a mean over T pruning iterations, while experiments describe per-step averages over pre-training; the identity is asserted implicitly and not established.
  • standard math First-order Taylor expansion of the loss decrement is valid at each pruning iteration.
    Section 5.2 Eq. 4, derivative of A^alpha C^{-alpha} is elementary calculus, but truncation to first order is an approximation whose error is not quantified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws." pith.science (2026). https://pith.science/paper/RFQ2B3AL

@misc{pith2026250112486,
  author       = {Pith},
  title        = {Pith review of: The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RFQ2B3AL}},
  note         = {Machine review of arXiv:2501.12486}
}
read the original abstract

Pruning eliminates unnecessary parameters in neural networks; it offers a promising solution to the growing computational demands of large language models (LLMs). While many focus on post-training pruning, sparse pre-training--which combines pruning and pre-training into a single phase--provides a simpler alternative. In this work, we present the first systematic exploration of optimal sparse pre-training configurations for LLMs through an examination of 80 unique pruning schedules across different sparsity levels and training durations. We find that initiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss. These findings provide valuable insights for efficient and effective sparse pre-training of LLMs. Furthermore, we propose a new scaling law that modifies the Chinchilla scaling law to use the average parameter count over pre-training. Through empirical and theoretical validation, we demonstrate that this modified scaling law accurately models evaluation loss for both sparsely and densely pre-trained LLMs, unifying scaling laws across pre-training paradigms. Our findings indicate that while sparse pre-training achieves the same final model quality as dense pre-training for equivalent compute budgets, it provides substantial benefits through reduced model size, enabling significant potential computational savings during inference.

Figures

Figures reproduced from arXiv: 2501.12486 by the authors.

Figure 1
Figure 1. We show the predictive power of average active parameters by creating two families of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Left: Loss vs effective compute for 410M models for various sparsity levels and training durations. Right: Estimated α coefficient in the scaling law for 410M model. Assumption (2) has been heavily used in previous scaling law work (Kaplan et al., 2020; Sardana et al., 2024; Frantar et al., 2024), which model compute as proportional to the number of parameters, batch size, and number of iterations. Since we use a fi… view at source ↗
Figure 3
Figure 3. Across different model sizes and training durations, our fitted scaling law models the [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Optimal sparsity schedule sweep for 162M-10 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Batch size and learning rate sweep for 162M models. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: A closer look at failure modes for non-optimal sparsity schedules. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. QuEST: Stable Training of LLMs with 1-Bit Weights and Activations

    cs.LG 2025-02 conditional novelty 7.0 of 10

    A quantization-aware training method with Hadamard normalization and a trust gradient mask trains Llama models stably down to 1-bit weights and activations and makes 4-bit precision Pareto-optimal in accuracy per memory.

Reference graph

Works this paper leans on

47 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    Progressive gradient flow for robust n:m sparsity training in transformers

    Abhimanyu Rajeshkumar Bambhaniya, Amir Yazdanbakhsh, Suvinay Subramanian, Sheng-Chun Kao, Shivani Agrawal, Utku Evci, and Tushar Krishna. Progressive gradient flow for robust n:m sparsity training in transformers. arXiv preprint arXiv:2402.04744, 2024

    arXiv 2024

  2. [2]

    Scaling to very very large corpora for natural language disambiguation

    Michele Banko and Eric Brill. Scaling to very very large corpora for natural language disambiguation. In Annual Meeting of the Association for Computational Linguistics, 2001

    work page 2001

  3. [3]

    Data scaling laws in NMT : The effect of noise and architecture

    Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in NMT : The effect of noise and architecture. In International Conference on Machine Learning, 2022

    work page 2022

  4. [4]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373, 2023

    arXiv 2023

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    work page 2020

  6. [6]

    Maxtext: A framework for training large language models

    Matthew Davidow, Mohit Khatwani, and Michelle Yoo et al. Maxtext: A framework for training large language models. https://github.com/AI-Hypercomputer/maxtext, 2024

    work page 2024

  7. [7]

    Rigging the lottery: Making all tickets winners

    Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, 2020

    work page 2020

  8. [8]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019

    2019

Show all 47 references
  1. [9]

    Roy, and Michael Carbin

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, 2019

    2019

  2. [10]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774, 2023

    2023 arXiv

  3. [11]

    Scaling laws for sparsely-connected foundation models

    Elias Frantar, Carlos Riquelme Ruiz, Neil Houlsby, Dan Alistarh, and Utku Evci. Scaling laws for sparsely-connected foundation models. In International Conference on Learning Representations, 2024

    2024

  4. [12]

    The state of sparsity in deep neural networks

    Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574, 2019

    1902 arXiv

  5. [13]

    Scaling laws for neural machine translation

    Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. In International Conference on Learning Representations, 2022

    2022

  6. [14]

    A bit of progress in language modeling

    Joshua Goodman. A bit of progress in language modeling. arXiv preprint arXiv:cs.CL/0108005, 2001

    2001

  7. [15]

    Data and parameter scaling laws for neural machine translation

    Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Conference on Empirical Methods in Natural Language Processing, 2021

    2021

  8. [16]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  9. [17]

    OLM o: Accelerating the science of language models

    Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tusha...

    2024

  10. [18]

    Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Advances in Neural Information Processing Systems, 2015

    2015

  11. [19]

    Hassibi, D.G

    B. Hassibi, D.G. Stork, and G.J. Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks, 1993

    1993

  12. [20]

    Channel pruning for accelerating very deep neural networks

    Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In IEEE International Conference on Computer Vision, 2017

    2017

  13. [21]

    Rae, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2024

  14. [22]

    Roy, Jonathan Frankle, and Gintare Karolina Dziugaite

    Tian Jin, Michael Carbin, Daniel M. Roy, Jonathan Frankle, and Gintare Karolina Dziugaite. Pruning's effect on generalization through the lens of training and regularization. In Advances in Neural Information Processing Systems, 2022

    2022

  15. [23]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  16. [24]

    Accurate neural network pruning requires rethinking sparse optimization

    Denis Kuznedelev, Eldar Kurtic, Eugenia Iofinova, Elias Frantar, Alexandra Peste, and Dan Alistarh. Accurate neural network pruning requires rethinking sparse optimization. Transactions on Machine Learning Research, 2024

    2024

  17. [25]

    Optimal brain damage

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, 1989

    1989

  18. [26]

    Liu and Jorge Nocedal

    Dong C. Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical Programming, 45 0 (1): 0 503--528, 1989

    1989

  19. [27]

    Step: learning n:m structured sparsity masks from scratch with precondition

    Yucheng Lu, Shivani Agrawal, Suvinay Subramanian, Oleg Rybakov, Christopher De Sa, and Amir Yazdanbakhsh. Step: learning n:m structured sparsity masks from scratch with precondition. In International Conference on Machine Learning, 2023

    2023

  20. [28]

    Deep double descent: Where bigger models and more data hurt

    Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations, 2020

    2020

  21. [29]

    Efficient stagewise pretraining via progressive subnetworks

    Abhishek Panigrahi, Nikunj Saunshi, Kaifeng Lyu, Sobhan Miryoosefi, Sashank Reddi, Satyen Kale, and Sanjiv Kumar. Efficient stagewise pretraining via progressive subnetworks. arXiv preprint arXiv:2402.05913, 2024

    2024 arXiv

  22. [30]

    Larsen, Jonathan Frankle, Surya Ganguli, and Gintare Karolina Dziugaite

    Mansheej Paul, Feng Chen, Brett W. Larsen, Jonathan Frankle, Surya Ganguli, and Gintare Karolina Dziugaite. Unmasking the lottery ticket hypothesis: What's encoded in a winning ticket's mask? In International Conference on Learning Representations, 2023

    2023

  23. [31]

    AC / DC : Alternating compressed/decompressed training of deep neural networks

    Alexandra Peste, Eugenia Iofinova, Adrian Vladu, and Dan Alistarh. AC / DC : Alternating compressed/decompressed training of deep neural networks. In Advances in Neural Information Processing Systems, 2021

    2021

  24. [32]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 0 (1), 2020

    2020

  25. [33]

    Comparing rewinding and fine-tuning in neural network pruning

    Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020

    2020

  26. [34]

    On the predictability of pruning across scales

    Jonathan S Rosenfeld, Jonathan Frankle, Michael Carbin, and Nir Shavit. On the predictability of pruning across scales. In International Conference on Machine Learning, 2021

    2021

  27. [35]

    Creating sparse gpt-3 models with iterative pruning

    Anshul Samar. Creating sparse gpt-3 models with iterative pruning. https://cerebras.ai/blog/creating-sparse-gpt-3-models-with-iterative-pruning, 2022. Accessed: 2025-01-12

    2022

  28. [36]

    Beyond chinchilla-optimal: Accounting for inference in language model scaling laws

    Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. In International Conference on Machine Learning, 2024

    2024

  29. [37]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2020

    1909 arXiv

  30. [38]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In International Conference on Learning Representations, 2024

    2024

  31. [39]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  32. [40]

    Sheared llama: Accelerating language model pre-training via structured pruning

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694, 2023

    2023 arXiv

  33. [41]

    Step: Staged parameter-efficient pre-training for large language models

    Kazuki Yano, Takumi Ito, and Jun Suzuki. Step: Staged parameter-efficient pre-training for large language models. In Annual Meeting of the Association for Computational Linguistics (Student Research Workshop), 2024

    2024

  34. [42]

    Masked structural growth for 2x faster language model pre-training

    Yiqun Yao, Zheng Zhang, Jing Li, and Yequan Wang. Masked structural growth for 2x faster language model pre-training. In International Conference on Learning Representations, 2024

    2024

  35. [43]

    To prune, or not to prune: exploring the efficacy of pruning for model compression

    Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017

    2017 arXiv

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  37. [45]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  38. [46]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  39. [47]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.