Pith. sign in

REVIEW 4 major objections 5 minor 42 references

AutoPrune claims a language model, guided by a structured reasoning graph, can design its own pruning rule that beats human-designed heuristics like Wanda and SparseGPT on pruned LLaMA models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 21:24 UTC pith:BP3QDX6Y

load-bearing objection The 'LLM designs its own pruning rule' idea is worth a look, but the paper's central claim is contradicted by its own tables, especially the OPT-13B result. the 4 major comments →

arxiv 2511.15390 v2 pith:BP3QDX6Y submitted 2025-11-19 cs.CV

Automatic Pruning Discovery for Large Language Models

Haidong Kang , Lihong Lin , Enneng Yang , Hongning Dai , Hao Wang This is my paper
classification cs.CV
keywords LLM pruningtraining-free pruningautomated algorithm discoverygraph chain-of-thoughtlayerwise sparsity allocationskewnessoutlier weightsAutoPrune
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

AutoPrune tries to show that an LLM, guided by a structured reasoning graph, can design its own pruning algorithm—one that predicts which weights to remove more accurately than human-crafted rules such as Wanda and SparseGPT. The discovered rule normalizes each weight by the L1 norm of its row and combines the L1 and squared L2 norms of its activation column. The paper also identifies an 'outlier value issue': layers whose weight magnitudes are strongly positively skewed are the most sensitive to pruning, and uniform sparsity is the root cause. To address this, it introduces SDSA, a skewness-based layerwise sparsity allocator that protects such layers. If the claims hold, pruning LLMs no longer requires expert-derived heuristics, and high-ratio pruning can be made much safer.

Core claim

On the paper's own terms, the central discovery is that an LLM, prompted through a four-stage graph-structured chain of thought (analysis, hypothesis, conceptual formula, computable concept), can search over candidate pruning rules and select one by minimizing perplexity on WikiText-2. The selected rule is importance = |W_ij| / ||W_i:||_1 x sqrt(||X_:j||_1 + ||X_:j||_2^2). In experiments, this training-free rule beats Wanda, SparseGPT, and magnitude pruning across LLaMA-1 and LLaMA-2 at 50%, 60%, 2:4 and 4:8 sparsity, and the skewness-based SDSA allocation improves all methods, including Wanda, at high sparsity.

What carries the argument

Two mechanisms carry the argument. Graph-driven Chain-of-Thought (GCoT) generalizes linear prompting into a directed acyclic reasoning graph: branches are sampled at each stage, every root-to-leaf path yields a complete pruning algorithm, and candidate algorithms are ranked by WikiText-2 perplexity. Skew-aware Dynamic Sparsity Allocation (SDSA) computes the biased sample skewness of each layer's absolute weight distribution, normalizes it, and converts it into softmax retention weights whose contrast grows with global sparsity, so highly skewed outlier-bearing layers are pruned less.

Load-bearing premise

The paper assumes that picking the pruning rule that minimizes WikiText-2 perplexity on the same model family used in the final evaluation yields a rule that generalizes, rather than one that is simply fitted to that particular calibration set.

What would settle it

Find a model family and sparsity level where AutoPrune's discovered rule is evaluated without any search on that family, and compare it to Wanda's rule on identical calibration data; if it does not beat Wanda there, the claimed automatic advantage is not established. Concretely, run AutoPrune's search on LLaMA-2-7B and then evaluate the resulting rule on a held-out family such as Mistral or Phi, using fresh calibration samples, and check whether perplexity still beats Wanda.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Pruning rules for LLMs can be discovered automatically by the model itself, eliminating the manual trial-and-error used to design Wanda and SparseGPT.
  • The discovered metric improves perplexity under 50%, 60%, 2:4, and 4:8 sparsity across LLaMA-1 and LLaMA-2, including large 65B/70B models.
  • SDSA reduces perplexity on all tested pruning methods at 60% sparsity, indicating that uniform sparsity is a fixable source of high-ratio degradation.
  • The GCoT reasoning pipeline outperforms both naive single-shot prompting and linear step-by-step CoT, showing that branching search is what enables the improvement.
  • AutoPrune generalizes to unseen model families, matching or beating Wanda and SparseGPT on Pythia-12B and remaining competitive on OPT-13B, without weight updates.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the search objective and final evaluation were fully decoupled (selecting on one data distribution and evaluating on a disjoint family), the superiority over Wanda might shrink; the paper's protocol does not rule out overfitting to WikiText-2.
  • The same GCoT pipeline could be applied to other model-compression design tasks—quantization schemes, attention sparsity, MoE routing—where there is a cheap scoring objective.
  • A testable extension: run AutoPrune's search on a small model and apply the discovered rule to a much larger, different-family model; if the rule still wins, it is a stable property of transformer pruning rather than a search artifact.
  • The skewness prior suggests a concrete diagnostic: models with more layers of high positive skew should benefit more from SDSA; measuring that correlation would clarify when the allocator is necessary.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AutoPrune, an LLM-driven framework that uses a Graph-driven Chain-of-Thought (GCoT) procedure to automatically discover pruning importance formulas, combined with a Skew-aware Dynamic Sparsity Allocation (SDSA) module that adjusts per-layer sparsity based on weight-distribution skewness. The authors claim that AutoPrune consistently outperforms existing training-free pruning methods such as Wanda and SparseGPT across model scales and sparsity settings, and that SDSA mitigates the outlier-value issue under high pruning ratios. Experiments are reported on LLaMA-1/2, OPT, and Pythia models using WikiText perplexity, zero-shot accuracy, MMLU, and an active-reasoning benchmark.

Significance. If validated, the idea of using LLMs to search for pruning algorithms in a structured, interpretable way could reduce human design effort and offer a new paradigm for model compression. The paper also ships a public code release and provides an unusually transparent view of the discovered importance formula (Table 2), which is a genuine strength. However, the significance is heavily compromised by the fact that the paper's own numerical tables contradict the central claim of consistent superiority, and the algorithm-selection protocol is circular with respect to the final evaluation metric. The claimed contribution therefore is not established by the presented evidence.

major comments (4)
  1. [§4.1, Table 4] The central claim that AutoPrune 'consistently outperforms' Wanda and SparseGPT is contradicted by the paper's own Table 4. At 50% unstructured sparsity on LLaMA-1 65B, AutoPrune reports PPL 4.71, while both Wanda and SparseGPT report 4.57. On LLaMA-2 70B at 50%, AutoPrune reports 4.00 versus 3.98 for Wanda and SparseGPT. Additionally, the text states that AutoPrune achieves a 'notable 1.30 decrease' on LLaMA-2 13B, but the table shows 5.56 to 5.43, a decrease of 0.13. These are not marginal presentation errors; they directly invalidate the abstract's and §4.1's 'consistently excels' assertion.
  2. [App. C.2, Table 13] The held-out architecture experiment contradicts the generalization claim. On OPT-13B at 50% unstructured sparsity, AutoPrune gives WikiText PPL 20.33, whereas Wanda gives 11.42 and SparseGPT 11.19—a roughly 78% relative degradation. The accompanying text claims AutoPrune 'demonstrates stable performance and remains competitive across the board,' which is falsified by the same table. This result is particularly damaging because OPT is exactly the kind of unseen architecture the generalization claim needs to cover, and it shows the discovered importance formula does not transfer.
  3. [§3.1, Eqs. (2)–(3); Table 4] The algorithm selection procedure is circular with respect to the main evaluation. Eq. (2) scores each candidate algorithm by perplexity on wikitext-2, and Eq. (3) selects the candidate with minimum perplexity. Table 4 then reports perplexity on the WikiText dataset family—the same objective used for selection. The GCoT search is also run on LLaMA models, and the SDSA hyperparameter M is tuned on LLaMA-1/2 models in App. D.2. Consequently, the comparison against Wanda and SparseGPT on WikiText PPL is not an independent test of the discovered rule's generality; it is a benchmark-fitting result. A held-out protocol that separates algorithm search from final evaluation is required to support the claimed cross-model superiority.
  4. [§3.2, Eqs. (7)–(8)] The SDSA formulation is underspecified. Eq. (7) defines a softmax over layers of normalized skewness scores and calls ωℓ the 'retention fraction,' but the paper never specifies how ωℓ is converted into per-layer sparsity ratios (e.g., a normalization to satisfy the global sparsity budget). Equation (8) then takes a ratio of softmax outputs and writes it as exp(β Δγ~), which ignores the softmax denominator except in special cases. The definition of Δγ~ in Eq. (8) also conflicts with the earlier Δγ in Eq. (5). This makes the SDSA contribution difficult to reproduce or verify.
minor comments (5)
  1. [Eq. (1)] The text says 'k is the XXX'—the number of branches is never defined. This is a missing technical detail.
  2. [Table 4] The Magnitude 60% entries contain values like '6e2', '2e2', and '4e3', which appear to be formatting errors or placeholder numbers. They should be corrected to the actual perplexity values.
  3. [App. C.2, Table 13] The table header says 'Pythia-13B' while the text says 'Pythia-12B'. This inconsistency should be fixed.
  4. [§4.1] The claimed '1.30 decrease' on LLaMA-2 13B appears to be a typo for 0.13; please correct and re-check the associated narrative.
  5. [App. C.2 text] The phrase 'remains competitive across the board' is contradicted by the OPT-13B results; at minimum the text should be revised to acknowledge the failure, not assert competitiveness.

Circularity Check

2 steps flagged

Selection objective (WikiText-2 PPL, Eqs. 2-3) is identical to the main reported evaluation metric; the headline WikiText perplexity results are the fitted quantity rather than an independent prediction.

specific steps
  1. fitted input called prediction [§3.1 (Eqs. 2-3) and §4.1 (Table 4)]
    "Each candidate algorithm induces an importance score function I(p)(W), which we evaluate by pruning the target model to sparsity α and computing perplexity on wikitext-2: E = PPL(Prune(W, I(p), α)). We then select the best algorithm by: p* = arg min p E(p). ... Table 4 reports the perplexity of our method (AutoPrune) and several strong baselines on pruned LLaMA-1 and LLaMA-2 models evaluated on the WikiText dataset. ... our method consistently outperforms existing training-free pruning approaches across various sparsity settings and model scales."

    The rule p* is chosen to minimize WikiText-2 perplexity on the LLaMA family, and Table 4's headline evidence of 'consistently outperforms' is perplexity on the same WikiText benchmark and the same model family. The reported superiority is therefore the value of the optimized objective, not a held-out prediction of generalization. The paper does not separate algorithm search (Eq. 3) from final evaluation (Table 4) with a fresh benchmark or a disjoint model family; the central WikiText result reduces to the search objective by construction.

  2. fitted input called prediction [App. D.2 / Table 16 and App. B (hyperparameter M)]
    "To evaluate the robustness of our method, we conduct a sensitivity analysis on the hyperparameter M. For this study, we apply 70% unstructured pruning on both the LLaMA-7B and LLaMA-2-7B models while varying the value of M. ... For the Skew-Aware Dynamic Sparsity scheduler, the hyperparameter M is set to 1.8 for layerwise allocation."

    The contrast bound M in SDSA is selected by sweeping on LLaMA-7B and LLaMA-2-7B, and the selected value is then used in the main perplexity tables on those same model families. This is tuning on the evaluation target; the reported gains absorb the benefit of having chosen M on the exact models/datasets used for the headline comparison, further weakening the claim that the results are an independent prediction.

full rationale

The pipeline is not citation-circular: there are no load-bearing self-citations, and GCoT is a genuine search over LLM-generated formulas. The circularity is in the evaluation protocol. Eqs. 2-3 make WikiText-2 perplexity the objective of algorithm selection, and Table 4 then presents WikiText perplexity as evidence that AutoPrune 'consistently outperforms' baselines. On the search model and benchmark, the reported value is the optimized quantity, so the main superiority claim over Wanda/SparseGPT on that table is not an out-of-sample confirmation. The M=1.8 hyperparameter is likewise chosen on the same model variants that appear in the main tables. There is independent content: zero-shot accuracy, MMLU, and the OPT/Pythia generalization test are not the selection objective, so the framework is not vacuous. However, that independent evidence does not rescue the headline claim — App. C.2/Table 13 shows AutoPrune at 50% sparsity on OPT-13B with WikiText PPL 20.33 versus Wanda 11.42, directly contradicting 'consistently outperforms' and 'stable performance.' This is a correctness/falsification concern rather than an additional circular step, but it confirms that the consistent gains are confined to the benchmark used for selection. Overall, the central WikiText result reduces by construction to the search objective, warranting a partial-circularity score of 6 rather than a higher score, because the method has some genuinely non-fitted evaluations.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The method is a search-and-fit procedure over pruning heuristics. Its main contribution rests on assumptions about LLM prompt-driven creativity, WikiText-2 PPL as a transferable selection signal, and skewness as a sensitivity prior. The only explicit fitted constants are M and the linear beta schedule; GCoT temperatures and branch count are unreported free choices.

free parameters (4)
  • SDSA contrast cap M = 1.8 layerwise, 1.5 blockwise
    Chosen from a sensitivity sweep on LLaMA-1/2-7B at 70% sparsity (Table 16). PPL varies strongly with M (79.94 at M=1.8 on LLaMA-7B vs 99.30 at M=1.2; the best value for LLaMA-V2-7B is M=2.2), yet M=1.8 is fixed globally.
  • GCoT branch count k = not reported
    Eq. (1) contains the placeholder 'k is the XXX'. Appendix D.3 shows PPL varies with branch count (7.49 with 1 branch vs 7.09 with 10 on LLaMA-7B), but the deployed k is not stated.
  • Sampling temperatures {T_i} in GCoT branching = not reported
    Expand(s) = {LLM(s; T_i)}_{i=1}^k requires temperature choices that affect search diversity; no values are given.
  • beta(S_g) linear schedule = beta = S_g * ln M / (Delta_gamma + epsilon)
    An ad hoc schedule modulating skewness contrast by global sparsity; no derivation is provided, and it depends on the tuned M.
axioms (5)
  • domain assumption LLM latent knowledge, when prompted through GCoT, can synthesize novel and effective pruning importance metrics.
    Section 3 states LLMs 'possess vast, latent knowledge of network principles' and can automate pruning design; this capacity is assumed, not proven.
  • domain assumption Positive skewness of per-layer absolute weight magnitudes marks pruning-sensitive layers.
    Section 2.1-2.2 provides correlational evidence on LLaMA-1-7B (Figs. 3-4) but treats skewness as a causal sensitivity prior for SDSA.
  • domain assumption Uniform sparsity is the root cause of the outlier-value performance drop.
    Section 2.2 concludes this from sensitivity experiments and uses it to justify the whole SDSA design.
  • domain assumption WikiText-2 perplexity is a sufficient objective for selecting a transferable pruning algorithm.
    Eq. (2)-(3) select p* by minimizing PPL on WikiText-2; this assumes transfer to other tasks and models.
  • ad hoc to paper A softmax over layers of skewness scores yields valid per-layer retention fractions.
    Eq. (7) labels omega_l as a retention fraction, but sum omega_l = 1; no mapping to layerwise sparsity or enforcement of global sparsity S_g is given.

pith-pipeline@v1.3.0-alltime-deepseek · 20816 in / 17816 out tokens · 158262 ms · 2026-08-03T21:24:26.566405+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) have achieved remarkable performance on a wide range of tasks, hindering real-world deployment due to their massive size. Existing pruning methods (e.g., Wanda) tailored for LLMs rely heavily on manual design pruning algorithms, thereby leading to huge labor costs and requires expert knowledge. Furthermore, we are the first to identify the serious outlier value issue behind dramatic performance degradation under high pruning ratios that are caused by uniform sparsity, raising an additional concern about how to design adaptive pruning sparsity ideal for LLMs. Can LLMs prune by themselves? In this work, we introduce an affirmative answer by proposing a novel pruning method called AutoPrune, which first overcomes expert knowledge limits by leveraging LLMs to design optimal pruning algorithm for themselves automatically without any expert knowledge. Specifically, to mitigate the black-box nature of LLMs, we propose a Graph-driven Chain-of-Thought (GCoT) to optimize prompts, significantly enhancing the reasoning process in learning the pruning algorithm and enabling us to generate pruning algorithms with superior performance and interpretability in the next generation. Finally, grounded in insights of outlier value issue, we introduce Skew-aware Dynamic Sparsity Allocation (SDSA) to overcome the outlier value issue, mitigating performance degradation under high pruning ratios. We conduct extensive experiments on mainstream LLMs benchmarks, demonstrating the superiority of AutoPrune, which consistently excels state-of-the-art competitors.

Figures

Figures reproduced from arXiv: 2511.15390 by Enneng Yang, Haidong Kang, Hao Wang, Hongning Dai, Lihong Lin.

Figure 1
Figure 1. Figure 1: (a) AutoPrune v.s. (b) Manual Design. Manual de￾sign requires expert knowledge, resulting in huge labor costs. In contrast, our AutoPrune can efficiently design several specialized pruning algorithms by leveraging LLMs. ties of LLMs (e.g., code/text generation, contextual reason￾ing), we raise a new question: Can LLMs prune by them￾selves? In this paper, we propose a novel paradigm to auto￾matically design… view at source ↗
Figure 2
Figure 2. Figure 2: Our AutoPrune v.s. peer competitors on 7 zero-shot tasks. (a) LLaMA-1 7b. (b) LLaMA-2 7b. (c) LLaMA-2 13b [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Validation of layer sensitivity to pruning ratios. [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: SDAS v.s. Uniform allocation at 70% sparsity [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The intuitive framework of GCoT-driven self-pruning. This framework leverages an LLM to automatically generate pruning [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Results on the From Passive to Active Reasoning bench￾mark [41] under zero-shot settings. DC and GN are evaluated by accuracy, while SP is measured by F1 score. 4.6. Additional Evaluation Due to the page limit of the main text, we provide more re￾sults in App. C-G. ❶ Generalization for more LLMs and zero-shot tasks are shown in App. C. ❷ More ablation studies are shown in App. D. ❸ Detailed Prompt Engineer… view at source ↗
Figure 9
Figure 9. Figure 9: Validation of layer sensitivity to pruning ratios. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Validation of layer sensitivity to pruning ratios. [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 8
Figure 8. Figure 8: Validation of layer sensitivity to pruning ratios. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 17 linked inside Pith

  1. [1]

    https://openai.com/blog/chatgpt, 2022

    ChatGPT. https://openai.com/blog/chatgpt, 2022. 1

    2022

  2. [2]

    Lan- guage models are few-shot learners.Advances in neural in- formation processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners.Advances in neural in- formation processing systems, 33:1877–1901, 2020. 1

    1901

  3. [3]

    Sparks of ar- tificial general intelligence: Early experiments with gpt-4

    S ´ebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Jo- hannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of ar- tificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. 1

    Pith/arXiv arXiv 2023

  4. [4]

    Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044, 2019

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044, 2019. 6, 10

    Pith/arXiv arXiv 1905

  5. [5]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457,

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457,

    Pith/arXiv arXiv

  6. [6]

    The lottery ticket hy- pothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635, 2018

    Jonathan Frankle and Michael Carbin. The lottery ticket hy- pothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635, 2018. 15

    Pith/arXiv arXiv 2018

  7. [7]

    Sparsegpt: Massive lan- guage models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive lan- guage models can be accurately pruned in one-shot. InIn- ternational Conference on Machine Learning, pages 10323– 10337. PMLR, 2023. 1, 7, 8, 16

    2023

  8. [8]

    A frame- work for few-shot language model evaluation.Version v0

    Leo Gao, Jonathan Tow, Stella Biderman, Shawn Black, An- thony DiPofi, Charles Foster, Laurence Golding, Jasmine Hsu, Kyle McDonell, Niklas Muennighoff, et al. A frame- work for few-shot language model evaluation.Version v0. 0.1. Sept, 10:8–9, 2021. 6

    2021

  9. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 1, 4

    Pith/arXiv arXiv 2025

  10. [10]

    Op- timal brain surgeon and general network pruning

    Babak Hassibi, David G Stork, and Gregory J Wolff. Op- timal brain surgeon and general network pruning. InIEEE International Conference on Neural Networks, pages 293– 299, 1993. 1

    1993

  11. [11]

    Amc: Automl for model compression and ac- celeration on mobile devices

    Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and ac- celeration on mobile devices. InProceedings of the Euro- pean conference on computer vision (ECCV), pages 784– 800, 2018. 16

    2018

  12. [12]

    Mea- suring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding. InPro- ceedings of the International Conference on Learning Rep- resentations, 2021. arXiv:2009.03300. 2, 11

    Pith/arXiv arXiv 2021

  13. [13]

    Network trimming: A data-driven neuron pruning approach towards efficient deep architectures.arXiv preprint arXiv:1607.03250, 2016

    H Hu. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures.arXiv preprint arXiv:1607.03250, 2016. 15

    Pith/arXiv arXiv 2016

  14. [14]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 1, 6, 8, 10

    Pith/arXiv arXiv 2024

  15. [15]

    Analogcoder: Analog circuit design via training-free code generation

    Yao Lai, Sungyoung Lee, Guojin Chen, Souradip Poddar, Mengkang Hu, David Z Pan, and Ping Luo. Analogcoder: Analog circuit design via training-free code generation. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 379–387, 2025. 1

    2025

  16. [16]

    Optimal brain damage.Advances in neural information processing systems, 2, 1989

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage.Advances in neural information processing systems, 2, 1989. 15

    1989

  17. [17]

    Layer-adaptive sparsity for the magnitude-based pruning

    Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jin- woo Shin. Layer-adaptive sparsity for the magnitude-based pruning. InInternational Conference on Learning Represen- tations, 2021. 1, 7, 8

    2021

  18. [18]

    Discovering sparsity allocation for layer-wise pruning of large language models

    Lujun Li, Peijie Dong, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. Discovering sparsity allocation for layer-wise pruning of large language models. InThe Thirty-eighth An- nual Conference on Neural Information Processing Systems,

  19. [19]

    Adaptive layer sparsity for large language models via activation corre- lation assessment

    Wei Li, Lujun Li, Mark G Lee, and Shengjie Sun. Adaptive layer sparsity for large language models via activation corre- lation assessment. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 7, 16

    2024

  20. [20]

    Llm-pruner: On the structural pruning of large language models.Ad- vances in neural information processing systems, 36:21702– 21720, 2023

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models.Ad- vances in neural information processing systems, 36:21702– 21720, 2023. 16

    2023

  21. [21]

    Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843, 2016. 2, 6

    Pith/arXiv arXiv 2016

  22. [22]

    Llama 3 8b instruct

    Meta AI. Llama 3 8b instruct. https : / / github. com / meta- llama/llama3/blob/main/MODEL CARD.md, 2024. Instruction-tuned model with 8 billion parameters. 6, 8, 10

    2024

  23. [23]

    Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sab- harwal. Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018. 6, 10

    Pith/arXiv arXiv 2018

  24. [24]

    Pruning convolutional neural networks for re- source efficient inference.arXiv preprint arXiv:1611.06440,

    Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for re- source efficient inference.arXiv preprint arXiv:1611.06440,

    Pith/arXiv arXiv

  25. [25]

    Importance estimation for neural net- work pruning

    Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Fro- sio, and Jan Kautz. Importance estimation for neural net- work pruning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264– 11272, 2019

    2019

  26. [26]

    Sosp: Efficiently capturing global correla- tions by second-order structured pruning.arXiv preprint arXiv:2110.11395, 2021

    Manuel Nonnenmacher, Thomas Pfeil, Ingo Steinwart, and David Reeb. Sosp: Efficiently capturing global correla- tions by second-order structured pruning.arXiv preprint arXiv:2110.11395, 2021. 16

    Pith/arXiv arXiv 2021

  27. [27]

    Gpt-o3 system card.OpenAI System Card, 2025

    OpenAI. Gpt-o3 system card.OpenAI System Card, 2025. Technical report, April 16, 2025. 4

    2025

  28. [28]

    Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64 (9):99–106, 2021

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64 (9):99–106, 2021. 6, 10

    2021

  29. [29]

    The skewness of science.Journal of the Amer- ican society for information science, 43(9):628–638, 1992

    Per O Seglen. The skewness of science.Journal of the Amer- ican society for information science, 43(9):628–638, 1992. 2

    1992

  30. [30]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. InProceedings of the Twelfth International Confer- ence on Learning Representations (ICLR), 2024. 1, 7, 8, 10, 11, 16

    2024

  31. [31]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1, 6

    Pith/arXiv arXiv 2023

  32. [32]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 6

    Pith/arXiv arXiv 2023

  33. [33]

    Glue: A multi-task benchmark and analysis platform for natural language understanding.arXiv preprint arXiv:1804.07461, 2018

    Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding.arXiv preprint arXiv:1804.07461, 2018. 6, 10

    Pith/arXiv arXiv 2018

  34. [34]

    Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router.arXiv preprint arXiv:2410.12013, 2024

    Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router.arXiv preprint arXiv:2410.12013, 2024. 1

    Pith/arXiv arXiv 2024

  35. [35]

    Besa: Pruning large language models with block- wise parameter-efficient sparsity allocation.arXiv preprint arXiv:2402.16880, 2024

    Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. Besa: Pruning large language models with block- wise parameter-efficient sparsity allocation.arXiv preprint arXiv:2402.16880, 2024. 16

    Pith/arXiv arXiv 2024

  36. [36]

    Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity

    Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, et al. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. In International Conference on Machine Learning, 2024. 7

    2024

  37. [37]

    Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity

    Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, et al. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. In International Conference on Machine Learning, 2024. 10, 16

    2024

  38. [38]

    Auto graph encoder-decoder for neural network pruning

    Sixing Yu, Arya Mazaheri, and Ali Jannesari. Auto graph encoder-decoder for neural network pruning. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 6362–6372, 2021. 16

    2021

  39. [39]

    Carrying out cnn channel pruning in a white box.IEEE Transactions on Neural Networks and Learning Systems, 34(10):7946– 7955, 2022

    Yuxin Zhang, Mingbao Lin, Chia-Wen Lin, Jie Chen, Yongjian Wu, Yonghong Tian, and Rongrong Ji. Carrying out cnn channel pruning in a white box.IEEE Transactions on Neural Networks and Learning Systems, 34(10):7946– 7955, 2022. 16

    2022

  40. [40]

    A review on edge large language models: Design, execution, and applications.ACM Comput- ing Surveys, 57(8):1–35, 2025

    Yue Zheng, Yuhao Chen, Bin Qian, Xiufang Shi, Yuanchao Shu, and Jiming Chen. A review on edge large language models: Design, execution, and applications.ACM Comput- ing Surveys, 57(8):1–35, 2025. 1

    2025

  41. [41]

    Zhanke Zhou, Xiao Feng, Zhaocheng Zhu, Jiangchao Yao, Sanmi Koyejo, and Bo Han. From passive to active reason- ing: Can large language models ask the right questions under incomplete information? InProceedings of the 42nd In- ternational Conference on Machine Learning (ICML 2025), page –. ML Research Press, 2025. Poster. 6, 8, 10

    2025

  42. [42]

    Perp: Rethinking the prune-retrain paradigm in the era of llms.arXiv preprint arXiv:2312.15230, 2023

    Max Zimmer, Megi Andoni, Christoph Spiegel, and Sebas- tian Pokutta. Perp: Rethinking the prune-retrain paradigm in the era of llms.arXiv preprint arXiv:2312.15230, 2023. 6, 10 17

    arXiv 2023