Pith. sign in

REVIEW 4 major objections 5 minor 49 references

Adaptive Pruning for Large Language Models with Structural Importance Awareness

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that the safest structures to prune from a large language model are the ones whose importance scores vary most across calibration inputs, and that pruning those with grouped int4 fine-tuning beats four existing pruning…

desk verdict Broad experiments and a plausible fusion idea, but unreported fusion weights, a likely quantization sign error, and a self-contradictory ablation make the method irreproducible as written. read the letter →

arxiv 2412.15127 v1 pith:LBSC4WVD submitted 2024-12-19 cs.CL cs.AIcs.LG

Haotian Zheng , Jinke Ren , Yushan Sun , Ruichen Zhang , Wenbo Zhang , Zhen Li , Dusit Niyato , Shuguang Cui
show 1 more author
Yatong Han
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords largelanguagemodelstructuredpruningimportancefluctuationadaptivefusionstabilityindicatorgroup-wisefine-tuninglow-rankadaptationquantization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that a large language model can be pruned safely by removing structures whose importance scores are most volatile across a small calibration set, rather than structures with the lowest average importance. On this principle it builds SAAP, a structured-pruning pipeline that fuses coarse-grained and fine-grained importance into one adaptive score, standardizes the score's fluctuation across layers, and prunes the most fluctuating coupled structures. A grouped int4 fine-tuning step then recovers accuracy while reducing memory and increasing token-generation speed. The paper reports that SAAP beats four existing pruning methods on zero-shot classification and text generation, with accuracy gains of 2.17% on LLaMA-7B, 2.37% on Vicuna-7B, and 2.39% on LLaMA-13B, plus a 5% token-generation speed improvement. If the volatility-equals-redundancy premise holds, pruning decisions become cheaper because they can be made from only fifty random calibration samples rather than from retraining.

What carries the argument

The load-bearing object is the adaptive stability indicator $\hat{M}_{l,j}$ of Eq. (11), built from the importance fluctuation $M_{l,j}$ of Eq. (10), which measures how much the adaptively fused importance score $I^{\mathrm{ada}}_i$ for channel $j$ in layer $l$ varies across the calibration samples relative to the weight norm. The adaptive importance score itself fuses vector-wise and element-wise importance from LLM-pruner using a homoscedastic Gaussian likelihood, so the two importance views are weighted by their observation noise parameters $\lambda_1$ and $\lambda_2$ rather than by a fixed linear combination. The third mechanism is grouped int4 fine-tuning, which partitions each column of the weight matrix into $L$ groups, quantizes each group with its own scale $a$ and offset $b$, and adds a low-rank update $s\cdot AB$, cutting the number of fine-tuning parameters from $D_{\mathrm{in}}D_{\mathrm{int}} + D_{\mathrm{int}}D_{\mathrm{out}}$ to $LD_{\mathrm{int}} + D_{\mathrm{int}}D_{\mathrm{out}}$.

What would settle it

A controlled rerun that compares SAAP's volatility-based selection with random selection at the same pruning ratios and with the same grouped fine-tuning would settle whether fluctuation identifies redundancy: if random pruning matches or beats SAAP on the held-out zero-shot tasks, the volatility criterion is not what carries the gain. The paper's own ablation tables (Tables X and XI) report contradictory WikiText2 perplexities for the SAAP and without-stability-indicator conditions (16.35 vs 15.82 at 20% and 32.41 vs 31.73 at 50%), so such a rerun using the released code is the direct way to resolve it.

Watch

Extended reading notes

Core claim

The central claim is that in a transformer-based LLM, the coupled structures that are redundant are the ones whose importance scores fluctuate most across different calibration inputs. SAAP therefore computes an adaptive importance score by fusing vector-wise and element-wise importance under a homoscedastic Gaussian likelihood, then builds a stability indicator that standardizes each structure's importance variance across calibration samples, and removes the structures with the highest relative volatility. After pruning, it quantizes each column group of the weight matrix to int4 and fine-tunes low-rank adapters per group, so the precision and memory benefits of quantization combine with LoRA-style recovery. The paper argues that this pipeline outperforms LLM-pruner, LoraPrune, Wanda, and LoRAShear, with average accuracy gains of 2.17% on LLaMA-7B, 2.37% on Vicuna-7B, and 2.39% on LLaMA-13B, and a 5% token-generation speed improvement over the baselines.

Load-bearing premise

The load-bearing premise is that a structure whose importance score swings most across fifty randomly chosen Bookcorpus text samples is redundant and therefore safe to remove; if volatility does not track redundancy, the pruning selection loses its basis.

Editorial extensions

If this is right

  • SAAP turns layer-wise pruning ratios into an emergent property of the model: one standardized fluctuation score ranks all coupled structures across layers, so the pruning decision no longer needs a uniform ratio chosen by hand.
  • Because the importance estimate uses only fifty Bookcorpus calibration samples, the estimation stage is cheap enough to run at 33B and 65B scale; the paper reports SAAP beating LLM-pruner on WikiText2 perplexity at both 20% and 50% pruning ratios.
  • Grouped int4 fine-tuning makes the pruned model deployable on a single consumer GPU, with the paper's measurements showing token-generation speed rising by about 65% at a 50% pruning ratio for LLaMA-7B.
  • The volatility criterion appears to transfer across model families: the paper reports gains on LLaMA-7B, Vicuna-7B, and LLaMA-13B, and also runs LLaMA2 and LLaMA3 variants, suggesting the method is not tied to one architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If volatility of importance scores is a stable proxy for redundancy, then any cheap importance estimator could be made uncertainty-aware; a natural extension is to compute the coefficient of variation of existing scores such as weight-times-activation and prune by that, giving methods without SAAP's fusion the same cross-layer ranking.
  • The paper normalizes fluctuation by the weight norm and then standardizes across structures; an untested question is whether rank-based or unnormalized variance changes the pruning order, which would separate the role of scaling from the role of fluctuation.
  • The abstract's 5% speed improvement is smaller than the 32-65% speedups reported in the hardware table, so the 5% figure likely refers to the gain over the best baseline after fine-tuning rather than over the dense model; a direct head-to-head benchmark would disambiguate.
  • Since the calibration set is only fifty samples, the method's success depends on those samples representing the operating distribution; a testable extension is to draw calibration samples from different domains and measure how the pruning order and downstream accuracy change under distribution shift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SAAP, a structured pruning method for LLMs. It computes coarse- and fine-grained importance scores, fuses them through an uncertainty-weighted sum, defines an adaptive stability indicator based on the relative volatility of importance scores across calibration samples, prunes the most volatile structures while keeping fixed early/late layers, and recovers performance with a grouped quantized low-rank fine-tuning scheme. Experiments on LLaMA, Vicuna, LLaMA2, and LLaMA3 at 20% and 50% pruning ratios report accuracy gains over LLM-pruner, LoraPrune, Wanda, and LoRAShear on zero-shot classification and text generation, together with lower perplexity and higher token throughput.

Significance. If the claimed results hold, SAAP would offer a practical structured-pruning recipe with meaningful speed and memory gains, and the paper's breadth of model and dataset coverage is a strength. The paper also includes explicit ablations of its three components and reports memory/throughput statistics, which are useful. However, the central novelty—the adaptive stability indicator—is contradicted by the paper's own ablation, and key parameters and baseline results are missing; in its current form the paper does not establish the headline claims.

major comments (4)
  1. [Section V-D2, Table XI] The ablation of the adaptive stability indicator (ASI) contradicts the paper's central claim. Table XI reports WikiText2 perplexity of 15.82 at 20% pruning and 31.73 at 50% pruning 'Without ASI', versus 16.35 and 32.41 for SAAP. Since lower perplexity is better, removing ASI improves the pruned model, yet the text states that the volatility-based metric 'can greatly improve the accuracy of the model'. Moreover, the 'Without ASI' numbers equal the 'Separate Cal' entries and the 'SAAP' numbers equal the 'Weight Fusion' entries of Table X, which suggests a label swap or copying error. Because ASI is the paper's distinguishing contribution and the basis for 'structural importance awareness', this internal contradiction is load-bearing and must be resolved before the empirical claims can be assessed.
  2. [Section IV-B, Eq. (9)] The adaptive importance fusion metric in Eq. (9) is a weighted combination of the coarse- and fine-grained importance scores with coefficients 1/(2λ1^2) and 1/(2λ2^2) plus log λ1λ2. The paper never specifies how λ1 and λ2 are chosen or optimized, nor reports their values in any experiment. If the λ's are fit on the same 50 Bookcorpus calibration samples used for pruning selection, then part of the selection criterion is calibrated on the pruning data; if they are manually chosen, they are unreported hyperparameters. Either way, the evaluation is incomplete.
  3. [Section IV-C, Eq. (14)] Equation (14), W_l = a_g(W_g − b_g), is inconsistent with the definitions in Eq. (13), where a is a scale and b = min(W) is a zero-point offset. With those definitions, dequantization should be W_l = a_g W_g + b_g (or equivalently W_l = a_g(W_g + b_g) under a different convention); subtracting b_g as written would not place the dequantized values in the original range. This suggests a sign error in the core grouped fine-tuning formula and undermines the description of the recovery stage.
  4. [Section V-B, Tables I and V] The paper claims in the abstract and text that SAAP outperforms LoRAShear, but no LoRAShear results appear anywhere in the comparison tables: Table I lists only '-' for every LoRAShear row at both pruning ratios, and Tables IV–VI and the generalization tables do not include LoRAShear at all. Without LoRAShear measurements on any model, the claimed advantage over this baseline is unsupported.
minor comments (5)
  1. [Section IV-B, Eq. (10)] The symbol D is used both for the number of calibration samples in the averaging in Eq. (10) and for the dataset in Eq. (2), which is confusing; use different symbols.
  2. [References] Reference [36] is cited for Bessel correction but is a control-theory paper and does not seem to be the intended source; the statistical bias correction should be referenced to an appropriate statistics text or derived.
  3. [Section IV-C, Eq. (13)] The zero-point offset is defined as b = min(W), but the scale is defined using the full range; for asymmetric quantization the offset is usually round(min(W)/a) or min(W), and the formula should state the rounding convention.
  4. [Fig. 4] The caption 'LLM's answer' is grammatically awkward; consider 'LLM responses at different pruning ratios.'
  5. [Section V-A] The sentence 'the performance of the proposed method is further validated on the WikiText2 test set' appears in the discussion of Table II, but no test-set split is specified; clarify whether WikiText2 refers to the validation or test split.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor definitional circularity in Eq. (9); the central empirical claims are externally benchmarked and not circular.

  1. self definitional [Section IV-B, Eq. (9) (adaptive importance fusion)]
    "The adaptive importance score I ada i is calculated as I ada i = − log P (y1, y2|F (IW )) ∝ 1 2λ1 2 |y1 − F (I V i )|2 + 1 2λ2 2 |y2 − F (I E i )|2 + log λ1λ2 = 1 2λ1 2 I V i + 1 2λ2 2 I E i + log λ1λ2. (9)... We define I V i as the coarse-grained importance score, denoted as |y1 − F (I V i )|2."

    Eq. (9) defines the proposed adaptive importance fusion metric as a weighted sum of the two input importance scores I_V and I_E from LLM-pruner, with weights 1/(2λ1^2) and 1/(2λ2^2). The Gaussian likelihood derivation supplies only these weights; the fused score is the same information recombined, and the text then redefines I_V as the squared residual |y1−F(I_V)|^2, making the equation self-referential. Thus the claimed fusion is equivalent to its inputs by construction. This is not load-bearing for the headline results because final accuracy and perplexity are measured on held-out benchmarks, but it is a genuine definitional circularity in the method's derivation, and λ1 and λ2 are unreported.

full rationale

The central comparisons of SAAP are made on held-out zero-shot classification datasets (ARC, BoolQ, HellaSwag, PIQA, WinoGrande, OBQA), WikiText2/PTB perplexity, and token generation speed, so the main empirical claim is not circular: the pruning criterion is calibrated on Bookcorpus samples, while the reported gains are computed on external benchmarks. The only identifiable circularity is in the construction of the adaptive importance fusion metric in Eq. (9), where the fused score reduces by definition to a weighted sum of the two input importance scores, with the text redefining one input in terms of the same Gaussian residual. This step is minor and not load-bearing for the final benchmark conclusions. I also flag a non-circular but serious evidence-integrity issue: Table XI shows Without ASI achieving lower (better) WikiText2 perplexity than SAAP (15.82 vs 16.35 at 20%; 31.73 vs 32.41 at 50%), directly contradicting the sentence that the volatility-based metric can greatly improve accuracy. This appears to be a label or copy error and must be corrected, but it is a correctness/support issue rather than a circularity. Overall circularity burden is low.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central pruning decision depends on four hand-set or unreported parameters (fusion weights, group count, calibration size, layer protection rule) and on the assumption that volatility implies redundancy. No new physical or conceptual entities are introduced.

free parameters (4)
  • lambda_1, lambda_2 (fusion noise weights) = not reported
    Eq. (9) weights I_V and I_E by 1/(2*lambda_1^2) and 1/(2*lambda_2^2), but the paper gives no optimization procedure, initial values, or final values. These weights control which structures are pruned.
  • group count L = 32
    Set by hand in Section V-A; determines quantization group size and low-rank update shape in Eqs. (12)-(14).
  • calibration sample count = 50
    Chosen in Section V-A; the ablation in Fig. 6 varies 10, 30, and 50 samples and shows sensitivity.
  • middle-layer pruning ratio adjustment = 25% when target is 20%
    Section V-A keeps the first three and last layers fixed and increases the pruning ratio on the remaining layers; the adjustment is hand-set.
assumptions (4)
  • standard math Taylor expansion of loss change in Eqs. (1)-(2) can be truncated at second order
    The importance scores inherit LLM-pruner's second-order approximation; higher-order terms are dropped with no error bound in the context of pruning decisions.
  • domain assumption Importance scores follow a Gaussian likelihood with homoscedastic noise in Eq. (8)
    The fusion formula assumes the vector-wise and element-wise importance scores are noisy Gaussian observations of the same underlying importance; no diagnostic is given.
  • ad hoc to paper High relative volatility of importance scores indicates redundancy
    Eqs. (10)-(11) define the stability indicator, and the method prunes structures with maximum volatility; the paper offers no independent justification beyond its own ablation.
  • ad hoc to paper First three and last transformer layers are always kept
    Section V-A fixes these layers, which restricts the search space and affects all reported numbers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Pruning for Large Language Models with Structural Importance Awareness." pith.science (2026). https://pith.science/paper/LBSC4WVD

@misc{pith2026241215127,
  author       = {Pith},
  title        = {Pith review of: Adaptive Pruning for Large Language Models with Structural Importance Awareness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LBSC4WVD}},
  note         = {Machine review of arXiv:2412.15127}
}
read the original abstract

The recent advancements in large language models (LLMs) have significantly improved language understanding and generation capabilities. However, it is difficult to deploy LLMs on resource-constrained edge devices due to their high computational and storage resource demands. To address this issue, we propose a novel LLM model pruning method, namely structurally-aware adaptive pruning (SAAP), to significantly reduce the computational and memory costs while maintaining model performance. We first define an adaptive importance fusion metric to evaluate the importance of all coupled structures in LLMs by considering their homoscedastic uncertainty. Then, we rank the importance of all modules to determine the specific layers that should be pruned to meet particular performance requirements. Furthermore, we develop a new group fine-tuning strategy to improve the inference efficiency of LLMs. Finally, we evaluate the proposed SAAP method on multiple LLMs across two common tasks, i.e., zero-shot classification and text generation. Experimental results show that our SAAP method outperforms several state-of-the-art baseline methods, achieving 2.17%, 2.37%, and 2.39% accuracy gains on LLaMA-7B, Vicuna-7B, and LLaMA-13B. Additionally, SAAP improves the token generation speed by 5%, showcasing its practical advantages in resource-constrained scenarios.

Figures

Figures reproduced from arXiv: 2412.15127 by the authors.

Figure 1
Figure 1. The pipeline of existing LLM pruning methods. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An overview of the SAAP method. Given a foundation LLM, SAAP first removes the most volatile structure by adaptive importance assessment. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Average of adaptive importance fusion metrics of each layer in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: LLM’s answer under different pruning ratios. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The results of SAAP and LLM-pruner at different pruning ratios. (a) and (b) show the results of the Vicuna-7B model on the PTB and WikiText2 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Ablation study for calibration sample numbers. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 22 canonical work pages

  1. [1]

    LaMDA: Language models for dialog applications,

    R. Thoppilan et al., “LaMDA: Language models for dialog applications,” arXiv preprint arXiv:2201.08239 , Jan. 2022

    arXiv 2022

  2. [2]

    A survey on large language models: Applications challenges limitations and practical usage,

    M. U. Hadi et al. , “A survey on large language models: Applications challenges limitations and practical usage,” TechRxiv, Jul. 2023

    work page 2023

  3. [3]

    Emergent abilities of large language models,

    J. Wei et al. , “Emergent abilities of large language models,” arXiv preprint arXiv:2206.07682, Jun. 2022

    arXiv 2022

  4. [4]

    Toward democratized generative AI in next-generation mobile edge networks,

    R. Zhang et al., “Toward democratized generative AI in next-generation mobile edge networks,” arXiv preprint arXiv:2411.09148 , Nov. 2024

    arXiv 2024

  5. [5]

    Sparks of artificial general intelligence: Early exper- iments with gpt-4,

    S. Bubeck et al., “Sparks of artificial general intelligence: Early exper- iments with gpt-4,” arXiv preprint arXiv:2303.12712 , Mar. 2023

    arXiv 2023

  6. [6]

    IoT in the era of generative AI: Vision and challenges,

    X. Wang, Z. Wan, A. Hekmati, M. Zong, S. Alam, M. Zhang, and B. Krishnamachari, “IoT in the era of generative AI: Vision and challenges,” arXiv preprint arXiv:2401.01923 , Jan. 2024

    arXiv 2024

  7. [7]

    An overview of neural network compression,

    J. O. Neill, “An overview of neural network compression,” arXiv preprint arXiv:2006.03669, Jun. 2020

    arXiv 2006

  8. [8]

    Structured Model Pruning of Convolutional Networks on Tensor Processing Units

    K. Chen, K. Franko, and R. Sang, “Structured model pruning of convolutional networks on tensor processing units,” arXiv preprint arXiv:2107.04191, Jul. 2021

    work page Pith review arXiv 2021

Show all 49 references
  1. [9]

    Personalized federated learning by structured and unstructured pruning under data heterogeneity,

    S. Vahidian, M. Morafah, and B. Lin, “Personalized federated learning by structured and unstructured pruning under data heterogeneity,” in Proc. IEEE Int. Conf. on Distrib. Comput. Syst. (ICDCS) , Washington, DC, USA, Jul. 2021, pp. 27-34

    2021

  2. [10]

    Generative AI agents with large language model for satellite networks via a mixture of experts transmission,

    R. Zhang et al. , “Generative AI agents with large language model for satellite networks via a mixture of experts transmission,” IEEE J. Sel. Areas Commun., early access , Nov.2024

    2024

  3. [11]

    Optimal brain damage,

    Y . LeCun, J. Denker, and S. Solla, “Optimal brain damage,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS) , pp. 1-8, Nov. 1990

    1990

  4. [12]

    Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,

    S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149 , Oct. 2015. 12

    2015 arXiv

  5. [13]

    The design and implementation of xiaoice an empathetic social chatbot,

    L. Zhou, J. Gao, D. Li, and H.-Y . Shum, “The design and implementation of xiaoice an empathetic social chatbot,” Comput. Linguistics , vol. 46, no. 1, pp. 53-93, 2020

    2020

  6. [14]

    Interactive AI with retrieval-augmented generation for next generation networking,

    R. Zhang et al., “Interactive AI with retrieval-augmented generation for next generation networking,” IEEE Netw., early access , Nov.2024

    2024

  7. [15]

    Dynamic sparse graph for efficient deep learning,

    L. Liu, L. Deng, X. Hu, M. Zhu, G. Li, Y . Ding, and Y . Xie, “Dynamic sparse graph for efficient deep learning,” arXiv preprint arXiv:1810.00859, Oct. 2018

    2018 arXiv

  8. [16]

    ZipLM: Inference-aware struc- tured pruning of language models,

    E. Kurti ´c, E. Frantar, and D. Alistarh, “ZipLM: Inference-aware struc- tured pruning of language models,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), New Orleans, USA, Dec. 2023, pp. 65597-65617

    2023

  9. [17]

    Fluctuation-based adaptive structured pruning for large language models,

    Y . An, X. Zhao, T. Yu, M. Tang, and J. Wang, “Fluctuation-based adaptive structured pruning for large language models,” in AAAI Conf. Artif. Intell. (AAAI) , Vancouver, Canada, Feb. 2024, pp. 10865-10873

    2024

  10. [18]

    Llm-pruner: On the structural pruning of large language models,

    X. Ma, G. Fang, and X. Wang, “Llm-pruner: On the structural pruning of large language models,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), New Orleans, USA, Dec. 2023, pp. 21702-21720

    2023

  11. [19]

    Llama: Open and efficient foundation language models,

    H. Touvron et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, Feb. 2023

    2023 arXiv

  12. [20]

    Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality

    W. L. Chiang et al. , “Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality”, Apr. 2023. [Online]. Available: https://vicuna. lmsys. org

    2023

  13. [21]

    Glm: General language model pretraining with autoregressive blank infilling,

    Z. Du, Y . Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J, Tang, “Glm: General language model pretraining with autoregressive blank infilling,” arXiv preprint arXiv:2103.10360 , Mar. 2021

    2021 arXiv

  14. [22]

    A simple and effective pruning approach for large language models,

    M. Sun, Z. Liu, A. Bair, and J. Z. Kolter, “A simple and effective pruning approach for large language models,” arXiv preprint arXiv:2306.11695, 2023, Jun. 2022

    2023 arXiv

  15. [23]

    Lorashear: Effi- cient large language model structured pruning and knowledge recovery,

    T. Chen, T. Ding, B. Yadav, I. Zharkov, and L. Liang, “Lorashear: Effi- cient large language model structured pruning and knowledge recovery,” arXiv preprint arXiv:2310.18356 , Oct. 2023

    2023 arXiv

  16. [24]

    Sparsegpt: Massive language models can be accurately pruned in one-shot,

    E. Frantar and D. Alistarh, “Sparsegpt: Massive language models can be accurately pruned in one-shot,” in Proc. Int. Conf. Mach. Learn. (ICML), Hawaii, USA, Jul. 2023, pp.10323-10337

    2023

  17. [25]

    Pruning large language models via accuracy predictor,

    Y . Ji, Y . Cao, and J. Liu, “Pruning large language models via accuracy predictor,” arXiv preprint arXiv:2309.09507, Sep. 2023

    2023 arXiv

  18. [26]

    MINI-LLM: Memory-Efficient Structured Pruning for Large Language Models,

    B. K. Kim, G. Kim, T. H. Kim, T. Castells, S. Choi, J. Shin, and H. K. Song, “MINI-LLM: Memory-Efficient Structured Pruning for Large Language Models,” arXiv preprint arXiv:2402.02834 , Feb. 2024

    2024 arXiv

  19. [27]

    Shortened llama: A simple depth pruning for large language models,

    H. Cheng, M. Zhang, and J. Q. Shi, “Shortened llama: A simple depth pruning for large language models,” arXiv preprint arXiv:2407.11681 , Jul. 2024

    2024 arXiv

  20. [28]

    EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and V oting,

    Z. Yu et al. , “EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and V oting,” arXiv preprint arXiv:2406.15758 , Jun. 2024

    2024 arXiv

  21. [29]

    AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models,

    H. Lu et al. , “AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models,” arXiv preprint arXiv:2410.10912 , Oct. 2024

    2024 arXiv

  22. [30]

    Parameter-efficient fine-tuning for large models: A comprehensive survey,

    Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang, “Parameter-efficient fine-tuning for large models: A comprehensive survey,” arXiv preprint arXiv:2403.14608, Mar. 2024

    2024 arXiv

  23. [31]

    Parameter-efficient transfer learning for NLP,

    N. Houlsby et al. ,“Parameter-efficient transfer learning for NLP,” in Proc. Int. Conf. Mach. Learn (ICML) , California, USA, Jun, 2019, pp.2790-2799

    2019

  24. [32]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu et al., “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , Jun. 2021

    2021 arXiv

  25. [33]

    Qlora: Efficient finetuning of quantized llms

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), New Orleans, USA, Dec, 2023, pp. 10088-10115

    2023

  26. [34]

    A fast post-training pruning framework for transformers,

    W. Kwon, S. Kim, M. W. Mahoney, J. Hassoun, K. Keutzer, and A. Gholami, “A fast post-training pruning framework for transformers,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS) , New Orleans, USA, Nov, 2022, pp. 24101-24116

    2022

  27. [35]

    UPop: Unified and progressive pruning for compressing vision-language transformers,

    D. Shi, C. Tao, Y . Jin, Z. Yang, C. Yuan, and J. Wang, “UPop: Unified and progressive pruning for compressing vision-language transformers,” in Proc. Int. Conf. Mach. Learn. (ICML) , Hawaii, USA, Jul. 2023, pp.31292-31311

    2023

  28. [36]

    Polynomial-type Lya- punov–Krasovskii functional and Jacobi–Bessel inequality: Further re- sults on stability analysis of time-delay systems,

    Y . B. Huang, Y . He, J. An, and M. Wu, “Polynomial-type Lya- punov–Krasovskii functional and Jacobi–Bessel inequality: Further re- sults on stability analysis of time-delay systems,” IEEE Trans. Automat. Contr., vol. 66, no. 6, pp. 2905-2912, Jun. 2021

    2021

  29. [37]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron et al. ,“Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, Jul, 2023

    2023 arXiv

  30. [38]

    How good are low-bit quantized llama3 models? An empirical study,

    W. Huang et al., “How good are low-bit quantized llama3 models? An empirical study,” arXiv preprint arXiv:2404.14047 , Apr. 2024

    2024 arXiv

  31. [39]

    Think you have solved question answering? try arc, the ai2 reasoning challenge,

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” arXiv preprint arXiv:1803.05457 , Mar. 2018

    2018 arXiv

  32. [40]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions,

    C. Clark, K. Lee, M. W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “BoolQ: Exploring the surprising difficulty of natural yes/no questions,” arXiv preprint arXiv:1905.10044 , May 2019

    1905 arXiv

  33. [41]

    Hel- laswag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hel- laswag: Can a machine really finish your sentence?” arXiv preprint arXiv:1905.07830, May 2019

    1905 arXiv

  34. [42]

    Piqa: Reasoning about physical commonsense in natural language,

    Y . Bisk, R. Zellers, R. L. Bras, J. Gao, and Y . Choi, “Piqa: Reasoning about physical commonsense in natural language,” in AAAI Conf. Artif. Intell.(AAAI), New York, USA, Feb. 2020, pp. 7432-7439

    2020

  35. [43]

    Winogrande: An adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: An adversarial winograd schema challenge at scale,” Communications of the ACM , vol. 64, no. 9, pp. 99-106, Aug. 2021

    2021

  36. [44]

    Can a suit of armor conduct electricity? A new dataset for open book question answering,

    T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? A new dataset for open book question answering,” arXiv preprint arXiv:1809.02789 , Sep. 2018

    2018 arXiv

  37. [45]

    Building a large annotated corpus of English: The penn treebank,

    M. Marcus, B. Santorini, and M. A. Marcinkiewicz, “Building a large annotated corpus of English: The penn treebank,” Computational lin- guistics, vol. 19, no. 2, pp. 313-330, 1993

    1993

  38. [46]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843 , Sep. 2016

    2016 arXiv

  39. [47]

    LoRAPrune: Structured pruning meets low-rank parameter-efficient fine-tuning,

    M. Zhang, H. Chen, C. Shen, Z. Yang, L. Ou, X. Yu, and B. Zhuang, “LoRAPrune: Structured pruning meets low-rank parameter-efficient fine-tuning,” arXiv preprint arXiv:2305.18403 , May 2023

    2023 arXiv

  40. [48]

    Aligning books and movies: Towards story-like visual ex- planations by watching movies and reading books,

    Y . Zhu, “Aligning books and movies: Towards story-like visual ex- planations by watching movies and reading books,” arXiv preprint arXiv:1506.06724, Jun. 2015

    2015 arXiv

  41. [49]

    WizardLM: Empowering large language models to follow complex instructions,

    C. Xu et al., “WizardLM: Empowering large language models to follow complex instructions,” arXiv preprint arXiv:2304.12244 , Apr. 2023

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.