Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Information

Hyunjik Jo; Jinsik Lee; JongJin Kim; Seungcheol Park; Sojin Lee; U Kang

REVIEW 4 major objections 4 minor 43 references

Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Information

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sublayer pruning score gains up to 23.88 accuracy points on Llama

desk verdict Solid incremental sublayer pruning paper with clean ablations; the main caveat is an unvalidated selection proxy and some benchmark-tuned hyperparameters. read the letter →

arxiv 2506.03510 v1 pith:JOTSOH4J submitted 2025-06-04 cs.CL

Seungcheol Park , Sojin Lee , Jongjin Kim , Jinsik Lee , Hyunjik Jo , U Kang This is my paper

classification cs.CL

keywords sublayerpruninglargelanguagemodelslatency-awareimportancescoringtunability-awaresensitivityLLMaccelerationin-compressiontuningstructuralzero-shotcommonsensereasoning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes SPRINT, a method for speeding up large language models by deleting whole sublayers—either the multi-head attention block or the MLP block—rather than fine-grained weights. Its central claim is that prior sublayer-pruning rules pick the wrong layers because they ignore two measurable properties: how much latency a layer removal actually saves, and how much of the accuracy damage tuning can repair. SPRINT scores each candidate sublayer as the ratio of post-tuning sensitivity to latency saved, and iteratively removes the lowest-score sublayer while quickly re-tuning the survivors. On Llama-2 and Llama-3 models at matched speedups, the authors report the best accuracy-versus-speedup trade-off among compared methods, with up to 23.88 percentage points higher zero-shot commonsense accuracy than existing sublayer-pruning baselines. A sympathetic reader would take this as evidence that sublayer selection should be formulated as a cost-effectiveness problem rather than a pure damage-minimization problem.

What carries the argument

The central object is the ratio $\eta(s)=\zeta(s)/t(s)$ together with the tunability-aware sensitivity estimator $\zeta(s)$ of Equation 5. To measure $\zeta(s)$, SPRINT locates the nearest MLP sublayer above the candidate, prunes the candidate, performs fast in-compression tuning on that MLP's output projection by solving $\arg\min_{\hat W^{(d)}} \|(\hat X^{(d)}+\hat W^{(d)}\hat Z^{(d)}) - X^{(d+1)}\|_F^2$, and takes the normalized Frobenius distance between original and tuned outputs. Two efficiency mechanisms carry the method's practical feasibility: activation checkpointing reuses sensitivities that cannot change after a given prune, and fast candidate selection first scores all sublayers with a cheap untuned pseudo-importance $\tilde\eta(s)=\tilde\zeta(s)/t(s)$ and runs the expensive tuned evaluation only on the top $\beta$ candidates.

What would settle it

On a small Llama model, compute the correlation between SPRINT's sensitivity scores $\zeta(s)$ and the actual task-accuracy drop from pruning each sublayer; if the ranking correlation is weak or negative, the selection criterion is not faithful. A stronger version: exhaustively search all sublayer subsets within a fixed latency budget and show a subset that beats SPRINT's chosen model by a clear margin.

Watch

Extended reading notes

Core claim

At the center of SPRINT is the importance score $\eta(s)=\zeta(s)/t(s)$ (Equation 4), where $t(s)$ is the measured latency reduction from deleting sublayer $s$ and $\zeta(s)$ is the sublayer's tunability-aware sensitivity. Sensitivity is computed not from the immediate output of the pruned sublayer but at the closest upper MLP sublayer, after a fast in-compression tuning step fits the surviving output projection via least squares to match the original model's activations (Equations 3 and 5). The paper's claim is that repeatedly pruning the sublayer with the smallest $\eta(s)$—not the smallest raw sensitivity—yields the best accuracy for a given latency budget, because it favors sublayers whose removal costs little after tuning relative to the latency they return. The experiments support this by showing SPRINT dominating ShortGPT, SLEB, and BlockPruner on Llama-2 7B/13B/70B and Llama-3 8B/70B across five zero-shot tasks, and the ablation study attributes the gain specifically to the latency and tunability terms.

Load-bearing premise

The method assumes that the quick local repair test it runs on each candidate sublayer reliably predicts which sublayer will hurt real task accuracy least once the whole model is pruned and tuned; the paper uses this proxy to choose layers without ever validating it against task scores during selection.

Editorial extensions

If this is right

At a 1.4× speedup, SPRINT keeps Llama-2 7B at 66.23% average accuracy versus 60.02% for BlockPruner and 53.12% for SLEB, and Llama-3 8B at 69.82% versus 63.63% for BlockPruner.
Because pruning one MLP sublayer saves about as much latency as pruning roughly three MHA sublayers, latency-aware scoring systematically pushes selection toward MHA blocks, so a fixed latency budget removes fewer sublayers overall.
Tuning changes the damage ranking: sublayers that look least harmful before tuning are not the ones that stay least harmful after tuning, so selection must use post-tuning sensitivity.
The pruning patterns imply that in Llama-2 and Llama-3, MLP sublayers and lower-position sublayers are the load-bearing ones, while upper-middle MHA sublayers are the safest to remove.
On 70B models, the method prunes Llama-2 70B and Llama-3 70B in hours rather than weeks, whereas the comparison method BlockPruner was estimated to take more than a week.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test the paper does not run: on a small Llama model, compare SPRINT's selected sublayer sequence against exhaustive search over all subsets meeting the same latency budget; the size of the accuracy gap would quantify how much the proxy-based selection leaves on the table.
The same $\zeta(s)/t(s)$ cost-effectiveness form could transfer to other sequentially executed blocks—vision transformers or encoder-only models—where block latencies differ and post-removal tuning recovers unevenly; the paper's experiments are limited to Llama decoder models.
The pattern that MLP sublayers are pruned only in the 70B models hints that scale changes which sublayers are redundant; if so, the optimal pruning pattern for future larger models cannot be read off from small-model results and needs re-measurement.
Because the importance score uses measured latency rather than FLOPs, the method's layer choices are hardware- and batch-size-dependent: a different GPU or serving configuration could reorder $\eta(s)$ and select a different pruning set.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

One line: SPRINT is a genuine improvement to sublayer pruning, and its real virtue is computational economy. The novelty is real but narrow: they divide sensitivity (measured after a fast least-squares tune) by per-sublayer latency, so the importance score actually reflects 'bang per millisecond.' That combination—latency-aware scoring plus tunability-aware sensitivity—is, as far as I know, new in this line of work. The ablations in Table 1 give it credit: dropping either signal costs 2-3 points on Llama-3 8B at 1.4x. The efficiency tricks (checkpointing, fast candidate selection) are not deep but they are sensible, and the pruning-time numbers back them up. Code is released; I'd want to run it before relying on the speedups.

The soft spots are real but not fatal. The selection score zeta (Eq. 5) is a normalized activation distance measured at the nearest upper MLP after one-layer least-squares tuning. The whole method assumes this proxy ranks sublayers the way end-task accuracy would. The paper never checks that alignment—no correlation, no oracle comparison. Figure 3(b) shows the ranking changes when you account for tuning, which motivates the idea, but it doesn't show the proxy is faithful enough to greedily pick sublayers. This is the same gap you see in most pruning papers, but here the claim is 'accurate selection,' so the gap is load-bearing. I'd want a small experiment: select sublayers with zeta, with random selection, and with oracle (actual accuracy after pruning), and show zeta tracks oracle better than random. That would settle it.

Second, the hyperparameter c (channel-selection) is chosen on the same five benchmarks used to report final accuracy (Table 5), and beta is tuned on Llama-3 8B final accuracy (Table 6). That injects a bit of selection into the headline numbers. Not disqualifying—they report three seeds and the gains over baselines are large—but it's worth noting in any review.

Third, the Llama-2-70B unpruned baseline has a standard deviation of 3.51 points (Table 10), which is way out of line with every other model. That smells like something noisy (or a typo) and puts a question mark over the 70B comparisons. Likewise, BlockPruner accuracy at 70B is simply absent; they estimate its pruning time but don't report its accuracy, so the 'best trade-off' claim at 70B is against a partial set. This is a missing baseline, not a fudge, but it should be called out.

Who is this for? People working on structural pruning or LLM inference optimization. It's a competent empirical paper, not a breakthrough. It deserves a serious referee—the kind who will ask for the proxy validation and the 70B baseline. I'd accept it for review and push for a moderate revision.

Referee Report

4 major / 4 minor

Summary. The paper proposes SPRINT, an iterative sublayer pruning method for LLMs. It scores each sublayer by eta(s) = zeta(s)/t(s), where t(s) is the measured latency reduction from pruning the sublayer and zeta(s) is a tunability-aware sensitivity defined as the normalized distance between the unpruned and fast-tuned outputs at the closest upper MLP sublayer after pruning (Eq. 4 and Eq. 5). SPRINT greedily removes the lowest-eta sublayer and applies fast in-compression tuning to the remaining model, using activation checkpointing and fast candidate selection to reduce pruning cost. Experiments on Llama-2 7B/13B/70B and Llama-3 8B/70B across five zero-shot commonsense benchmarks report the best accuracy-speedup trade-off and up to 23.88 percentage points higher accuracy than baselines.

Significance. If the empirical claims hold, SPRINT is a practically useful contribution: it introduces a simple, inexpensive selection signal that combines latency and post-tuning sensitivity, and it is evaluated across five model scales and five benchmarks with source code released. The ablations in Table 1 support that both latency-aware scoring and tunability-aware sensitivity contribute to accuracy, and the two efficiency techniques are clearly motivated. However, the current evidence has important gaps concerning validation of the selection proxy, selection of hyperparameters on evaluation benchmarks, and statistical reliability at the 70B scale, so the central accuracy claim is not yet fully established.

major comments (4)

[Section 3.3, Eq. (5)] The paper never validates that the tunability-aware sensitivity zeta(s) ranks sublayers by their true effect on end-task accuracy after the final pruning. The ablation in Table 1 shows that including tunability changes the final accuracy (SPRINT-t 67.62 vs SPRINT 69.82 on Llama-3 8B), but it does not show that the ranking is faithful for the greedy multi-step selection loop. Please add a direct validation: for one model and pruning step, compute zeta(s) for all candidates and compare it with the actual task-accuracy drop after pruning and tuning each candidate, or compare SPRINT's choices with an oracle that selects by downstream benchmark accuracy. Without such a check, the central selection mechanism remains an unmeasured proxy.
[Appendix B.3, Tables 5-6] The channel-selection hyperparameter c is chosen on Llama-3 70B by maximizing average accuracy on the same five benchmarks that are later reported in Table 12, and the candidate hyperparameter beta is tuned on Llama-3 8B final accuracy (Table 6). This is selection on the evaluation set and can inflate the headline results. Please select hyperparameters on a held-out validation set or a proxy such as validation perplexity and report test-set accuracy, or at minimum report the full c and beta sensitivity grid so readers can see how much the main comparisons depend on these choices.
[Table 10] The unpruned Llama-2 70B row reports an average of 77.72 with a standard deviation of 3.51, while SPRINT rows in the same table report standard deviations of at most 0.66. This noise level is large relative to many of the reported differences at 70B (e.g., SPRINT 1.1x 78.38 vs ShortGPT 1.1x 76.23, or SPRINT 1.1x vs unpruned). Please report per-seed results, confidence intervals, or paired tests and explain the source of the 3.51-point variance; without this, the 70B comparisons are not statistically defensible.
[Section 4.2, Tables 10 and 12] BlockPruner accuracy is not reported at 70B (the tables show '-'), and its 70B pruning time is estimated rather than measured (Section B.4). Since BlockPruner is one of the three sublayer-pruning baselines and is the strongest competitor at smaller scales, the statement that SPRINT achieves the best accuracy-speedup trade-off at 70B is not fully supported. Please either provide BlockPruner 70B accuracy, run it on a smaller model with a comparable setup, or explicitly restrict the claim to the scales where BlockPruner was evaluated.

minor comments (4)

[Section 3.3] The statement that an MLP sublayer has three times more parameters than an MHA sublayer is model-dependent; please qualify it for the specific Llama configurations used.
[Section 3.4] The relation between the number alpha of checkpoints and the memory/time trade-off is described only qualitatively; a small sensitivity study of alpha would help readers choose it in practice.
[Tables 8-12] Standard deviations are reported with inconsistent precision (e.g., +/-0.01 vs +/-0.00); please use a uniform number of decimals throughout.
[Figure 5] The trade-off figure is dense; adding error bars or confidence bands and marking where BlockPruner is absent would improve readability and support the strength of the comparisons.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the core selection score is measured and benchmark results are external, with only minor test-set hyperparameter fitting.

full rationale

The derivation chain is not circular. The selection score eta(s) = zeta(s)/t(s) (Eq. 4) is a measured quantity: t(s) is measured latency reduction and zeta(s) is the normalized output distance after fast in-compression tuning (Eq. 5), evaluated on calibration activations rather than on the downstream accuracy benchmarks. The model is then pruned greedily (Algorithm 1), and the claimed accuracy-speedup trade-off is obtained by running the pruned model on external commonsense benchmarks; the headline numbers are not algebraically entailed by the score definition. The cited fast in-compression tuning [Park et al., 2024a] is prior published work and is additionally supported in this paper by the SPRINT-t ablation (Table 1: 67.62 vs 69.82), so the self-citation is not load-bearing in the sense of forbidding alternatives or supplying the central result. The only notable concern is methodological rather than circular: c and beta are selected in Appendix B.3 (Tables 5 and 6) by maximizing average accuracy on the same five benchmarks later reported (e.g., c=75 chosen on Llama-3 70B at 40% speedup; beta=5 chosen on Llama-3 8B at 40% speedup), which inflates those specific reported accuracies and should be disclosed as test-set selection. Whether Eq. 5 faithfully ranks end-task loss after final pruning is an unvalidated proxy assumption, but that is a correctness risk, not circularity. No equation reduces to a pre-fitted value, and no prediction is forced by the definition of the importance score. Overall: no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

SPRINT introduces no new physical or conceptual entities; it selects existing sublayers. The central dependencies are the uniform-latency-per-type assumption, the post-tuning output-distance proxy for accuracy, and the hyperparameters c, beta, and alpha, one of which (c) is tuned on the evaluation benchmark.

free parameters (3)

c (channel-selection hyperparameter) = 100 for Llama-2 7B/13B and Llama-3 8B; 75 for 70B models
Percentage of rows in the output projection that are tuned during sensitivity evaluation. Chosen by searching [25,50,75,100] on Llama-3 70B at 40% speedup using the same five evaluation benchmarks, and set to 100 for small models without search. This is a hand-tuned, evaluation-dependent parameter.
beta (candidate hyperparameter) = 5 for all models
Number of candidate sublayers selected via pseudo-importance before tunability-aware evaluation. Chosen on Llama-3 8B at 40% speedup by comparing beta in [1,3,5,7,9] on the evaluation benchmarks.
alpha (checkpointing hyperparameter) = 8
Number of activation checkpoints. Set to 8 for all models; no sensitivity analysis is reported, so it is a fixed hyperparameter.

assumptions (5)

domain assumption Removing an entire sublayer (MHA or MLP) preserves a functional model because of residual connections.
Invoked throughout Section 2.3 and in the iterative pruning loop; the paper cites prior sublayer pruning work rather than proving that removal is safe.
domain assumption The normalized output distance at the closest upper MLP sublayer (Equation 5) is a valid proxy for downstream task accuracy loss after pruning.
Introduced in Section 3.3; the paper does not independently validate this proxy against actual task accuracy during selection.
domain assumption Fast in-compression tuning via least squares (Equation 3) adequately restores model outputs and downstream accuracy.
Adopted from Park et al. 2024a in Section 2.4; the paper relies on this for both sensitivity measurement and final pruning.
domain assumption Latency reduction is approximately constant per sublayer type (t(MHA) or t(MLP)) and can be measured before pruning.
Used in Equation 4 and Section 3.2; the paper measures t for each type once, assuming it does not depend on which specific sublayer is removed.
domain assumption The five commonsense reasoning benchmarks used for evaluation are representative of LLM capability for the pruning decision.
The method is tuned and evaluated on the same set of benchmarks in the hyperparameter search, so the reported trade-off is specific to these tasks.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Information." pith.science (2026). https://pith.science/paper/JOTSOH4J

@misc{pith2026250603510,
  author       = {Pith},
  title        = {Pith review of: Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Information},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JOTSOH4J}},
  note         = {Machine review of arXiv:2506.03510}
}

read the original abstract

How can we accelerate large language models(LLMs) without sacrificing accuracy? The slow inference speed of LLMs hinders us to benefit from their remarkable performance in diverse applications. This is mainly because numerous sublayers are stacked together in LLMs. Sublayer pruning compresses and expedites LLMs via removing unnecessary sublayers. However, existing sublayer pruning algorithms are limited in accuracy since they naively select sublayers to prune, overlooking the different characteristics of each sublayer. In this paper, we propose SPRINT (Sublayer PRuning wIth LateNcy and Tunability Information), an accurate sublayer pruning method for LLMs. SPRINT accurately selects a target sublayer to prune by considering 1) the amount of latency reduction after pruning and 2) the tunability of sublayers. SPRINT iteratively prunes redundant sublayers and swiftly tunes the parameters of remaining sublayers. Experiments show that SPRINT achieves the best accuracy-speedup trade-off, exhibiting up to 23.88%p higher accuracy on zero-shot commonsense reasoning benchmarks compared to existing pruning algorithms.

Figures

Figures reproduced from arXiv: 2506.03510 by the authors.

**Figure 1.** (a) Comparison of different schemes in pruning for large language models, given the characteristics of each sublayer. Each scheme [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** An illustration of the overall process of SPRINT. Given a pretrained LLM and a latency constraint, SPRINT iteratively identifies [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** (a) Latency change after pruning MHA and MLP sublayers in Llama-3 8B model. MHA sublayers impact more than MLP sublayers [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: (a) Unnecessary computations in iterative sublayer pruning. The sensitivity of a sublayer is unchanged if its closest upper MLP [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Accuracy-speedup trade-off curves of SPRINT and competitors. SPRINT shows the best trade-off among all the methods. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Pruning time and accuracy trade-off of SPRINT and competitors under 1.4 [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Pruning patterns of SPRINT on Llama models (best [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 43 canonical work pages

[1]

Slicegpt: Compress large language models by deleting rows and columns

[Ashkboos et al., 2024] Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv,

work page 2024
[3]

• ARC-Challenge and ARC-Easy[Clark et al., 2018] are datasets composed of grade-school level multiple-choice science questions

We utilize evaluation source codes in language model evaluation harness [Gao et al., 2023]. • ARC-Challenge and ARC-Easy[Clark et al., 2018] are datasets composed of grade-school level multiple-choice science questions. The challenge set contains only ques- tions that neither a retrieval-based algorithm nor a word co-occurrence algorithm correctly answers...

work page 2023
[4]

Pea-kd: Parameter-efficient and accurate knowledge distillation on bert

[Cho and Kang, 2022] Ikhyun Cho and U Kang. Pea-kd: Parameter-efficient and accurate knowledge distillation on bert. PLOS ONE, 17(2),

work page 2022
[6]

This demonstrates that five candidates are sufficient to find the sublayer with the least importance score

Note that the accuracy remains constant when β is 5 or greater. This demonstrates that five candidates are sufficient to find the sublayer with the least importance score. Hence, we set β to 5 for all experiments. β 1 3 5 7 9 Step 7 74.14 74.19 74.19 74.19 74.19 Step 13 70.71 70.66 70.71 70.71 70.71 Table 6: Hyperparameter search for β. B.4 Implementation...

work page 2023
[9]

Sparsegpt: Massive language models can be accurately pruned in one-shot

[Frantar and Alistarh, 2023] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In ICML,

work page 2023
[10]

OPTQ: Accurate quantization for generative pre-trained transformers

[Frantar et al., 2023] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In ICLR,

work page 2023
[11]

Lazyllm: Dynamic token pruning for efficient long context llm inference

[Fu et al., 2024] Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, and Mahyar Najibi. Lazyllm: Dynamic token pruning for efficient long context llm inference. arXiv,

work page 2024
[12]

A frame- work for few-shot language model evaluation, 12

[Gao et al., 2023] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and ...

work page 2023

Show all 43 references

[13]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

[Hu et al., 2022] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large lan- guage models. In ICLR,

2022
[14]

Falcon: lightweight and accurate convo- lution based on depthwise separable convolution

[Jang et al., 2023] Jun-Gi Jang, Chun Quan, Hyun Dong Lee, and U Kang. Falcon: lightweight and accurate convo- lution based on depthwise separable convolution. Knowl. Inf. Syst., 65(5):2225–2249,

2023
[15]

[Jeon et al., 2023] Hyojin Jeon, Seungcheol Park, Jin-Gee Kim, and U. Kang. Pet: Parameter-efficient knowledge distillation on transformer. PLOS ONE, 18(7),

2023
[16]

[Kim et al., 2021] Junghun Kim, Jinhong Jung, and U. Kang. Compressing deep graph convolution network with multi- staged knowledge distillation. PLOS ONE, 16,

2021
[18]

[Lee et al., 2021] Hyun Dong Lee, Seongmin Lee, and U. Kang. Auber: Automated bert regularization. PLOS ONE, 16(6),

2021
[20]

DDK: distilling domain knowledge for efficient large language models

[Liu et al., 2024] Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Zhiqi Bai, Jie Liu, Ge Zhang, Jiakai Wang, Yanan Wu, Congnan Liu, Jiamang Wang, Lin Qu, Wenbo Su, and Bo Zheng. DDK: distilling domain knowledge for efficient large language mode...

2024
[21]

Llm-pruner: On the structural pruning of large lan- guage models

[Ma et al., 2023] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large lan- guage models. In NeurIPS,

2023
[22]

Shortgpt: Layers in large language models are more redundant than you expect

[Men et al., 2024] Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv,

2024
[23]

Pointer sentinel mixture models

[Merity et al., 2016] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv,

2016
[24]

Sensimix: Sensitivity-aware 8-bit index & 1-bit value mixed precision quantization for bert compression

[Piao et al., 2022] Tairen Piao, Ikhyun Cho, and U Kang. Sensimix: Sensitivity-aware 8-bit index & 1-bit value mixed precision quantization for bert compression. PloS one, 17(4),

2022
[25]

Mixture-of-depths: Dynamically al- locating compute in transformer-based language models

[Raposo et al., 2024] David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically al- locating compute in transformer-based language models. arXiv,

2024
[26]

Exaone 3.0 7.8 b instruction tuned language model

[Research et al., 2024] LG Research, Soyoung An, Kyunghoon Bae, Eunbi Choi, Stanley Jungkyu Choi, Yemuk Choi, Seokhee Hong, Yeonjung Hong, Junwon Hwang, Hyojin Jeon, et al. Exaone 3.0 7.8 b instruction tuned language model. arXiv,

2024
[27]

Confident adaptive language modeling

[Schuster et al., 2022] Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. In NeurIPS,

2022
[28]

Omniquant: Omnidi- rectionally calibrated quantization for large language mod- els

[Shao et al., 2024] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidi- rectionally calibrated quantization for large language mod- els. In ICLR,

2024
[29]

Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks

[Song et al., 2024] Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. arXiv,

2024
[30]

A simple and effective pruning approach for large language models

[Sun et al., 2023] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv,

2023
[31]

Gemini: a family of highly capable multimodal models

[Team et al., 2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Mil- lican, et al. Gemini: a family of highly capable multimodal models. arXiv,

2023
[32]

Accelerating llama infer- ence by enabling intermediate layer decoding via instruc- tion tuning with lite

[Varshneyet al., 2023] Neeraj Varshney, Agneet Chatterjee, Mihir Parmar, and Chitta Baral. Accelerating llama infer- ence by enabling intermediate layer decoding via instruc- tion tuning with lite. arXiv,

2023
[33]

Attention is all you need

[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS,

2017
[35]

Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning llms to high sparsity

[Yin et al., 2024] Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Kumar Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bender- sky, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for prunin...

2024
[36]

Knowledge extraction with no observable data

[Yoo et al., 2019] Jaemin Yoo, Minyong Cho, Taebum Kim, and U Kang. Knowledge extraction with no observable data. In NeurIPS,

2019
[37]

Hellaswag: Can a ma- chine really finish your sentence? arXiv,

[Zellers et al., 2019] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? arXiv,

2019
[38]

Opt: Open pre-trained transformer language models

[Zhang et al., 2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv,

2022
[39]

Blockpruner: Fine- grained pruning for large language models

[Zhong et al., 2024] Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzhi Li. Blockpruner: Fine- grained pruning for large language models. arXiv,

2024
[43]

LLM-Pruner cannot prune Llama-2 70B and Llama-3 8B, 70B since it does not support group query attention

The values in parentheses denote the standard devia- tions. LLM-Pruner cannot prune Llama-2 70B and Llama-3 8B, 70B since it does not support group query attention. We do not report the results of BlockPruner for 70B models due to its excessive pruning cost. Method Speedup Pru...
[1024]

We use a single GPU to evaluate Llama-2 7B, 13B and Llama-3 8B

We set the size of the batch to 1 when we measure the laten- cies of models. We use a single GPU to evaluate Llama-2 7B, 13B and Llama-3 8B. We use two GPUs to evaluate Llama-2 70B and Llama-3 70B. For fine-grained pruning methods, we use the settings with the highest accuraci...

2024
[2017]

Qa-lora: Quantization-aware low-rank adaptation of large language models

[Xu et al., 2024] Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xi- aopeng Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models. In ICLR,

2024
[2018]

Boolq: Exploring the surprising diffi- culty of natural yes/no questions

[Clark et al., 2019] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising diffi- culty of natural yes/no questions. arXiv,

2019
[2019]

The llama 3 herd of models

[Dubey et al., 2024] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv,

2024
[2020]

Language models are few-shot learners

[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS,

2020
[2021]

Flexround: Learnable rounding based on element-wise division for post-training quantiza- tion

[Lee et al., 2023] Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. Flexround: Learnable rounding based on element-wise division for post-training quantiza- tion. In ICML,

2023
[2022]

Palm: Scaling language modeling with pathways

[Chowdhery et al., 2023] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, ...

2023
[2023]

Think you have solved question answer- ing? try arc, the ai2 reasoning challenge

[Clark et al., 2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answer- ing? try arc, the ai2 reasoning challenge. arXiv,

2018
[2024]

Piqa: Reasoning about physical commonsense in natural language

[Bisk et al., 2020] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In AAAI,

2020
[2025]

Distillm: Towards streamlined distil- lation for large language models

[Ko et al., 2024] Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distil- lation for large language models. In ICML,

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Slicegpt: Compress large language models by deleting rows and columns

[Ashkboos et al., 2024] Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv,

work page 2024

[2] [3]

• ARC-Challenge and ARC-Easy[Clark et al., 2018] are datasets composed of grade-school level multiple-choice science questions

We utilize evaluation source codes in language model evaluation harness [Gao et al., 2023]. • ARC-Challenge and ARC-Easy[Clark et al., 2018] are datasets composed of grade-school level multiple-choice science questions. The challenge set contains only ques- tions that neither a retrieval-based algorithm nor a word co-occurrence algorithm correctly answers...

work page 2023

[3] [4]

Pea-kd: Parameter-efficient and accurate knowledge distillation on bert

[Cho and Kang, 2022] Ikhyun Cho and U Kang. Pea-kd: Parameter-efficient and accurate knowledge distillation on bert. PLOS ONE, 17(2),

work page 2022

[4] [6]

This demonstrates that five candidates are sufficient to find the sublayer with the least importance score

Note that the accuracy remains constant when β is 5 or greater. This demonstrates that five candidates are sufficient to find the sublayer with the least importance score. Hence, we set β to 5 for all experiments. β 1 3 5 7 9 Step 7 74.14 74.19 74.19 74.19 74.19 Step 13 70.71 70.66 70.71 70.71 70.71 Table 6: Hyperparameter search for β. B.4 Implementation...

work page 2023

[5] [9]

Sparsegpt: Massive language models can be accurately pruned in one-shot

[Frantar and Alistarh, 2023] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In ICML,

work page 2023

[6] [10]

OPTQ: Accurate quantization for generative pre-trained transformers

[Frantar et al., 2023] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In ICLR,

work page 2023

[7] [11]

Lazyllm: Dynamic token pruning for efficient long context llm inference

[Fu et al., 2024] Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, and Mahyar Najibi. Lazyllm: Dynamic token pruning for efficient long context llm inference. arXiv,

work page 2024

[8] [12]

A frame- work for few-shot language model evaluation, 12

[Gao et al., 2023] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and ...

work page 2023