Pith. sign in

REVIEW 3 major objections 5 minor 80 references

Static and dynamic pruning, often treated as separate techniques, can be combined in one GPU execution framework; doing so yields 1.24x–1.37x average speedups over static-sparse baselines at matched perplexity.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:09 UTC pith:DDA5ASN7

load-bearing objection Solid kernel-level systems contribution with an honest limitations section; the matched-perplexity comparison needs a held-out split selection protocol before the end-to-end quality claim is credible. the 3 major comments →

arxiv 2607.21985 v1 pith:DDA5ASN7 submitted 2026-07-24 cs.DC cs.AIcs.ARcs.LG

Unified Static-Dynamic Pruning for Efficient LLM Inference

Jinhyeok Kim , Yejoon Lee , Jaeyoung Do This is my paper
classification cs.DC cs.AIcs.ARcs.LG
keywords static pruningdynamic pruningactivation sparsityLLM inferenceGPU kernelssparse matrix-vector multiplybitmap compressiondecode acceleration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SPDP sets out to prove that static and dynamic pruning—permanently removing redundant weights versus skipping input-dependent dead activations—are complementary rather than competing, and that a single GPU execution framework can realize the combination. The paper's core claim is that the two sparsities act on different dimensions of the weight matrix (rows vs columns), so their effects multiply: measured overlap stays close to the product of the two sparsity ratios, yielding effective sparsity up to 25% higher than static-only pruning at matched perplexity. To make that multiplicative benefit real on hardware, it contributes a column-addressable bitmap-compressed format and two co-designed kernels, one for token-by-token decode and one for prompt prefill, so a single compressed representation serves both phases. If the paper is right, small-batch LLM serving on memory-bandwidth-limited GPUs can get 1.24x–1.37x average kernel speedups (up to 2.51x) over static-sparse frameworks while preserving model quality—a direct advance on the speed–quality Pareto frontier of inference.

Core claim

Static pruning removes individual weights along rows; dynamic pruning skips weight columns whose activation is small for the current token. The paper's central claim is that these two patterns are nearly orthogonal, so their effective sparsity multiplies. To realize this, Tiled-CBC stores statically pruned weights in column-major micro-tiles with per-column metadata, and the HAD-SMBD decode step checks whether a column is active before unpacking its bitmap. That lets the decode kernel read only activation-relevant fragments with coalesced access, while a prefill kernel reuses the same format for Tensor-Core matrix multiplication. The authors report 1.24x–1.37x average kernel speedups over st

What carries the argument

Tiled-CBC (Tiled Column-wise Bitmap Compressed): a hierarchical, column-major format that packs statically pruned weights into 256×1 column tiles inside larger tiles, using uint64 bitmaps, a per-column offset/nonzero-count pair, and tile offsets so any column can be located and skipped at runtime. HAD-SMBD (Hybrid Activation-aware Dynamic Shared-Memory Bitmap Decoding): a two-phase decoder that first tests a column against the dynamic activation threshold, skips inactive columns entirely, then uses masked popcounts to unpack only active columns' values. The work these mechanisms do is to convert the theoretical product of static and dynamic sparsity into actual bandwidth savings during memor

Load-bearing premise

The load-bearing assumption is that static pruning and dynamic pruning select nearly independent sets of weights, so their sparsities multiply; if the same weight columns that static pruning removes are also the ones dynamic pruning skips for typical inputs, the combined sparsity collapses toward the larger of the two and the bandwidth saving disappears.

What would settle it

Run the combined-pruning pipeline on a diverse set of real prompts, count how many weight columns are removed by both static and dynamic masks, and compare with the product of the two sparsity ratios. If the average overlap is materially above the product (or near worst-case), the effective total sparsity falls below 1−(1−s_p)(1−s_d), and the claimed speedups would not survive on those inputs; the current evidence is a single deviation figure.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At matched perplexity, the joint method reaches up to 25% more total sparsity than static-only pruning, translating into fewer bytes transferred per generated token and lower energy per token.
  • Because the same Tiled-CBC weights feed both the decode and prefill kernels, serving stacks avoid duplicating or reformatting a separate dense or static-sparse copy of the model.
  • Threshold-based dynamic pruning methods can be swapped into the column-skipping interface with small changes, making the framework independent of any single dynamic-pruning rule.
  • The speedup is bounded by the memory-bound decode phase and by nonsparse layers: even large kernel wins become modest end-to-end TPOT gains, so the main benefit is latency and energy at equal quality rather than raw FLOPs.
  • The design is deliberately specialized to moderate-sparsity LLM decode workloads; the paper's own evaluation on very sparse non-LLM matrices shows it does not replace general sparse linear algebra.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The near-independence of static and dynamic masks is the soft spot: both the static and dynamic criteria are activation-magnitude based, yet the paper reports overlap slightly below random. An obvious stress test is to rerun the overlap measurement on out-of-distribution prompts; if overlap rises, the multiplicative sparsity gain shrinks exactly where robustness matters.
  • The same column-addressable layout could be extended to batched dynamic pruning, which the paper leaves for future work; if overlap stays low across batch items, per-token savings should compound with batch size instead of being amortized away.
  • The paper's storage formulas imply that quantization and sparsity do not combine multiplicatively because bitmap and column metadata stay in full precision; coarser static patterns would be needed to stack 4-bit weights with pruning while keeping metadata overhead low.
  • The one-format-serves-both-phases design suggests a clean deployment story for disaggregated serving: run dense prefill and switch to Tiled-CBC compressed weights only in decode, a configuration the paper notes as practical.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces SPDP, a unified sparse-inference framework that combines unstructured static weight pruning (Wanda) with input-dependent dynamic activation pruning (TEAL) for LLM decoding. It proposes the Tiled-CBC compressed format and two kernels: a CUDA-core spMspV decode kernel with HAD-SMBD and a Tensor-Core SpMM prefill kernel. Evaluations against cuBLAS, cuSPARSE, Sputnik, SparTA, Flash-LLM, and SpInfer on A10G/L4/L40S report 1.24–1.37× average kernel speedups over SpInfer and up to 25% higher total sparsity at matched perplexity. The authors also disclose regimes where SPDP is not superior: the prefill kernel does not beat dense cuBLAS (Fig 12) and SPDP is not competitive on extreme-sparsity SuiteSparse matrices (Fig 13).

Significance. If the empirical claims hold, SPDP is a useful step for small-batch LLM serving: it shows a single format can serve both decode-time spMspV with dynamic column skipping and prefill SpMM, and the kernel measurements are on realistic LLM weight shapes. The paper is unusually honest: it measures with Nsight, releases source code, and explicitly states the boundaries of its applicability. However, the headline quality-sparsity claim rests on a test-set selection protocol that needs correction, and the independence assumption between SP and DP is under-validated.

major comments (3)
  1. [§5.5, Fig 10] The matched-perplexity comparison is the load-bearing support for the '25% higher sparsity' claim. The text states: 'For each configuration, SPDP determines the optimal combination of SP and DP that minimizes perplexity.' If the configuration that minimizes WikiText PPL is selected on the same WikiText split used for reporting, the reported PPL is a minimum over a grid, whereas SpInfer/Wanda is a fixed method with no equivalent tuning. This asymmetry can inflate the sparsity advantage at equal PPL. Please select SP/DP ratios on a validation split (or fix a policy) and report PPL on a held-out test set; also show the full SP/DP grid as a scatter so the reader can see the trade-off.
  2. [§3.2.2, Fig 3] The independence of SP and DP is assumed via the product formula 1-(1-sp)(1-dp). Both Wanda and TEAL score columns by activation magnitude, so positive overlap (correlation) is a real risk; if overlap > sp×dp, the effective total sparsity falls below the formula and the bandwidth saving shrinks. Fig 3 shows only a color map without error bars, scale, or statistical test, and the direction 'slightly below' is counterintuitive. Please report per-layer/token overlap distributions, correlations, and confidence intervals; if the deviation is truly negative, explain the mechanism.
  3. [§5.3, Fig 8] The kernel-level comparison does not state the sparsity configuration used for each baseline. If SPDP uses (SP=30%, DP=30%) while SpInfer is evaluated at SP=30%, the comparison is at different total sparsity; if SpInfer is at SP=50%, the comparison is at approximately equal total sparsity but depends on the SP-DP overlap assumption. Please report the exact SP/DP values for every marker and, ideally, show SPDP and SpInfer at the same total sparsity and at the same SP to separate the contribution of DP.
minor comments (5)
  1. [§3.2.3, Eq (3)] Eq (3) appears algebraically inconsistent with Eq (2): under column skipping both the weight and activation terms should scale by (1-s), so the arithmetic intensity is unchanged (the benefit is lower total bytes/latency, not higher CI). As written, only the weight term is divided by (1-s), which artificially inflates CI. Correct or clarify the derivation.
  2. [Fig 3] Add a colorbar and define the sign convention of the deviation. 'Slightly below' should be quantified with per-layer ranges or box plots.
  3. [Fig 10] The axis labels (25%, 30%, 40%, 50%, 55%, 65%) are cryptic. Clarify whether these are total sparsity, SP, or DP, and annotate the SP/DP split for each point.
  4. [Abstract] Typo in the abstract: 'matching. perplexity' should be 'matching perplexity'.
  5. [§5.5] The 'performance-per-watt' improvement is inferred from reduced data movement rather than directly measured. State this explicitly, or report power/energy measurements if available.

Circularity Check

1 steps flagged

Partial circularity in the matched-perplexity claim: the SP/DP split is selected by minimizing the same WikiText perplexity that is then reported as the method's quality; kernel-level speedups remain independent external measurements.

specific steps
  1. fitted input called prediction [Section 5.5, Figures 10 and 11; Section 5.1 setup]
    "For each configuration, SPDP determines the optimal combination of SP and DP that minimizes perplexity."

    The quality metric in the central claim is WikiText perplexity, reported in Figures 10 and 11. The SP/DP split is selected by minimizing exactly that same test-set perplexity on the same benchmark used for the matched-perplexity comparison. Therefore the reported 'matching perplexity with up to 25% higher sparsity' is the minimum over a search grid, not the expected perplexity of a fixed method. SpInfer/Wanda is evaluated at fixed sparsity with no equivalent test-set split tuning, so the quality half of the claim is an optimized quantity presented as a predictive result, making the matched-PPL comparison asymmetric and partly an artifact of test-set selection.

full rationale

The paper is largely a systems paper with externally measured kernel and end-to-end results against cuBLAS, cuSPARSE, Sputnik, SparTA, Flash-LLM, and SpInfer, so the speedup claims do not reduce to their inputs. The overlap-independence analysis is empirical (Figure 3), not definitional. There are no load-bearing self-citations and no imported uniqueness theorems; HAD-SMBD is explicitly an extension of SpInfer's SMBD and is not used as proof. The one circular element is the quality comparison: the SP/DP split is tuned on the same WikiText perplexity used to report the matched-PPL result, so that comparison is an oracle selection rather than an independent prediction. Kernel-level throughput, TPOT at fixed sparsity ratios, and the profiling results remain independently measured, which keeps the overall circularity partial rather than total.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The system contribution rests on a small set of calibrated parameters (DP thresholds, SP/DP splits, tile sizes) and on the domain assumption that static and dynamic pruning patterns are independent. No new physical entities are postulated; Tiled-CBC and HAD-SMBD are software artifacts. The most fragile item is the overlap-independence assumption, which the paper supports with a single figure lacking error analysis.

free parameters (4)
  • DP sparsity ratio s_d = 0.30 in kernel sweeps; 0.25-0.50 in end-to-end
    Target activation sparsity chosen by the authors; the claimed DP benefit is a function of this chosen target (Sec 5.1, Fig 7).
  • Layer-wise DP thresholds (TEAL ECDF quantiles) = Calibrated on 300 Alpaca samples
    Per-layer activation-magnitude thresholds fitted offline to achieve the target DP ratio (Sec 5.1); these are fitted to data and then treated as fixed at inference.
  • SP/DP split at each total sparsity = e.g., best config at 65% total is 30% SP + 50% DP
    Chosen per configuration to minimize perplexity (Sec 5.5); the matched-quality claims partly encode this search.
  • Kernel tile sizes TILE_M=256, TILE_K=16 = 256 x 16
    Design parameters stated in Sec 4.2.3; presumably tuned rather than derived, and they shape all kernel-level results.
axioms (5)
  • standard math Roofline model P = min(P_peak, CI x B_mem) with homogeneous memory traffic
    Standard performance model used as motivation (Sec 3.1); acceptable as background.
  • domain assumption Static and dynamic pruning act independently; overlap ~ s_p x s_d
    Load-bearing for Eq (3) and the expectation that total sparsity 1-(1-s_p)(1-s_d) holds (Sec 3.2.2, Fig 3). Both Wanda and TEAL are driven by activation-magnitude statistics, so correlation is plausible; the paper provides only Fig 3 without error analysis.
  • domain assumption TEAL ECDF thresholds calibrated on Alpaca transfer to arbitrary inference inputs
    Runtime DP ratio is assumed to track the calibrated target (Sec 5.1); if actual activation sparsity deviates from 30%, kernel gains change.
  • domain assumption Moderate static sparsity (30-50%) is the regime where model quality permits pruning
    Frames the evaluation sweep (Sec 5.3) and the response to SuiteSparse results (Sec 6).
  • domain assumption At decode (N=1) the activation vector bytes are negligible; only weight bytes count
    XTile is loaded dense (Sec 4.2); the bandwidth-saving claim counts weight bytes only. With dense X, the true CI is approximately invariant to s once both FLOPs and weight bytes shrink by (1-s).
invented entities (2)
  • Tiled-CBC compressed format no independent evidence
    purpose: Column-addressable sparse layout enabling runtime DP column skipping while preserving static compression
    Software artifact; falsifiable only through the claimed speedups and the shipped code, which is not independently verified here.
  • HAD-SMBD hybrid activation-aware bitmap decoder no independent evidence
    purpose: Two-phase decode of bitmap-compressed tiles with activation-aware column skipping on CUDA cores
    Kernel mechanism built on SpInfer's SMBD; no external handle beyond the paper's measurements.

pith-pipeline@v1.3.0-alltime-deepseek · 24167 in / 25767 out tokens · 249285 ms · 2026-08-01T06:09:57.529771+00:00 · methodology

0 comments
read the original abstract

The increasing deployment of large language models (LLMs) has magnified the computational and memory bottlenecks of autoregressive decoding, where low compute intensity and bandwidth-bound kernels dominate inference cost. Weight pruning offers a promising remedy, but existing methods remain confined to either static pruning (SP), which permanently removes redundant weights but lacks adaptivity, or dynamic pruning (DP), which adapts to input sparsity but introduces runtime irregularity. This paper presents SPDP, a unified sparse-inference framework that integrates unstructured SP with input-adaptive DP for efficient LLM inference on GPUs. SPDP co-designs a new Tiled-Column-wise Bitmap Compressed (Tiled-CBC) format and two complementary GPU kernels: (1) a CUDA-core spMspV kernel featuring Hybrid Activation-aware Dynamic Shared-Memory Bitmap Decoding (HAD-SMBD) for fine-grained, runtime activation skipping, and (2) a Tensor-Core SpMM kernel optimized for prefill computation. This joint format-kernel design harmonizes static and dynamic sparsity, maintaining bandwidth-efficient memory access and high compute intensity under both phases of LLM inference. Comprehensive evaluations on inference-optimized GPUs demonstrate that SPDP achieves 1.24x-1.37x average speedup (up to 2.51x) over state-of-the- art sparse frameworks such as SpInfer, while matching. perplexity with up to 25% higher sparsity. SPDP advances the inference efficiency-quality Pareto frontier, showing that unified static-dynamic pruning can deliver substantial throughput and performance-per-watt improvements in large-scale LLM serving

Figures

Figures reproduced from arXiv: 2607.21985 by Jaeyoung Do, Jinhyeok Kim, Yejoon Lee.

Figure 1
Figure 1. Figure 1: Roofline analysis of matrix multiplication on an [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Perplexity comparison across pruning strategies. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Deviation of measured overlapping sparsity from [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: (a) Tiled-CBC stores nonzeros within each GTile in column-major order using a hierarchical layout. Each GTile has [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Asynchronous pipeline design for SPDP-spMspV and SPDP-SpMM kernels. Both kernels use HAD-SMBD for efficient decompression and fine-grained asynchronous group management to fully overlap stages without stalls. The SPDP-SpMM kernel adds a Layout Alignment stage for Tensor Core execution, carefully scheduled to minimize overhead. Activation-aware Shared Memory Bitmap Decoding (HAD-SMBD). Since these tasks are… view at source ↗
Figure 6
Figure 6. Figure 6: Accuracy on downstream tasks and compatibility with quantization. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Kernel speedup on A10G GPU over cuBLAS_TC [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Kernel-level performance comparison of the [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Kernel profiling results on the A10G GPU. [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Perplexity vs. TPOT on A10G, L4, and L40S GPUs. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Sparsity ratio vs. TPOT speedup across different GPU architectures. SPDP reduces TPOT while maintaining reasonable [PITH_FULL_IMAGE:figures/full_fig_p012_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prefill kernel comparison on A10G under SP=30%. [PITH_FULL_IMAGE:figures/full_fig_p012_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: SuiteSparse evaluation. SPDP targets moderate [PITH_FULL_IMAGE:figures/full_fig_p012_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

80 extracted references · 1 canonical work pages

  1. [1]

    Marah Abdin et al. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv:2404.14219 [cs.CL] https://arxiv.org/abs/ 2404.14219

    Pith/arXiv arXiv 2024

  2. [2]

    Deepak Bhaskar Acharya, Karthigeyan Kuppan, and B. Divya. 2025. Agentic AI: Autonomous Intelligence for Complex Goals—A Comprehensive Survey.IEEE Access13 (2025), 18912–18936. https://doi.org/10.1109/ACCESS.2025.3532853

    arXiv 2025

  3. [3]

    Sandhini Agarwal et al . 2025. gpt-oss-120b & gpt-oss-20b Model Card. arXiv:2508.10925 [cs.CL] https://arxiv.org/abs/2508.10925

    Pith/arXiv arXiv 2025

  4. [4]

    Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2024. Fluctuation- based adaptive structured pruning for large language models. InAAAI (AAAI’24/IAAI’24/EAAI’24). AAAI Press, Article 1212, 9 pages. https://doi.org/ 10.1609/aaai.v38i10.28960

    doi:10.1609/aaai.v38i10.28960 2024

  5. [5]

    Jinze Bai et al. 2023. Qwen Technical Report. arXiv:2309.16609 [cs.CL] https: //arxiv.org/abs/2309.16609

    Pith/arXiv arXiv 2023

  6. [6]

    Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. 2023. Rethinking the Role of Scale for In-Context Learn- ing: An Interpretability-based Case Study at 66 Billion Scale. InACL, Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 11833...

    doi:10.18653/v1/2023.acl- 2023

  7. [7]

    Mark Chen et al. 2021. Evaluating Large Language Models Trained on Code. ArXivabs/2107.03374 (2021)

    Pith/arXiv arXiv 2021

  8. [8]

    Yuli Chen, Bo Cheng, Jiale Han, Yingying Zhang, Yingting Li, and Shuhao Zhang

  9. [9]

    Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christo- pher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems.ArXivabs/2110.14168 (2021)

    Pith/arXiv arXiv 2021

  10. [10]

    Davis and Yifan Hu

    Timothy A. Davis and Yifan Hu. 2011. The university of Florida sparse matrix collection.ACM Trans. Math. Softw.38, 1, Article 1 (Dec. 2011), 25 pages. https: //doi.org/10.1145/2049662.2049663

    arXiv 2011

  11. [11]

    Harry Dong, Beidi Chen, and Yuejie Chi. 2024. Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation. InCoLM

    2024

  12. [12]

    Zhen Du, Ying Liu, Ninghui Sun, Huimin Cui, Xiaobing Feng, and Jiajia Li

  13. [13]

    Ruibo Fan, Wei Wang, and Xiaowen Chu. 2024. DTC-SpMM: Bridging the Gap in Accelerating General Sparse Matrix Multiplication with Tensor Cores. InASPLOS (La Jolla, CA, USA)(ASPLOS ’24). Association for Computing Machinery, New York, NY, USA, 253–267. https://doi.org/10.1145/3620666.3651378

    arXiv 2024

  14. [14]

    SRSparse: Generating Codes for High-Performance Sparse Matrix-Vector Semiring Computations.ACM Trans. Archit. Code Optim.22, 2, Article 69 (July 2025), 26 pages. https://doi.org/10.1145/3722114

    work page doi:10.1145/3722114 2025

  15. [15]

    Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. 2025. MaskLLM: learnable semi-structured sparsity for large language models. InNeurIPS(Vancouver, BC, Canada)(NeurIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article 248, 23 pages

    2025

  16. [16]

    Ruibo Fan, Xiangrui Yu, Peijie Dong, Zeyu Li, Gu Gong, Qiang Wang, Wei Wang, and Xiaowen Chu. 2025. SpInfer: Leveraging Low-Level Sparsity for Efficient Large Language Model Inference on GPUs. InEuroSys(Rotterdam, Netherlands)(EuroSys ’25). Association for Computing Machinery, New York, NY, USA, 243–260. https://doi.org/10.1145/3689031.3717481

    arXiv 2025

  17. [17]

    Jared Fernandez, Clara Na, Vashisth Tiwari, Yonatan Bisk, Sasha Luccioni, and Emma Strubell. 2025. Energy Considerations of Large Language Model Inference and Efficiency Optimizations. arXiv:2504.17674 [cs.CL] https://arxiv.org/abs/ 2504.17674

    Pith/arXiv arXiv 2025

  18. [18]

    Marco Federici, Davide Belli, Mart Van Baalen, Amir Jalalirad, Andrii Skliar, Bence Major, Markus Nagel, and Paul Whatmough. 2025. Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking. InMLSys

    2025

  19. [19]

    Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. 2020. Sparse GPU Kernels for Deep Learning. InSC

    2020

  20. [20]

    Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot. InICML (Proceedings of Machine Learning Research), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engel- hardt, Sivan Sabato, and Jonathan Scarlett (Eds.), Vol. 202. PMLR, 10323–10337. https://proceedings.mlr.press/v202/frantar23a.html

    2023

  21. [21]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

    Pith/arXiv arXiv 2025

  22. [22]

    Aaron Grattafiori et al . 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  23. [23]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Un- derstanding. InICLR

    2021

  24. [24]

    Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. Learning both weights and connections for efficient neural networks. InNeurIPS(Montreal, Canada) (NeurIPS’15). MIT Press, Cambridge, MA, USA, 1135–1143

    2015

  25. [25]

    Hugging Face. n.d.. Huggingface Transformers. Retrieved 2025-10-29 from https://huggingface.co/docs/transformers/index

    2025

  26. [26]

    Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph (Seffi) Naor, and Daniel Soudry. 2021. Accelerated sparse neural training: a provable and effi- cient method to find N:M transposable masks. InNeurIPS (NeurIPS ’21). Curran Associates Inc., Red Hook, NY, USA, Article 1614, 13 pages

    2021

  27. [27]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven...

    Pith/arXiv arXiv 2024

  28. [28]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card.arXiv preprint arXiv:2412.16720(2024)

    Pith/arXiv arXiv 2024

  29. [29]

    Donghyeon Joo, Helya Hosseini, Ramyad Hadidi, and Bahar Asgari. 2025. Cor- uscant: Co-Designing GPU Kernel and Sparse Tensor Core to Advocate Unstruc- tured Sparsity in Efficient LLM Inference. InProceedings of the 58th IEEE/ACM International Symposium on Microarchitecture (MICRO ’25). Association for Com- puting Machinery, New York, NY, USA, 232–245. htt...

    doi:10.1145/3725843 2025

  30. [30]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2025. A Survey on Large Language Models for Code Generation.ACM Trans. Softw. Eng. Methodol.(July 2025). https://doi.org/10.1145/3747588 Just Accepted

    doi:10.1145/3747588 2025

  31. [31]

    Donghyun Lee, Jaeyong Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini

  32. [32]

    Fredrik Kjolstad, Shoaib Kamil, Stephen Chou, David Lugato, and Saman Amaras- inghe. 2017. The tensor algebra compiler.Proc. ACM Program. Lang.1, OOPSLA, Article 77 (Oct. 2017), 29 pages. https://doi.org/10.1145/3133901

    doi:10.1145/3133901 2017

  33. [33]

    Zhonggen Li, Xiangyu Ke, Yifan Zhu, Yunjun Gao, and Yaofeng Tu. 2025. HC- SpMM: Accelerating Sparse Matrix-Matrix Multiplication for Graphs with Hybrid GPU Cores . InICDE. IEEE Computer Society, Los Alamitos, CA, USA, 501–514. https://doi.org/10.1109/ICDE65448.2025.00044

    arXiv 2025

  34. [34]

    Hongyi Liu, Rajarshi Saha, Zhen Jia, Youngsuk Park, Jiaji Huang, Shoham Sabach, Yu-Xiang Wang, and George Karypis. 2025. PROXSPARSE: REGULARIZED LEARNING OF SEMI-STRUCTURED SPARSITY MASKS FOR PRETRAINED LLMS. InICML

    2025

  35. [35]

    Min Li, Yulong Ao, and Chao Yang. 2020. Adaptive SpMV/SpMSpV on GPUs for input vectors of varied sparsity.IEEE Transactions on Parallel and Distributed Systems32, 7 (2020), 1842–1853

    2020

  36. [36]

    Lawrence Liu, Alexander Liu, Mengdi Wang, Tuo Zhao, and Lin F. Yang. 2026. ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Fac- torization. InICLR

    2026

  37. [37]

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, An- shumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen

  38. [38]

    James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athi- waratkun. 2025. Training-Free Activation Sparsity in Large Language Models. InICLR

    2025

  39. [39]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. LLM-pruner: on the struc- tural pruning of large language models. InNeurIPS(New Orleans, LA, USA) (NeurIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 950, 19 pages

    2023

  40. [40]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer Sentinel Mixture Models. InICLR

    2017

  41. [41]

    Seyed Iman Mirzadeh, Keivan Alizadeh-Vahid, Sachin Mehta, Carlo C del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar

  42. [42]

    Mahoney, and Yaoqing Yang

    Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W. Mahoney, and Yaoqing Yang. 2024. AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models. InNeurIPS (NeurIPS ’24)

    2024

  43. [43]

    2023.cuBLAS Documentation

    NVIDIA. 2023.cuBLAS Documentation. Retrieved 2025-10-29 from https://docs. nvidia.com/cuda/cublas/index.html

    2023

  44. [44]

    NVIDIA. 2023. cuSPARSE Library. Retrieved 2025-10-29 from https://docs. nvidia.com/cuda/cusparse/index.html

    2023

  45. [45]

    NVIDIA. 2024. Nsight Compute. Retrieved 2025-10-29 from https://developer. nvidia.com/nsight-compute

    2024

  46. [46]

    ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models. InICLR

  47. [47]

    Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. 2021. Accelerating sparse deep neural networks.arXiv preprint arXiv:2104.08378(2021)

    Pith/arXiv arXiv 2021

  48. [48]

    2022.NVIDIA A100 Tensor Core GPU Architec- ture

    NVIDIA Corporation. 2022.NVIDIA A100 Tensor Core GPU Architec- ture. Technical Report. NVIDIA Corporation. Retrieved 2025-10-29 from https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia- ampere-architecture-whitepaper.pdf

    2022

  49. [49]

    2023.NVIDIA L4 Tensor Core GPU

    NVIDIA Corporation. 2023.NVIDIA L4 Tensor Core GPU. Technical Report. NVIDIA Corporation. Retrieved 2025-10-29 from https://www.nvidia.com/en- zz/data-center/l4/

    2023

  50. [50]

    2023.NVIDIA L40S

    NVIDIA Corporation. 2023.NVIDIA L40S. Technical Report. NVIDIA Corporation. Retrieved 2025-10-29 from https://www.nvidia.com/en-zz/data-center/l40s/

    2023

  51. [51]

    NVIDIA. 2024. Nsight Systems. Retrieved 2025-10-29 from https://developer. nvidia.com/nsight-systems

    2024

  52. [52]

    2021.NVIDIA A10 Tensor Core GPU

    NVIDIA Corporation. 2021.NVIDIA A10 Tensor Core GPU. Technical Report. NVIDIA Corporation. Retrieved 2025-10-29 from https://www.nvidia.com/en- zz/data-center/products/a10-gpu/

    2021

  53. [53]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Infer- ence Using Phase Splitting. InISCA. 118–132. https://doi.org/10.1109/ISCA59077. 2024.00019

    arXiv 2024

  54. [54]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.Journal of Machine Learning Research21, 140 (2020), 1–67. http://jmlr.org/papers/v21/20-074.html

    2020

  55. [55]

    Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A Con- versational Question Answering Challenge.Transactions of the Association for Computational Linguistics7 (2019), 249–266. https://doi.org/10.1162/tacl_a_00266

    doi:10.1162/tacl_a_00266 2019

  56. [56]

    Patrik Okanovic, Grzegorz Kwasniewski, Paolo Sylos Labini, Maciej Besta, Flavio Vella, and Torsten Hoefler. 2024. High performance unstructured spmm compu- tation using tensor cores. InSC. IEEE, 1–14

    2024

  57. [57]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human fee...

    2022

  58. [58]

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2024. A Simple and Effective Pruning Approach for Large Language Models. InICLR

    2024

  59. [59]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model. Retrieved 2025-10-29 from https: //github.com/tatsu-lab/stanford_alpaca

    2023

  60. [60]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288(2023)

    Pith/arXiv arXiv 2023

  61. [61]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. InNeurIPS (NeurIPS ’23)

    2023

  62. [62]

    Chenyang Song, Xu Han, Zhengyan Zhang, Shengding Hu, Xiyu Shi, Kuai Li, Chen Chen, Zhiyuan Liu, Guangli Li, Tao Yang, and Maosong Sun. 2025. ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models. InCOLING, Owen Rambow, Leo Wanner, Marianna Apid- ianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (E...

    2025

  63. [63]

    An Yang et al. 2024. Qwen2 Technical Report. arXiv:2407.10671 [cs.CL] https: //arxiv.org/abs/2407.10671

    Pith/arXiv arXiv 2024

  64. [64]

    An Yang et al. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https: //arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  65. [65]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InICLR

    2023

  66. [66]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. InNeurIPS(Long Beach, California, USA)(NeurIPS’17). Curran Asso- ciates Inc., Red Hook, NY, USA, 6000–6010

    2017

  67. [67]

    Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xi- afei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. 2023. Flash-LLM: En- abling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity.Proc. VLDB Endow.17, 2 (Oct. 2023), 211–224. https://doi.org/10.14778/3626292.3626303

    arXiv 2023

  68. [68]

    Yu, and Jiawei Zhang

    Haopeng Zhang, Philip S. Yu, and Jiawei Zhang. 2025. A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models.ACM Comput. Surv.57, 11, Article 277 (June 2025), 41 pages. https://doi.org/10.1145/ 3731445

    2025

  69. [69]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre-trained Transformer Language Models. arXiv:2205.01068 [cs.CL]...

    Pith/arXiv arXiv 2022

  70. [70]

    Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. 2024. Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models. InICLR

    2024

  71. [71]

    Zihao Ye, Ruihang Lai, Junru Shao, Tianqi Chen, and Luis Ceze. 2023. Sparsetir: Composable abstractions for sparse compilation in deep learning. InASPLOS. 660–678

    2023

  72. [72]

    Ruokai Yin, Yuhang Li, Donghyun Lee, and Priyadarshini Panda. 2025. DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs. arXiv:2506.20194 [cs.LG] https://arxiv.org/abs/2506.20194

    arXiv 2025

  73. [73]

    Ningxin Zheng, Bin Lin, Quanlu Zhang, Lingxiao Ma, Yuqing Yang, Fan Yang, Yang Wang, Mao Yang, and Lidong Zhou. 2022. SparTA: Deep-Learning Model Sparsity via Tensor-with-Sparsity-Attribute. InOSDI. USENIX Association, Carls- bad, CA, 213–232. https://www.usenix.org/conference/osdi22/presentation/ zheng-ningxin

    2022

  74. [74]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. InOSDI. USENIX Asso- ciation, Santa Clara, CA, 193–210. https://www.usenix.org/conference/osdi24/ presentation/zhong-yinmin

    2024

  75. [75]

    Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. 2021. Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch. InICLR

    2021

  76. [76]

    Haisha Zhao, San Li, Jiaheng Wang, Chunbao Zhou, Jue Wang, Zhikuang Xin, Shunde Li, Zhiqiang Liang, Zhijie Pan, Fang Liu, Yan Zeng, Yangang Wang, and Xuebin Chi. 2025. Acc-SpMM: Accelerating General-purpose Sparse Matrix- Matrix Multiplication with GPU Tensor Cores. InPPoPP(Las Vegas, NV, USA) (PPoPP ’25). Association for Computing Machinery, New York, NY...

    arXiv 2025

  77. [77]

    Haizhong Zheng, Xiaoyan Bai, Xueshen Liu, Zhuoqing Mao, Beidi Chen, Fan Lai, and Atul Prakash. 2024. Learn To be Efficient: Build Structured Sparsity in Large Language Models. InNeurIPS (NeurIPS ’24)

    2024

  78. [2023]

    InICML (Proceedings of Machine Learning Research), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.), Vol

    Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. InICML (Proceedings of Machine Learning Research), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.), Vol. 202. PMLR, 22137–22176. https://proceedings.mlr.press/v202/liu23am.html

  79. [2024]

    CATS: Context-Aware Thresholding for Sparsity in Large Language Models. InCoLM

  80. [2025]

    DLP: Dynamic Layerwise Pruning in Large Language Models. InICML