Pith. sign in

REVIEW 4 major objections 5 minor 69 references

To prune a language model safely, CausalGate does not observe activations—it intervenes: zero out each module's output, measure the shift in the final token distribution, and distill that causal ranking into a static gate.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 12:23 UTC pith:CQLVAOKD

load-bearing objection A sensible intervention-based pruning idea with a competent pipeline, but the empirical evidence as presented doesn't support the headline claims due to evaluation leakage and missing controls. the 4 major comments →

arxiv 2607.22720 v2 pith:CQLVAOKD submitted 2026-07-21 cs.LG cs.CLcs.CVstat.ML

CausalGate: Causal Importance Distillation for Transformer Module Pruning

Kiran Nair , Smriti Regmi , Rodrigue Rizk This is my paper
classification cs.LG cs.CLcs.CVstat.ML
keywords causal interventiontransformer module pruninglayer skippingKL divergencestatic gatesLLM inference accelerationattention and MLP sub-layerscompute-efficient inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CausalGate argues that the right way to decide which transformer sub-modules can be safely skipped is to intervene, not observe: zero out each attention or MLP module's output, run the model, and measure how much the predicted next-token distribution shifts, quantified by KL divergence. The paper contends that this causal score captures non-linear structural computations that activation-magnitude or similarity-based heuristics miss, so it should be the basis for structural pruning. It then distills the intervention scores into a fixed set of static per-module gates, trained once with EMA smoothing and a pairwise ranking loss, so inference is a straight-line execution with no routing overhead. Across language modeling and commonsense reasoning benchmarks on three model families, CausalGate is claimed to hold up better than dynamic routing and layer-skipping baselines at 5–40% module removal, with measured 1.08–1.20x latency speedups at 10–20% removal.

Core claim

CausalGate's central claim is that a transformer module's true global importance is its causal influence on the final logit distribution: zero out the module's residual output, run the model, and take the KL divergence between the intervened and original token distributions. The paper argues this intervention-based score is a better pruning criterion than observational heuristics such as activation magnitudes or hidden-state similarity, because it captures non-linear structural computations. It then shows that these scores can be distilled into a fixed ranking of static scalar gates—one per attention or MLP sub-module—trained with EMA-smoothed targets and a pairwise ranking loss, so that inf

What carries the argument

The central object is the causal importance score Δ_i = D_KL(p(y|x, m_i→0) ∥ p(y|x)): the KL divergence between the final logit distribution when module i's residual output is forced to zero and the original distribution. This score is computed per module over a calibration batch, normalized and EMA-smoothed to stabilize rankings, and then encoded into learned scalar gates g_i = σ(θ_i); a pairwise ranking loss forces the gates to respect the intervention-derived ordering. At inference, modules above a top-K cut are kept, the rest skipped.

Load-bearing premise

The load-bearing premise is that one-at-a-time interventions over a calibration batch produce a fixed input-independent ranking that stays valid when many modules are skipped together.

What would settle it

A concrete test: compute CausalGate's individual module KL scores; then, for a given budget, skip the lowest-ranked modules jointly and measure the actual output KL against the unpruned model. If joint removal causes much larger distribution shifts than the individual scores predict—or if a ranking explicitly computed from leave-K-out interventions gives better downstream accuracy at the same budget—the global additive ranking assumption is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If causal importance is the right criterion, static module selection can replace dynamic routers, eliminating branch-prediction and routing overhead while getting comparable or better quality at matched compute budgets.
  • Treating attention and MLP blocks as independently prunable units gives finer-grained trade-offs than uniform layer dropping, because importance is concentrated in a subset of MLPs while many later-layer attentions are near-redundant.
  • The learned gate ranking (Spearman 0.78 against intervention scores) means expensive per-module zeroing is needed only once, at calibration; deployment runs the frozen mask.
  • Measured speedups of 1.08–1.20x at 10–20% removal indicate the theoretical FLOP savings convert to real latency gains on current hardware.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves input-specific importance untested; a natural extension is to compare the global gate ranking against input-conditioned rankings to quantify when the fixed mask sacrifices accuracy for simplicity.
  • Because interventions are applied one-at-a-time, interaction effects among simultaneously skipped modules are never measured; a leave-multiple-out diagnostic would reveal whether the additive ordering assumption holds at aggressive budgets.
  • The calibration cost (one forward pass per module, per batch) is not reported; a deployment-focused extension would measure total calibration wall-clock and whether fewer intervention steps could produce the same ranking.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CausalGate, a post-hoc structural pruning method for Transformer LLMs. During a calibration phase, each attention and MLP sub-module is individually zeroed out, and its importance is measured by the KL divergence between the resulting final-token distribution and that of the intact model (Eq. 1). These intervention scores are normalized, floored, and smoothed with an EMA, then distilled into a single learnable scalar gate per module using MSE, pairwise ranking, sparsity, and language-modeling losses (Eqs. 2–10). At inference, a static top-K mask is derived from the learned gate values (Eq. 12), so no per-input routing is needed. Experiments on TinyLlama-1.1B, Qwen2.5-3B, and Llama-3.1-8B report perplexity on WikiText-2/C4/PTB and accuracy on HellaSwag, PIQA, CSQA, and WinoGrande, along with measured latency speedups. The central claim is that one-at-a-time intervention-based KL importance defines a 'true global' module hierarchy that, once distilled into static gates, consistently outperforms dynamic-routing and layer-skipping baselines at 5–40% module removal.

Significance. If the empirical claims were valid, the paper would make a useful contribution to LLM structural pruning: it proposes a conceptually clear intervention-based criterion, operates at the finer attention/MLP sub-layer level, and demonstrates real hardware speedups at 10–20% removal. The idea of distilling costly intervention scores into static scalar gates is attractive and directionally plausible. However, the current evidence is not sufficient to support the central claims. The evaluation conflates training and test data (WikiText-2 is used both for calibration/gate training and for evaluation), hyperparameters are selected on the same test metric, the headline assertion of consistent improvement is contradicted by multiple rows of Table 1, and the key additivity assumption behind the static top-K mask is never tested. These are not cosmetic issues; they directly undermine the paper's main conclusions. The paper does not provide code, error bars, or ablations that would allow a reader to separate genuine gains from calibration leakage.

major comments (4)
  1. [§4, Table 4 and §3, Eq. (10)] Training/evaluation leakage is explicit: Table 4 lists WikiText-2 as both 'Training + Evaluation,' and the CausalGate objective (Eq. 10) includes an LM loss λ_LM L_LM that is trained on calibration data, which is evidently WikiText-2. The reported WikiText-2 perplexities in Table 1 are therefore not held-out measurements. In addition, Table 3 selects hyperparameters using WikiText-2 perplexity at the 10% removal budget, so the WikiText-2 column is further optimistically biased by model selection on the test set. The C4 and commonsense results partially mitigate this, but the central demonstration of quality preservation relies heavily on the WikiText-2 column, and the paper does not report a valid held-out language-modeling evaluation.
  2. [§3, Eq. (1) and Eq. (12)] The central methodological assumption is that the marginal, one-at-a-time intervention scores of Eq. (1) form an additive, order-stable importance hierarchy that remains valid when many modules are zeroed simultaneously to form the static top-K mask of Eq. (12). Nothing in the paper tests this. At 20% removal, 9 of 44 modules are skipped together; module-module interactions can then easily reorder marginal importance. The only limitation acknowledged in §5 is input-specific variability, not interaction effects. Without a joint-removal experiment comparing the marginal ranking against greedy or jointly optimized rankings, the paper's claim that the learned gates encode the 'true global computational importance' is unsupported.
  3. [Table 1] The abstract claims CausalGate 'consistently outperforms' all baselines, but Table 1 contains counterexamples even at the lowest budget. At 5% removal, CALM Hidden-State achieves PIQA 0.734 vs CausalGate 0.723; at 30%, CALM (Softmax) and CALM (Hidden-State) both achieve PIQA 0.600 vs CausalGate 0.540; at 40%, CALM Hidden-State achieves WikiText PPL 1472 vs CausalGate 1365 and PIQA 0.570 vs 0.500. These are not matched with error bars, and the single omnibus Wilcoxon test over task-wise profiles does not establish 'consistently' superior behavior. Moreover, the baselines are not obviously matched: CALM and MoD are dynamic, per-input policies, while CausalGate removes the same modules for every input, so the comparison at a fixed 'architectural savings tier' is not apples-to-apples and the paper does not specify how the tiers were enforced for dynamic baselines.
  4. [§4, Experimental Setup] The paper reports no variance or replication information for any number in Table 1 or Table 2. A single run on a single V100 with no seeds, no confidence intervals, and no code release makes it impossible to assess whether the reported differences are meaningful. Given that Table 3 shows small perplexity differences across many hyperparameter settings (e.g., 18.44 vs 18.45), noise may be comparable to the reported improvements. The lack of reproducibility artifacts is a serious limitation for an empirical paper whose core contribution is a specific performance comparison.
minor comments (5)
  1. [Appendix C, Figure 9] The notation P(Y | do(M3 = 0), X) is not a standard Pearl do-operator: the intervention is a concrete computational change (zeroing a residual branch), not a graph surgery with identifiable causal estimands. Using 'do' language is misleading unless the causal assumptions are formalized.
  2. [§3, Eq. (1)] Equation (1) defines the score for a single input x and a single forward pass, but the text and Figure 10 refer to 'average KL' over a calibration set. The paper should specify how scores are aggregated over sequences, token positions, and batches; this matters because the method is described as 'input-agnostic' without formally demonstrating that the ranking is stable across inputs.
  3. [§5, Figure 7] The text says Figure 7 verifies 'multiple compression tiers,' but the figure is described as reporting only approximately 5% and 10% budgets. Either report the full set of tiers or adjust the description.
  4. [Table 2] Latency and throughput are reported as single numbers with no measurement protocol or repetition. It is unclear whether these are median, mean, or best-of runs, and whether the speedup comes from skipping modules in a memory-bound regime or from other implementation details.
  5. [§5] The Spearman (0.78) and Pearson (0.51) correlations in Figure 5 are reported without confidence intervals or statistical tests. The claim that gates 'recover much of the causal hierarchy' is not substantiated by a single correlation point.

Circularity Check

1 steps flagged

Main results are externally benchmarked, so only a minor internal 'recovery' check is circular: gates are trained to match the KL scores they are then said to recover.

specific steps
  1. fitted input called prediction [Section 5, 'Recovery of the Causal Hierarchy'; cf. Eq. (1), Eq. (6), Eq. (8)]
    "A key objective of CausalGate is to distill expensive intervention-derived importance estimates into a compact set of learned gate parameters. Figure 5 demonstrates that the learned gates recover much of the causal hierarchy identified through intervention analysis. The resulting gate rankings exhibit a Spearman correlation of 0.78 with the intervention-derived KL rankings."

    The 'causal hierarchy' is defined by Eq. (1) as the KL score Delta_i. The gates are explicitly trained against EMA-smoothed versions of those same scores: L_MSE in Eq. (6) minimizes (g_i - zbar_i)^2, and L_rank in Eq. (8) uses s_ij = sign(zbar_i - zbar_j) as the ordering target. Reporting Spearman 0.78 between the learned gates and the intervention KL rankings is therefore a measure of fit to the training target, not independent confirmation that the gates discovered the hierarchy. The step is tautological but not load-bearing: the paper's main quality claims are tested on downstream PPL/accuracy rather than on agreement with the training signal.

full rationale

The derivation chain is not centrally circular. Intervention importance scores are measured experimentally (Eq. 1), EMA-smoothed, and used as regression/ranking supervision for scalar gates (Eqs. 3-8); the inference mask (Eq. 12) is then applied to external benchmarks. Perplexity and accuracy are independent of the KL supervision signal, so the Table 1 comparisons and latency measurements provide genuine empirical content. The only self-referential feature is the internal 'recovery' claim in Section 5, where gates trained to match KL are said to 'recover' the KL hierarchy; this is a training-fit diagnostic rather than a fresh prediction. The related-work citation to Nair, Rizk, and Santosh 2026 is not load-bearing for the method. Appendix C's do-calculus notation is interpretive framing rather than a formal derivation. The paper's real weakness, noted by the skeptic, is the untested additivity/order-stability assumption in moving from one-at-a-time interventions to a joint top-K mask; that is a correctness/generalization risk, not a circularity, so it does not raise the circularity score above 2.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

CausalGate's contribution is an empirical pipeline; it relies on a set of domain assumptions (intervention KL = importance; global static ranking; WikiText-2 representativeness) and many tuned hyperparameters. No new physical entities are postulated; the scalar gates are internal learned parameters.

free parameters (7)
  • EMA decay beta = 0.9
    Appendix Table 3 OFAT sweep using WikiText-2 PPL at 10% budget; smoothing target for intervention scores (Eq. 5).
  • Target floor alpha = 0.25
    Prevents low-importance target collapse (Eq. 4); tuned via OFAT.
  • Ranking margin gamma = 0.05
    Margin in pairwise ranking loss (Eq. 8); tuned via OFAT.
  • Loss weights (lambda_LM, lambda_sparse, lambda_causal, lambda_rank) = (1.0, 0.001, 10.0, 2.0)
    Final objective Eq. (10); selected by OFAT.
  • Rank pairs per iteration = 128
    Sampled module pairs for ranking loss; selected by OFAT.
  • Gate training learning rate = 0.01
    AdamW LR for gate logits; selected by OFAT.
  • Gate training steps = 1000
    Max optimization steps; selected by OFAT.
axioms (5)
  • domain assumption Importance = KL divergence after zeroing a module's output
    Eq. (1); defines ground truth of the method, but assumes final-logit KL measures true semantic utility.
  • domain assumption Independent, input-agnostic, additive module contributions
    Eqs. (11)-(12) skip modules by global ranking assuming no interaction effects; limitation discussion only partially acknowledges this.
  • domain assumption Calibration data representative (WikiText-2)
    Table 4; gates are learned on WikiText-2 and applied to all inputs/tasks.
  • ad hoc to paper EMA-normalized scores provide stable supervision
    Eqs. (3)-(5); normalization/floor/EMA choices are heuristic smoothing with no theoretical grounding.
  • domain assumption Pairwise ranking of scores is sufficient to preserve hierarchy
    Eqs. (7)-(8); assumes relative order is the only information needed for module selection.

pith-pipeline@v1.3.0-alltime-deepseek · 12768 in / 17434 out tokens · 142691 ms · 2026-08-01T12:23:46.161877+00:00 · methodology

0 comments
read the original abstract

Existing adaptive inference methods for Large Language Models rely on observational heuristics, such as hidden-state similarity or activation magnitudes, to drop redundant modules. However, these correlation-based metrics often fail to capture subtle, non-linear structural computations vital for semantic accuracy. We introduce CausalGate, an intervention-guided framework for compute-efficient transformer inference. During a calibration phase, CausalGate isolates individual Attention and MLP sub-layers, zeros out their respective outputs, and measures the exact semantic damage via the Kullback-Leibler divergence of the final logit distribution. To eliminate runtime routing overhead, this structural importance hierarchy is distilled into a global set of static, lightweight scalar gates using an Exponential Moving Average smoothing objective paired with a differentiable pairwise ranking loss. Evaluated on TinyLlama-1.1B, Qwen2.5-3B, and Llama-3.1-8B across language modeling and commonsense reasoning benchmarks, CausalGate consistently outperforms prominent dynamic routing and layer-skipping baselines, translating theoretical compute savings into concrete hardware latency reductions with zero operational overhead.

Figures

Figures reproduced from arXiv: 2607.22720 by Kiran Nair, Rodrigue Rizk, Smriti Regmi.

Figure 1
Figure 1. Figure 1: TinyLlama backbone [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison between intervention-derived causal [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Scaling behavior of CausalGate on Qwen2.5-3B [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 9
Figure 9. Figure 9: Interventional causal interpretation of Causal [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Module-level causal importance scores obtained through intervention analysis. Each cell reports the average KL [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 7 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2401.02385 , year=

    Tinyllama: An open-source small language model , author=. arXiv preprint arXiv:2401.02385 , year=

    Pith/arXiv arXiv

  2. [2]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    A Spike-Gated Residual Unit for Information Flow Control in Transformers , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  3. [3]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  4. [5]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Adaskip: Adaptive sublayer skipping for accelerating long-context llm inference , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  5. [6]

    Advances in Neural Information Processing Systems , volume=

    Confident adaptive language modeling , author=. Advances in Neural Information Processing Systems , volume=

  6. [7]

    Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    DeeBERT: Dynamic early exiting for accelerating BERT inference , author=. Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

  7. [8]

    IEEE transactions on pattern analysis and machine intelligence , volume=

    Dynamic neural networks: A survey , author=. IEEE transactions on pattern analysis and machine intelligence , volume=. 2021 , publisher=

    2021

  8. [9]

    IEEE Transactions on Sustainable Computing , volume=

    APPQ-CNN: An adaptive CNNs inference accelerator for synergistically exploiting pruning and quantization based on FPGA , author=. IEEE Transactions on Sustainable Computing , volume=. 2024 , publisher=

    2024

  9. [10]

    International conference on machine learning , pages=

    Adaptive neural networks for efficient inference , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  10. [11]

    2016 design, automation & test in europe conference & exhibition (DATE) , pages=

    Conditional deep learning for energy-efficient and enhanced pattern recognition , author=. 2016 design, automation & test in europe conference & exhibition (DATE) , pages=. 2016 , organization=

    2016

  11. [12]

    2016 23rd international conference on pattern recognition (ICPR) , pages=

    Branchynet: Fast inference via early exiting from deep neural networks , author=. 2016 23rd international conference on pattern recognition (ICPR) , pages=. 2016 , organization=

    2016

  12. [14]

    Advances in neural information processing systems , volume=

    Runtime neural pruning , author=. Advances in neural information processing systems , volume=

  13. [16]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    A-vit: Adaptive tokens for efficient vision transformer , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  14. [17]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Adavit: Adaptive vision transformers for efficient image recognition , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  15. [18]

    Advances in neural information processing systems , volume=

    Dynamicvit: Efficient vision transformers with dynamic token sparsification , author=. Advances in neural information processing systems , volume=

  16. [19]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Aim: Adaptive inference of multi-modal llms via token merging and pruning , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  17. [20]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Finding the sweet spot: Analysis and improvement of adaptive inference in low resource settings , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  18. [21]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Transkimmer: Transformer learns to layer-wise skim , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  19. [22]

    Findings of the Association for Computational Linguistics: ACL 2022 , pages=

    A simple hash-based early exiting approach for language understanding and generation , author=. Findings of the Association for Computational Linguistics: ACL 2022 , pages=

    2022

  20. [23]

    International Conference on Learning Representations , volume=

    Model tells you what to discard: Adaptive kv cache compression for llms , author=. International Conference on Learning Representations , volume=

  21. [24]

    International Conference on Learning Representations , volume=

    Duoattention: Efficient long-context llm inference with retrieval and streaming heads , author=. International Conference on Learning Representations , volume=

  22. [25]

    Advances in Neural Information Processing Systems , volume=

    Snapkv: Llm knows what you are looking for before generation , author=. Advances in Neural Information Processing Systems , volume=

  23. [27]

    Advances in neural information processing systems , volume=

    Zeroquant: Efficient and affordable post-training quantization for large-scale transformers , author=. Advances in neural information processing systems , volume=

  24. [28]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Outlier suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  25. [29]

    International conference on machine learning , pages=

    Smoothquant: Accurate and efficient post-training quantization for large language models , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  26. [31]

    International Conference on Machine Learning , pages=

    Flexgen: High-throughput generative inference of large language models with a single gpu , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  27. [32]

    International Conference on Learning Representations , volume=

    Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models , author=. International Conference on Learning Representations , volume=

  28. [33]

    Proceedings of machine learning and systems , volume=

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration , author=. Proceedings of machine learning and systems , volume=

  29. [34]

    Instantaneous grammatical error correction with shallow aggressive decoding , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages=

  30. [35]

    International Conference on Machine Learning , pages=

    Fast inference from transformers via speculative decoding , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  31. [36]

    Advances in Neural Information Processing Systems , volume=

    Blockwise parallel decoding for deep autoregressive models , author=. Advances in Neural Information Processing Systems , volume=

  32. [39]

    Proceedings of the 33rd ACM International Conference on Multimedia , pages=

    Aster: Adaptive dynamic layer-skipping for efficient transformer inference via markov decision process , author=. Proceedings of the 33rd ACM International Conference on Multimedia , pages=

  33. [41]

    International Conference on Learning Representations , volume=

    A simple and effective pruning approach for large language models , author=. International Conference on Learning Representations , volume=

  34. [42]

    Advances in neural information processing systems , volume=

    Llm-pruner: On the structural pruning of large language models , author=. Advances in neural information processing systems , volume=

  35. [43]

    International Conference on Learning Representations , volume=

    Sheared llama: Accelerating language model pre-training via structured pruning , author=. International Conference on Learning Representations , volume=

  36. [44]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Shortgpt: Layers in large language models are more redundant than you expect , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  37. [45]

    E.; Blankevoort, T.; and Welling, M

    Bejnordi, B. E.; Blankevoort, T.; and Welling, M. 2019. Batch-shaping for learning conditional channel gated networks. arXiv preprint arXiv:1907.06627

    Pith/arXiv arXiv 2019

  38. [46]

    Bolukbasi, T.; Wang, J.; Dekel, O.; and Saligrama, V. 2017. Adaptive neural networks for efficient inference. In International conference on machine learning, 527--536. PMLR

    2017

  39. [47]

    Cai, Z.; Zhang, Y.; Gao, B.; Liu, Y.; Li, Y.; Liu, T.; Lu, K.; Xiong, W.; Dong, Y.; Hu, J.; et al. 2024. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069

    Pith/arXiv arXiv 2024

  40. [48]

    Frantar, E.; Ashkboos, S.; Hoefler, T.; and Alistarh, D. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323

    Pith/arXiv arXiv 2022

  41. [49]

    Gao, X.; Zhao, Y.; Mullins, R.; Xu, C.-z.; et al. 2018. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv:1810.05331

    Pith/arXiv arXiv 2018

  42. [50]

    Ge, S.; Zhang, Y.; Liu, L.; Zhang, M.; Han, J.; and Gao, J. 2024. Model tells you what to discard: Adaptive kv cache compression for llms. In International Conference on Learning Representations, volume 2024, 22975--22988

    2024

  43. [51]

    Han, Y.; Huang, G.; Song, S.; Yang, L.; Wang, H.; and Wang, Y. 2021. Dynamic neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(11): 7436--7456

    2021

  44. [52]

    He, Z.; Yao, Y.; Zuo, P.; Gao, B.; Li, Q.; Zheng, Z.; and Wu, F. 2025. Adaskip: Adaptive sublayer skipping for accelerating long-context llm inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 24050--24058

    2025

  45. [53]

    G.; and Asano, Y

    Laitenberger, F.; Kopiczko, D.; Snoek, C. G.; and Asano, Y. M. 2025. What Layers When: Learning to Skip Compute in LLMs with Residual Gates. arXiv preprint arXiv:2510.13876

    arXiv 2025

  46. [54]

    Leviathan, Y.; Kalman, M.; and Matias, Y. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, 19274--19286. PMLR

    2023

  47. [55]

    Lin, J.; Rao, Y.; Lu, J.; and Zhou, J. 2017. Runtime neural pruning. Advances in neural information processing systems, 30

    2017

  48. [56]

    Lin, J.; Tang, J.; Tang, H.; Yang, S.; Chen, W.-M.; Wang, W.-C.; Xiao, G.; Dang, X.; Gan, C.; and Han, S. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6: 87--100

    2024

  49. [57]

    Liu, F.; Wang, J.; Yang, N.; Wang, Z.; Zhao, J.; Jiang, L.; and Guan, H. 2025. Aster: Adaptive dynamic layer-skipping for efficient transformer inference via markov decision process. In Proceedings of the 33rd ACM International Conference on Multimedia, 11853--11861

    2025

  50. [58]

    Liu, Y.; Meng, F.; and Zhou, J. 2024. Accelerating inference in large language models with a unified layer skipping strategy. arXiv preprint arXiv:2404.06954

    Pith/arXiv arXiv 2024

  51. [59]

    Luo, X.; Wang, W.; and Yan, X. 2025. Adaptive layer-skipping in pre-trained llms. arXiv preprint arXiv:2503.23798

    arXiv 2025

  52. [60]

    Ma, X.; Fang, G.; and Wang, X. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36: 21702--21720

    2023

  53. [61]

    Men, X.; Xu, M.; Zhang, Q.; Yuan, Q.; Wang, B.; Lin, H.; Lu, Y.; Han, X.; and Chen, W. 2025. Shortgpt: Layers in large language models are more redundant than you expect. In Findings of the Association for Computational Linguistics: ACL 2025, 20192--20204

    2025

  54. [62]

    Meng, L.; Li, H.; Chen, B.-C.; Lan, S.; Wu, Z.; Jiang, Y.-G.; and Lim, S.-N. 2022. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12309--12318

    2022

  55. [63]

    Nair, K.; Rizk, R.; and Santosh, K. 2026. A Spike-Gated Residual Unit for Information Flow Control in Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3469--3478

    2026

  56. [64]

    J.; Kim, B.; Lee, Y.; Lee, D.; et al

    Park, G.; Kim, M.; Lee, S.; Kim, J.; Kwon, B.; Kwon, S. J.; Kim, B.; Lee, Y.; Lee, D.; et al. 2024. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models. In International Conference on Learning Representations, volume 2024, 38069--38086

    2024

  57. [65]

    Rao, Y.; Zhao, W.; Liu, B.; Lu, J.; Zhou, J.; and Hsieh, C.-J. 2021. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34: 13937--13949

    2021

  58. [66]

    C.; and Santoro, A

    Raposo, D.; Ritter, S.; Richards, B.; Lillicrap, T.; Humphreys, P. C.; and Santoro, A. 2024. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258

    Pith/arXiv arXiv 2024

  59. [67]

    Rotem, D.; Hassid, M.; Mamou, J.; and Schwartz, R. 2023. Finding the sweet spot: Analysis and improvement of adaptive inference in low resource settings. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 14836--14851

    2023

  60. [68]

    Schuster, T.; Fisch, A.; Gupta, J.; Dehghani, M.; Bahri, D.; Tran, V.; Tay, Y.; and Metzler, D. 2022. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35: 17456--17472

    2022

  61. [69]

    Stern, M.; Shazeer, N.; and Uszkoreit, J. 2018. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31

    2018

  62. [70]

    Sun, M.; Liu, Z.; Bair, A.; and Kolter, Z. 2024. A simple and effective pruning approach for large language models. In International Conference on Learning Representations, volume 2024, 4942--4964

    2024

  63. [71]

    Sun, X.; Ge, T.; Wei, F.; and Wang, H. 2021. Instantaneous grammatical error correction with shallow aggressive decoding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 5937--5947

    2021

  64. [72]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  65. [73]

    Xia, M.; Gao, T.; Zeng, Z.; and Chen, D. 2024. Sheared llama: Accelerating language model pre-training via structured pruning. In International Conference on Learning Representations, volume 2024, 5385--5409

    2024

  66. [74]

    Xiao, G.; Lin, J.; Seznec, M.; Wu, H.; Demouth, J.; and Han, S. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, 38087--38099. PMLR

    2023

  67. [75]

    Xiao, G.; Tang, J.; Zuo, J.; Guo, J.; Yang, S.; Tang, H.; Fu, Y.; and Han, S. 2025. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. In International Conference on Learning Representations, volume 2025, 37228--37253

    2025

  68. [76]

    M.; Mallya, A.; Kautz, J.; and Molchanov, P

    Yin, H.; Vahdat, A.; Alvarez, J. M.; Mallya, A.; Kautz, J.; and Molchanov, P. 2022. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10809--10818

    2022

  69. [77]

    Zhang, X.; Xiao, G.; Duan, M.; Chen, Y.; and Li, K. 2024. APPQ-CNN: An adaptive CNNs inference accelerator for synergistically exploiting pruning and quantization based on FPGA. IEEE Transactions on Sustainable Computing, 9(6): 874--888

    2024