Pith. sign in

REVIEW 2 major objections 5 minor 31 references

Token-level dynamic width pruning keeps more LLM accuracy at 50% sparsity and turns it into real prefill and decode speedups.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 07:58 UTC pith:VRQJDIUQ

load-bearing objection Solid pruning–systems package: token-wise width routing plus mask-reorder kernels that actually beat depth routing on quality and show real prefill/decode speedups on Llama 3B/8B. the 2 major comments →

arxiv 2607.28418 v1 pith:VRQJDIUQ submitted 2026-07-30 cs.AI cs.CLcs.LG

WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning

Haozhe Hu , Hao Wu , Peiran Yin , Chao Han , Yunpu Ma , Xiaoyu Shen This is my paper
classification cs.AI cs.CLcs.LG
keywords LLM pruningdynamic width pruningtoken-level routingstructured sparsitymask reorderingGPU kernel co-designprefill and decode accelerationGQA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

WIDE claims that large language models waste compute when every token runs the same full layer path. Instead of permanently deleting layers or only skipping whole layers per token, it lets each token pick which groups of attention heads and feed-forward channels to run. Lightweight routers learn those choices from a small calibration set, with optional low-rank recovery. The same design packs the irregular choices into tile-aligned layouts so GPU kernels can skip idle blocks without expensive gather-scatter. On Llama-scale models at 50% target sparsity, the method retains far more zero-shot accuracy than static structured pruners and dynamic depth routers under matched training, and measures end-to-end speedups near 1.68× prefill and 1.55× decode. A sympathetic reader cares because the paper ties quality retention and hardware speed into one co-designed loop rather than treating pruning as a model-only compression step.

Core claim

The paper establishes that end-to-end differentiable token-level dynamic width pruning—each token selecting GQA-aligned attention-head groups and configurable FFN-channel groups—plus a two-stage router/LoRA pipeline and mask-reordering kernels, retains substantially more accuracy at aggressive sparsity than static structured or dynamic-depth baselines while converting that sparsity into near-theoretical kernel speedups and practical end-to-end acceleration for both prefill and decode.

What carries the argument

Token-wise group routers plus unified mask reordering: binary masks are sorted so active tokens form contiguous prefixes, then multi-granularity predicates skip idle CTA tiles, load packets, and MMA fragments inside otherwise dense GEMM and attention kernels.

Load-bearing premise

Group-structured masks and standalone reordering stay fine enough for quality yet regular enough that block skipping still yields real speedups under ordinary serving loads, not only the long-context single-batch CUDA-Graph settings the paper profiles.

What would settle it

Re-run the 50% sparsity Llama3.1-8B comparison under matched calibration and LoRA: if WIDE’s average zero-shot accuracy no longer beats the strongest dynamic-depth and static-width baselines by several points, or if end-to-end prefill/decode speedups collapse once batch mix, shorter contexts, or non-graph serving are used, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Dynamic structured pruning can move below layer/sublayer decisions to neuron-block groups without abandoning hardware-friendly layouts.
  • At 50% sparsity, calibration-only or lightly recovered models can keep roughly 85–90% of dense zero-shot accuracy while static and depth-only dynamic methods fall further.
  • Mask reordering plus staged CTA/intra-block skipping can approach ideal speedups on the prunable GEMM and attention path for both prefill and decode.
  • End-to-end gains remain bounded by fixed work (KV projections, element-wise ops, routers, launches), so co-design must target those residuals next.
  • Group size becomes an explicit knob trading routing flexibility against tiling efficiency.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Serving stacks that already rely on CUDA Graphs and GQA packing are the natural first home for this style of dynamic width routing.
  • If atomic writeback contention in K-axis kernels grows with smaller groups, production deployments may prefer coarser FFN groups even when quality allows finer ones.
  • The learned non-uniform attention-vs-FFN and layer-wise sparsity patterns suggest routers could be reused as cheap importance signals for static pruning or mixture-of-experts load balancing.
  • Extending the same mask-reorder abstraction to multimodal or mixture-of-experts backbones is a direct next measurement.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. WIDE proposes token-level dynamic width pruning for LLMs: lightweight bottleneck routers assign each token binary execute/skip decisions over GQA-aligned attention-head groups and tiled FFN-channel groups, trained with a two-stage router-then-optional-LoRA pipeline under an LM plus sparsity objective. To make irregular token-wise masks practical on GPUs, the paper co-designs a mask-reordering preprocessor with multi-granularity predicated kernels (CTA early-exit plus architecture-dependent load/MMA skipping) for GEMM-MN, GEMM-K, and attention, covering prefill and decode. On Llama3.1-8B and Llama3.2-3B at 25–50% target sparsity, calibration-only and LoRA-matched comparisons against static depth/width and dynamic-depth baselines (Tables 1–2, 5–7) show substantially higher zero-shot retention; kernel and end-to-end measurements report near-ideal accelerated-path speedups and about 1.68×/1.55× prefill/decode wall-clock gains under the profiled regime, with code released.

Significance. If the results hold under broader serving conditions, this is a meaningful advance on the quality–efficiency frontier of structured LLM pruning: it moves dynamic routing from coarse layer/sublayer skipping to neuron-block width allocation while coupling the routing representation to executable GPU kernels for both prefill and decode. Strengths include matched calibration/LoRA baselines, group-size and sparsity sweeps, an honest layer-latency breakdown of fixed vs accelerable work, detailed naive gather–scatter cost analysis, and public code. The pruning–kernel co-design (mask reordering plus progressive predication) is a concrete systems contribution beyond another router paper, and the uniform-sparsity appendix strengthens the quality claims.

major comments (2)
  1. [§4.3, Figs. 4 and 8; Appendix B] §4.1 and §4.3 (Figs. 4 and 8) ground the headline end-to-end claims (1.68× prefill, 1.55× decode) almost entirely in B=1, long-context (T=16,384), CUDA-Graph runs on sm120. Fig. 4 already shows non-accelerable work at ~16.7% (prefill) and ~29.1% (decode), and Appendix B notes GEMM-K atomic-reduction contention and group-local pipeline limits for small G. These factors can change under multi-batch, short-context, or non-graph serving. Either expand e2e results across batch/context mixes and at least one other GPU generation, or explicitly bound the acceleration claim to the profiled regime so the abstract’s “real-world inference” framing is not overstated.
  2. [Abstract; §4.2; Fig. 7] The abstract’s “55.1% performance boost … vs state-of-the-art dynamic depth pruning under calibration-only settings” is a relative accuracy lift on Llama3.2-3B (Table 1: 57.00 vs SkipGPT 36.74), not the primary 8B setting and not the absolute-point gaps emphasized in §4.2 (e.g., +8.80 vs DDP on 8B). Peak kernel figures (up to 4.95× decode) similarly sit above the layer-wise accelerated-path averages in Fig. 4 (~1.8–1.9×). Please restate abstract/contribution bullets with model, metric definition (relative vs absolute), and e2e vs kernel scope aligned to Tables 1–2 and Figs. 4/7–8 so the central empirical claim is not front-loaded by the most favorable single number.
minor comments (5)
  1. [Figure 1] Figure 1 caption and router diagram text contain OCR-style artifacts (“bo,leneck”, “A:en;on”, “ac&ve”). Clean these in the camera-ready figures.
  2. [§3.2 Eq. (6); §4.2; Appendix C] Eq. (6) uses a single global sparsity target S averaged over attention and FFN; Fig. 9 then shows highly skewed realized sparsity (Attn ~66%, FFN ~28%). A one-sentence pointer in §4.2 to the uniform-sparsity study (Appendix C, Table 4) would help readers who stop at the main tables.
  3. [§2] Related work cites PolarSparse and FastForward as scenario-limited dynamic width methods; a short explicit contrast table (prefill vs decode, training vs heuristic, kernel support) would sharpen the “first end-to-end … both prefill and decode” claim without relying only on prose.
  4. [§4.3] Table numbering in the main text refers to a “Table 5” group-size study that appears as an unnumbered/inline table (Figure 5 label in one place). Harmonize labels between body and floats.
  5. [§4.1 Inference Implementation] Report wall-clock variance or repeated-run statistics for e2e TTFT/TPOT (ELANA-style profiling), even briefly, given CUDA Graph and single-GPU measurements.

Circularity Check

0 steps flagged

No significant circularity: empirical train/eval systems paper; accuracy and speedups are measured, not redefined from fitted inputs.

full rationale

WIDE’s load-bearing claims are (i) better zero-shot/WikiText2 retention than matched static-width and dynamic-depth baselines after router calibration (and optional LoRA) at a stated sparsity budget S, and (ii) kernel/e2e speedups from mask-reordering plus multi-granularity skipping versus dense baselines. The training objective (Eq. 6) is standard LM loss plus an auxiliary sparsity penalty that drives average mask density toward hyperparameter S; quality is then reported on external suites (ARC, BoolQ, HellaSwag, etc.) and speedups are wall-clock/TFLOPs measurements (Figs. 3–8, Tables 1–2). Nothing in that chain equates a claimed “prediction” or first-principles quantity to a fitted constant by construction. Methodological inheritance from SkipGPT (loss shape, two-stage recipe, depth baseline) is ordinary related-work reuse, not a self-citation uniqueness theorem or ansatz that forces the accuracy/speedup results. S and α are chosen hyperparameters, not renamed outcomes. Score 0 is appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

Load-bearing content is methodological and empirical rather than axiomatic physics/math. The claim rests on standard Transformer/GQA/FlashAttention structure, a chosen group granularity G, a sparsity-augmented training loss, and the systems premise that reordered masks plus predicated skips preserve correctness while exposing GPU regularity. Free parameters are ordinary ML hyperparameters; invented entities are engineering modules (routers, reordering, skip predicates), not new physical objects.

free parameters (5)
  • target sparsity S = typically 0.5 in main claims
    Global compute budget enforced via auxiliary loss; central Pareto claims are reported at chosen S (especially 25%/50%).
  • sparsity loss weight α = 20
    Balances LM loss vs sparsity; set following SkipGPT rather than derived.
  • pruning group size G (attention/FFN) = e.g. G_ffn in {16..512}, G_attn GQA-aligned (often 512)
    Controls quality–kernel trade-off; attention tied to GQA group size, FFN chosen from GEMM tile set.
  • router bottleneck rank r and Gumbel temperature schedule = r in {16,32}; τ annealed 5→0.5
    Router capacity and exploration/exploitation during differentiable mask training.
  • LoRA rank/α/dropout and 10k-step recovery recipe = r=16, α=32, dropout=0.1, 10k steps
    Optional second-stage recovery that materially lifts several baselines and WIDE; shared recipe is a design choice.
axioms (5)
  • domain assumption Transformer attention/FFN compute dominates and can be structured-pruned along head groups and FFN intermediate groups without changing the functional API of residual blocks.
    §3.1 defines atomic pruning units from standard GQA and GEMM tiling.
  • domain assumption Hard routing at inference is adequately trained by straight-through Gumbel-Softmax masks plus sparsity penalty on calibration data (optionally + LoRA).
    §3.2 two-stage pipeline; quality claims depend on this train–test mask agreement.
  • domain assumption Sorting each routing column into active prefixes makes almost all CTA tiles fully active or inactive, so predicate skips preserve numerical results of the dense ops on kept pathways.
    §3.3 and Appendix B; correctness of acceleration path rests on this layout invariant.
  • domain assumption RedPajama-1T sample calibration and lm-eval zero-shot suite are adequate proxies for general quality retention under pruning.
    §4.1 evaluation protocol used for all main tables.
  • standard math Standard linear algebra / GEMM and FlashAttention online-softmax identities used in kernel pseudocode.
    Appendix A–B traffic accounting and Algorithms 2–4.
invented entities (2)
  • WIDE bottleneck routers producing per-token group execute/skip masks for attention and FFN independent evidence
    purpose: Learn token-conditional width allocation finer than layer/sublayer routing.
    Core model contribution in §3.1–3.2; standard MLP router pattern specialized to group masks.
  • Unified mask-reordering + multi-granularity skip kernel framework (GEMM-MN, GEMM-K, Attention) independent evidence
    purpose: Turn irregular dynamic width masks into practical GPU speedups without materializing token-specific weights.
    §3.3, Fig. 2, Appendix B; engineering construct validated by throughput/memory benchmarks.

pith-pipeline@v1.2.0-daily-grok45 · 38311 in / 4022 out tokens · 89435 ms · 2026-07-31T07:58:39.854750+00:00 · methodology

0 comments
read the original abstract

Pruning is a promising approach for improving the efficiency of LLMs. Existing static structured pruning methods are hardware-friendly and can deliver practical throughput gains, but their input-agnostic computation allocation often causes substantial accuracy degradation under aggressive sparsity. Recent dynamic sparsity methods improve quality retention by adapting computation to individual inputs, yet they remain largely limited to coarse-grained structural decisions and their practical acceleration under real-world inference scenarios remains challenging. To address these challenges, we present WIDE, the first end-to-end differentiable token-level dynamic width pruning framework designed for both prefill and decode scenarios. WIDE enables fine-grained computation allocation by allowing each token to dynamically select attention-head groups and FFN-channel groups, extending dynamic pruning beyond layer-level decisions to neuron-block-level granularity. Through a two-stage training pipeline, WIDE learns effective token-wise sparse execution patterns and achieves substantially better quality retention than existing approaches. To make such fine-grained dynamic pruning practical, we further propose a pruning--kernel co-design framework that decomposes dynamic sparsity acceleration into mask reordering, hardware-agnostic block-level skipping, and hardware-dependent intra-block skipping, enabling efficient execution across different granularities. At 50% sparsity, WIDE provides 55.1% performance boost when compared to the state-of-the-art dynamic depth pruning under calibration-only settings. Under prefill and decoding inference workloads, WIDE achieves close-to-theoretical kernel-level speedups of up to 1.98x for prefill and 4.95x for decoding, as well as 1.68x and 1.55x end-to-end acceleration. Our code is available at https://github.com/EIT-NLP/LLM-Pruning/tree/main/WIDE.

Figures

Figures reproduced from arXiv: 2607.28418 by Chao Han, Hao Wu, Haozhe Hu, Peiran Yin, Xiaoyu Shen, Yunpu Ma.

Figure 1
Figure 1. Figure 1: The pruning dimension of WIDE in the attention and FFN layer, with [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Workflow of WIDE for GEMM N-axis pruning. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: GEMM latency breakdown of dense and naive dynamic depth pruning. (2). Extra memory movement and storage. For dynamic depth pruning, ignoring mask and index￾processing overhead, explicit gather-scatter increases the memory traffic of a dense GEMM by roughly 3pM(N+K)+NK+MN MK+NK+MN , where p denotes the active ra￾tio. For finer-grained dynamic width pruning, each token may select a different subset of B. A n… view at source ↗
Figure 4
Figure 4. Figure 4: Layer-wise latency breakdown for Llama3.1-8B at 50% sparsity, including three types of [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: WIDE performance on Llama3.1- 8B 50% sparsity with different group size. G Non-LoRA LoRA WT2 Avg. Acc. WT2 Avg. Acc. Vary FFN, Gattn = 512 16 12.02 62.89 (87.89%) 11.37 64.92 (90.74%) 32 14.15 61.84 (86.42%) 11.99 64.82 (90.59%) 64 14.96 61.17 (85.50%) 12.05 64.65 (90.36%) 128 12.57 61.48 (85.93%) 11.43 64.33 (89.92%) 256 14.51 60.27 (84.24%) 12.37 64.81 (90.58%) 512 22.12 53.84 (75.25%) 13.86 63.79 (89.15… view at source ↗
Figure 6
Figure 6. Figure 6: Average zero-shot accuracy of [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Kernel-level speedups of the four types of WIDE kernels on Llama3.1-8B shapes under [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: End-to-end speedup of WIDE over the dense Llama3.1-8B baseline across sparsity and [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: WIDE and DDP’s layer-wise spar￾sity distribution of Llama3.1-8B on Wiki￾Text2, with target sparsity set to 50%. These profiles echo the layer sensitivity observed in static pruning studies (He et al., 2024; Huang et al., 2026), but are produced here by token-conditioned group routing. For the token-wise routing behav￾ior, [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Token-wise routing probability for Llama3.1-8B in selected Hellaswag sample. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Throughput and memory overhead benchmarks for M,N,K axis GEMM pruning based on [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: WIDE’s layer-wise sparsity with different [PITH_FULL_IMAGE:figures/full_fig_p026_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Attention decoding speedup of different group size on Llama3.1-8B shapes, with a random [PITH_FULL_IMAGE:figures/full_fig_p028_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Token-wise routing probability for Llama3.1-8B in selected ARC-Easy sample. [PITH_FULL_IMAGE:figures/full_fig_p029_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Token-wise routing probability for Llama3.1-8B in selected ARC-Challenge sample. [PITH_FULL_IMAGE:figures/full_fig_p029_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Token-wise routing probability for Llama3.1-8B in selected OpenbookQA sample. [PITH_FULL_IMAGE:figures/full_fig_p029_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Token-wise routing probability for Llama3.1-8B in selected OpenbookQA sample. [PITH_FULL_IMAGE:figures/full_fig_p030_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Token-wise routing probability for Llama3.1-8B in selected Hellaswag sample. [PITH_FULL_IMAGE:figures/full_fig_p030_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Group activation patterns of different layer in Llama3.1-8B on WikiText2. [PITH_FULL_IMAGE:figures/full_fig_p030_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 15 linked inside Pith

  1. [1]

    GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Check- points

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Check- points. InThe 2023 Conference on Empirical Methods in Natural Language Processing, December

    2023

  2. [8]

    Aayush Gautam, Mukul Gagrani, Junyoung Park, Mingu Lee, Chiris Lott, and Narasimha Reddy

    URLhttps://zenodo.org/records/12608602. Aayush Gautam, Mukul Gagrani, Junyoung Park, Mingu Lee, Chiris Lott, and Narasimha Reddy. Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsity. https://arxiv.org/abs/2602.00397v1, January

    arXiv

  3. [9]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava 1...

    Pith/arXiv arXiv

  4. [10]

    Informed Routing in LLMs: Smarter Token-Level Computation for Faster Inference

    Chao Han, Yijuan Liang, Zihao Xuan, Daokuan Wu, Wei Zhang, and Xiaoyu Shen. Informed Routing in LLMs: Smarter Token-Level Computation for Faster Inference. http://arxiv.org/abs/2510.13831, October

    arXiv

  5. [11]

    What Matters in Transformers? Not All Attention is Needed

    Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. What Matters in Transformers? Not All Attention is Needed. http://arxiv.org/abs/2406.15786, October

    Pith/arXiv arXiv

  6. [12]

    SkipOPU: An FPGA-based Overlay Processor for Large Language Models with Dynamically Allocated Computation

    Zicheng He, Anhao Zhao, Xiaoyu Shen, Chen Wu, and Lei He. SkipOPU: An FPGA-based Overlay Processor for Large Language Models with Dynamically Allocated Computation. https://arxiv.org/abs/2603.14785v1, March

    Pith/arXiv arXiv

  7. [14]

    Deterministic Differentiable Structured Pruning for Large Language Models

    Weiyu Huang, Pengle Zhang, Xiaolu Zhang, Jun Zhou, Jun Zhu, and Jianfei Chen. Deterministic Differentiable Structured Pruning for Large Language Models. http://arxiv.org/abs/2603.08065, May

    Pith/arXiv arXiv

  8. [15]

    Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods

    Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods. http://arxiv.org/abs/2402.02834, June

    Pith/arXiv arXiv

  9. [16]

    Gonzalez, Hao Zhang, and Ion Stoica

    14 Preprint Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. http://arxiv.org/abs/2309.06180, September

    Pith/arXiv arXiv

  10. [18]

    ISBN 979-8-89176-256-5

    Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.1035. Qianyu Meng, Yanan Wang, Liyi Chen, Wei Wu, Yihang Li, Wenyuan Jiang, Qimeng Wang, Chengqiang Lu, Yan Gao, Yi Wu, and Yao Hu. Agent harness for large language model agents: A survey

    doi:10.18653/v1/2025.findings-acl.1035 2025

  11. [19]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher

    doi: 10.20944/preprints202604.0428.v3. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843,

    arXiv

  12. [21]

    Mixture-of-Depths: Dynamically allocating compute in transformer-based language models

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-Depths: Dynamically allocating compute in transformer-based language models. http://arxiv.org/abs/2404.02258, April

    Pith/arXiv arXiv

  13. [22]

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, S. H. Cai, Yuan Cao, Y . Charles, H. S. Che, Cheng Chen, Guanduo Chen, Huarong Chen, Jia Chen, Jiahao Chen, Jianlong Chen, Jun Chen, Kefan Chen, Liang Chen, Ruijue Chen, Xinhao Chen, Yanru Chen, Yanxu Chen, Yicun Chen, Yimin Chen, Yingjiang Chen, Yuankun Chen, Yujie Chen, Yutian Chen, Zhirong Chen, Ziwei Che...

    Pith/arXiv arXiv

  14. [23]

    Philippe Tillet, H. T. Kung, and David Cox. Triton: An intermediate language and compiler for tiled neural network computations. InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, pp. 10–19, New York, NY , USA, June

    2019

  15. [24]

    ISBN 978-1-4503-6719-6

    Association for Computing Machinery. ISBN 978-1-4503-6719-6. doi: 10.1145/3315508.3329973. Lei Wang, Yu Cheng, Yining Shi, Zhiwen Mo, Zhengju Tang, Wenhao Xie, Tong Wu, Lingxiao Ma, Yuqing Xia, Jilong Xue, Fan Yang, and Zhi Yang. TileLang: Bridge Programmability and Performance in Modern Neural Kernels. InThe Fourteenth International Conference on Learnin...

    arXiv

  16. [25]

    From data to model: A survey of the compression lifecycle in mllms

    Hao Wu, Junlong Tong, Xudong Wang, Yang Tan, Changyu Zeng, Anastasia Antsiferova, and Xiaoyu Shen. From data to model: A survey of the compression lifecycle in mllms. February 2026a. doi: 10.36227/techrxiv.177220375.55495124/v1. URL http://dx.doi.org/10.36227/ techrxiv.177220375.55495124/v1. 16 Preprint Hao Wu, Xudong Wang, Jialiang Zhang, Junlong Tong, X...

    arXiv

  17. [26]

    Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830,

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830,

    Pith/arXiv arXiv 1905

  18. [27]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient Execution of Structured Language Model Programs. http://arxiv.org/abs/2312.07104, June

    Pith/arXiv arXiv

  19. [28]

    BlockPruner: Fine- grained Pruning for Large Language Models

    Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzhi Li. BlockPruner: Fine- grained Pruning for Large Language Models. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 5065–5080, Vienna, Austria, July

    2025

  20. [29]

    mk,mnk->mn

    Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.262. 17 Preprint A THECOST FORNAIVEGATHER-SCATTERIMPLEMENTATIONS In this section, we first provide a narrow, theoretical memory accounting for naive gather-scatter implementations of M-axis, N-axis, and K-axis dynamic pruning. We count logical data-elemen...

    doi:10.18653/v1/2025.findings-acl.262 2025

  21. [30]

    ,min(P−1, N K,g)−1do 6:PREFETCHK(g, q, qmodP, s (1),J)▷ ℓ= 1: predicated A loading 7:end for Group-local pipelined mainloop 8:fork c = 0,

    2:(s (0), s(1), s(2),J)←PREDICATES([m 0,min(m 0 +BM, M)), g) 3:ifs (0) then Group-local prologue 4:C acc ←0;N K,g ← ⌈|Kg|/BK⌉ 5:forq= 0, . . . ,min(P−1, N K,g)−1do 6:PREFETCHK(g, q, qmodP, s (1),J)▷ ℓ= 1: predicated A loading 7:end for Group-local pipelined mainloop 8:fork c = 0, . . . , NK,g −1do 9:s←k c modP; WAITSTAGE(s) 10:q←k c +P−1 11:ifq < N K,g th...

    2023

  22. [31]

    1:Maph q to KV headh kv and routing groupg 2:(s (0), s(1), s(2),J)←PREDICATES(b,[m 0,min(m 0 +BM, Tq)), g) 3:if¬s (0) then 4:return▷ ℓ= 0: CTA early exit 5:end if 6:A ← {r∈[m0,min(m 0 +BM, Tq)) : fMb,r,g = 1} Qprologue (ℓ= 1: Q loading) 7:Async-gather the activeQ[b,J, h q,:]tile according tos (1); wait until ready 8:(m,z,O acc)←(−∞,0,0) 9:δ←T k −T q ▷quer...

    2048

  23. [2016]

    Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789,

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789,

    Pith/arXiv arXiv

  24. [2018]

    CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation

    Weinan Dai, Hanlin Wu, Qiying Yu, Huan-ang Gao, Jiahao Li, Chengquan Jiang, Weiqiang Lou, Yufan Song, Hongli Yu, Jiaze Chen, Wei-Ying Ma, Ya-Qin Zhang, Jingjing Liu, Mingxuan Wang, Xin Liu, and Hao Zhou. CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation. http://arxiv.org/abs/2602.24286, February

    arXiv

  25. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    Pith/arXiv arXiv

  26. [2020]

    A VO: Agentic Variation Operators for Autonomous Evolutionary Search

    Terry Chen, Zhifan Ye, Bing Xu, Zihao Ye, Timmy Liu, Ali Hassani, Tianqi Chen, Andrew Kerr, Haicheng Wu, Yang Xu, Yu-Jung Chen, Hanfeng Chen, Aditya Kane, Ronny Krashinsky, Ming- Yu Liu, Vinod Grover, Luis Ceze, Roger Bringmann, John Tran, Wei Liu, Fung Xie, Michael Lightstone, and Humphrey Shi. A VO: Agentic Variation Operators for Autonomous Evolutionar...

    arXiv

  27. [2021]

    Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy

    Haozhe Hu, Hao Wu, Anhao Zhao, Longwei Ding, Peiran Yin, Yunpu Ma, and Xiaoyu Shen. Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy. http://arxiv.org/abs/2606.09080, June

    Pith/arXiv arXiv

  28. [2023]

    ShortGPT: Layers in Large Language Models are More Redundant Than You Expect

    Xin Men, Mingyu Xu, Qingyu Zhang, Qianhao Yuan, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 20192–20204, Vienna, Austria, July

    2025

  29. [2024]

    ELANA: A Simple Energy and Latency Analyzer for LLMs

    Hung-Yueh Chiang, Bokun Wang, and Diana Marculescu. ELANA: A Simple Energy and Latency Analyzer for LLMs. http://arxiv.org/abs/2512.09946, December

    arXiv

  30. [2025]

    Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044,

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044,

    Pith/arXiv arXiv 1905

  31. [2026]

    A survey on deep neural network pruning- taxonomy, comparison, analysis, and recommendations

    Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey on deep neural network pruning- taxonomy, comparison, analysis, and recommendations. http://arxiv.org/abs/2308.06767, August

    Pith/arXiv arXiv