Pith. sign in

REVIEW 2 major objections 5 minor 119 references

Depth pruning of vision transformers fails mainly because attention and activation layers behave differently; allocating a mixed budget by type and then merging linear layers recovers accuracy while raising speedup.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 23:57 UTC pith:EFSKHWR5

load-bearing objection Solid engineering paper: mixed attention/GELU depth pruning with a cheap MAP budget works, and the speedups are real; MAP ranking fidelity is the softest link but not a collapse of the claim. the 2 major comments →

arxiv 2607.03784 v1 pith:EFSKHWR5 submitted 2026-07-04 cs.CV cs.AI

Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective

Zhenfeng Su , Kang Zhao , Han Bao , Tao Yuan , Zhongzhe Hu , Xianzhi Yu , Wenxuan Wang This is my paper
classification cs.CV cs.AI
keywords vision transformersdepth pruningstructured pruningheterogeneityactivation-layer removalmodel accuracy predictorlinear-layer mergingextreme compression
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Vision transformers can be sped up more by removing whole layers (depth pruning) than by shrinking channels or heads inside layers (width pruning), but earlier depth-pruning methods lost too much accuracy. This paper argues that the real problem is heterogeneity: attention layers and GELU activation layers have very different gradient scales and very different recovery curves after pruning. It proposes HetDPT, which first decides how many of each type to remove by fitting a simple model-accuracy predictor on cheap short fine-tunes, then ranks layers only inside each type, prunes, fine-tunes, and merges the now-adjacent linear layers so dimensions stay consistent. On ImageNet the method keeps DeiT-Base accuracy at 1.58× measured speedup and nearly matches DeiT-Small at 1.39×; when stacked with existing width pruning it lifts an extreme compression point from 4.24× to 5.19× with near-lossless accuracy, and the same models transfer to detection and segmentation.

Core claim

The accuracy collapse of prior ViT depth pruning is caused less by coarse granularity than by treating attention and activation layers as interchangeable. Because those two layer types exhibit gradient disparity and recovery asymmetry, any metric that compares them directly is biased. Once a mixed pruning budget is chosen by a polynomial accuracy predictor that respects final recovered accuracy, and layers are selected only within type, the resulting network can be fine-tuned and the linear layers that no longer have activations between them can be merged, restoring dimension consistency and delivering large practical speedups.

What carries the argument

HetDPT’s Model Accuracy Predictor (MAP): a low-degree bivariate polynomial fitted on a small set of incremental prune–fast-finetune accuracy points that recommends the optimal split of the total layer budget between attention layers and activation layers, after which importance scores are computed only inside each homogeneous group and adjacent linear layers are merged.

Load-bearing premise

That a cheap polynomial fitted on short, subset-based fine-tunes still ranks pruning budgets the same way a full, expensive fine-tune would, so its recommended mix of attention and activation removals is near-optimal.

What would settle it

For a fixed total layer budget on DeiT-Base, exhaustively train every feasible (attention-count, activation-count) pair to full convergence and check whether the MAP-selected pair still yields the highest recovered accuracy; a systematic re-ranking would falsify the surrogate.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper argues that prior ViT depth pruning fails mainly because it ignores heterogeneity between attention and GELU activation layers (gradient disparity and recovery asymmetry), and that this, together with dimension mismatch when pruning linear layers, limits both pure depth and joint width–depth methods. It proposes HetDPT: a two-stage pipeline that (i) allocates a mixed pruning budget between attention and activation layers via a polynomial Model Accuracy Predictor (MAP) fitted on lightweight prune–finetune–evaluate data, (ii) selects specific layers within each type using learnable importance scores, and (iii) merges adjacent linear layers after activation removal to restore dimension alignment and gain speedup. Empirically, HetDPT reports 1.58× speedup on DeiT-B at baseline accuracy and 1.39× on DeiT-S with near-lossless accuracy, plus transfer results on CIFAR-100, COCO, and ADE20K; HetDPT+ combined with width pruning improves the Isomorphic-Pruning-2.6G configuration from 4.24× to 5.19× speedup at near-lossless accuracy. Theoretical appendices establish existence of an optimum, polynomial approximability under continuous relaxation, and maximizer preservation under constant bias.

Significance. If the heterogeneity diagnosis and the MAP-guided mixed-budget design hold, the work meaningfully advances structured ViT compression: depth pruning has long been known to offer higher hardware speedup than width pruning but has been hard to recover, and the paper shows that mixed attention/activation removal plus linear merging can deliver lossless or near-lossless depth compression and push extreme joint compression past prior SOTA. Strengths include multi-benchmark evaluation (ImageNet, CIFAR transfer, COCO, ADE20K), ablations isolating budget allocation and within-type selection, orthogonality checks with token pruning, a public code link, and an explicit theoretical appendix (existence, Stone–Weierstrass under continuous relaxation, bias-invariance of maximizers). The free parameters of MAP (degree, coefficients, short-finetune protocol) are acknowledged as practical surrogates rather than first-principles constants, which is appropriate for an empirical compression method.

major comments (2)
  1. The central causal claim—that neglect of heterogeneity is why prior depth pruning fails, and that MAP-guided mixed budgets fix it—depends on MAP’s ranking fidelity. Theorems 3.1, 3.3, 3.5 and Lemma 3.4 (and Appendix F) only guarantee existence of an optimum, uniform polynomial approximation under continuous relaxation (Assumption 3.2), and that a constant bias preserves the maximizer. They do not establish that short-finetune observations (subset data, ~10 epochs, incremental single-layer TE pruning in Algorithms 1–2 / Appendix C) preserve the ranking of full-finetune configurations. Recovery asymmetry (Observation 2 / Fig. 5) and multi-layer interactions could reorder budgets relative to full recovery even when the polynomial fit on cheap data looks reasonable (Table 10, MAE/RMSE ~0.4–0.5). Table 7 shows Stage-1 budget allocation is load-bearing; without a direct check that MAP’s recomm
  2. Figure 7 and the single-type baselines (attention-only vs. activation-only vs. HetDPT) are the main empirical support for the heterogeneity diagnosis, but the paper does not report a controlled comparison against a strong mixed-budget baseline that uses the same total layer budget without MAP (e.g., fixed 50/50 or TE-only mixed selection with full finetuning). Table 7 ablates components of Stage 1 but does not isolate whether MAP’s specific allocation is better than other mixed allocations of the same total k. Without that, it remains possible that any mixed attention+activation pruning plus merging would recover most of the gain, which would weaken the claim that MAP-based heterogeneity-aware allocation is the key contribution rather than activation removal and linear merging alone.
minor comments (5)
  1. Notation for retained ratios (a, t) in Appendix C vs. pruning ratios (˜ma, ˜mg) in the main text is easy to confuse; a short explicit conversion box near Eq. (5) would help.
  2. Figure 3’s dimension-mismatch cartoon is useful but the caption should state more clearly that activation removal is the mechanism that enables safe linear merging, not pruning of Linear1/Linear2 themselves.
  3. Table 1 and Table 2 report throughput on H800; stating batch size and whether TensorRT/cuDNN settings match prior work would improve reproducibility of the speedup claims.
  4. Related-work discussion of DepthShrinker is helpful; a short quantitative “DepthShrinker-style on ViT” baseline number (even in the appendix) would make the claimed gap more concrete.
  5. Typos and minor inconsistencies: “DePTh” capitalization in the method name, occasional “DW Prun.” vs. “WD-Pruning,” and slight baseline accuracy differences between tables (e.g., DeiT-S 79.8 vs. 79.9 in text) should be cleaned.

Circularity Check

0 steps flagged

No significant circularity: MAP is an openly empirical polynomial surrogate fitted to short-finetune observations; theorems only restate standard existence/approximation/bias-invariance facts and do not force the claimed speedups or accuracy recoveries.

full rationale

The paper's load-bearing claims (heterogeneity of attention vs. GELU layers, superiority of mixed budgets, 1.58 imes lossless speedup on DeiT-B, and the HetDPT+ extreme-compression gains) rest on empirical observations (Figs. 4–5, 7; Tables 1–7) and a practical two-stage pipeline, not on a first-principles derivation that reduces to its own inputs. MAP (Eq. 5, Appendix C) is explicitly constructed by least-squares fitting of a low-degree bivariate polynomial to a small set of cheap prune–short-finetune–evaluate points collected by Algorithms 1–2; the subsequent discrete search simply maximises that fitted surface under the linear budget constraint. The three theorems (3.1 existence on a finite set, 3.3 Stone–Weierstrass under continuous-relaxation Assumption 3.2, 3.5/Lemma 3.4 constant-bias invariance of the argmax) are textbook statements that justify using a polynomial surrogate; they do not smuggle the final accuracy numbers or the optimal (ma, mg) counts into the axioms, nor do they claim that short-finetune rankings are identical to full-finetune rankings. No self-citation supplies a uniqueness theorem that forbids alternatives, no ansatz is imported under the authors’ own prior work, and no known empirical pattern is merely renamed. The only mild circular flavour is the ordinary statistical one that any fitted surrogate will look good on the points it was trained on (Table 10 MAE/RMSE ~0.4–0.5); the paper never presents those fitted values as independent predictions of an external quantity. Hence the derivation chain is self-contained against external benchmarks and scores at most 1.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The method rests on standard optimization and approximation facts plus two domain-level modeling choices (continuous relaxation of discrete pruning ratios; sufficiency of short-finetune observations). Free parameters are the polynomial coefficients and the chosen degree; they are fitted per architecture but do not appear inside a claimed closed-form law. No new physical or mathematical entities are postulated.

free parameters (3)
  • MAP polynomial coefficients Θ = {θ_ij}
    Least-squares fitted on the collected (pruning-ratio, accuracy) pairs; degree κ selected by leave-2-out CV. Directly determines the recommended (attention, activation) budget.
  • Polynomial degree κ
    Chosen by L2OCV over {1,2,3,4}; typically κ=2. Controls expressivity of the accuracy surrogate.
  • Fast-finetune length / data subset size for MAP collection
    Typically ~10 epochs on a representative subset; controls the bias/noise of the observations that train MAP.
axioms (4)
  • standard math Stone–Weierstrass: continuous functions on a compact domain can be uniformly approximated by polynomials.
    Invoked to justify the polynomial MAP (Theorem 3.3 / Appendix F.3).
  • domain assumption Continuous relaxation of discrete pruning ratios: the accuracy map on the discrete grid extends to a continuous function on [0,1]².
    Assumption 3.2 / F.6; required for the approximation theorem. Motivated by observed smoothness but not proved.
  • domain assumption Fast-finetune observations equal true accuracy plus constant bias plus zero-mean noise independent of the pruning configuration.
    Assumption F.11; used to prove that the MAP maximizer coincides with the true maximizer (Theorem 3.5).
  • domain assumption Within-type learnable importance scores (propagated through the non-differentiable binary mask) correctly rank layers for removal once the budget per type is fixed.
    Stage-1 Step 2; standard in structured pruning but still an empirical modeling choice.
invented entities (1)
  • Model Accuracy Predictor (MAP) independent evidence
    purpose: Surrogate that maps (attention pruning ratio, activation pruning ratio) to predicted fine-tuned accuracy so the mixed budget can be chosen without exhaustive full training.
    A fitted polynomial regressor, not a new physical object; independent evidence is the leave-2-out validation error and the final ImageNet numbers.

pith-pipeline@v1.1.0-grok45 · 34522 in / 3086 out tokens · 23492 ms · 2026-07-11T23:57:52.846769+00:00 · methodology

0 comments
read the original abstract

While prior studies have successfully compressed vision Transformers (ViTs) through various pruning techniques, most have concentrated on width pruning to achieve significant reductions in model size. Depth pruning, which removes entire layers from a ViT, is notoriously difficult for accuracy recovery despite its potential to deliver higher speedups, limiting the acceleration achieved by existing joint width-and-depth pruning methods. In this work, we reveal that the failure of existing depth pruning methods lies in their neglect of heterogeneity between different layers, and we introduce HetDPT, a heterogeneity-aware depth pruning method that avoids dimension mismatch. Comprehensive experiments on ImageNet-1K, CIFAR-100, COCO, and ADE20K validate our method: HetDPT achieves a 1.58$\times$ speedup for DeiT-B while maintaining accuracy and a 1.39$\times$ speedup for DeiT-S with nearly no accuracy degradation. Furthermore, when combined with width pruning, HetDPT+ sets a new state-of-the-art record in extreme ViT pruning, enhancing the acceleration ratio from 4.24$\times$ to 5.19$\times$ for the Isomorphic-Pruning-2.6G configuration while maintaining near-lossless accuracy; our code is available at https://github.com/Efficient-AI-for-All/HetDPT.

Figures

Figures reproduced from arXiv: 2607.03784 by Han Bao, Kang Zhao, Tao Yuan, Wenxuan Wang, Xianzhi Yu, Zhenfeng Su, Zhongzhe Hu.

Figure 1
Figure 1. Figure 1: Compared with other work, our HetDPT and HetDPT+ offer a state-of-the-art accuracy-speedup Pareto frontier. strated remarkable performance across various domains. However, their large parameter counts and high computa￾tional costs lead to extended inference latency. Structured pruning (He & Xiao, 2023) is effective for model com￾pression and is generally easier to translate into practical speedup on mainst… view at source ↗
Figure 2
Figure 2. Figure 2: Practical inference speed analysis of ViTs. (a)Speedup comparison: Depth pruning exhibits significantly higher speedup efficiency than width pruning. (b)Latency breakdown of ViTs. trolled by a binary mask, rather than as a parameterized layer with trainable weights. In contrast, our Heterogeneity-aware DePTh pruning (HetDPT), which simultaneously prunes distinct layer types while addressing their heterogen… view at source ↗
Figure 3
Figure 3. Figure 3: The visualization of dimension mismatch. Dimension mismatch bottlenecks. In heterogeneous depth pruning, the two most time-consuming layer types in ViTs are generally attention layers and linear layers, which to￾gether account for over 50% of total inference time, as shown in [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Backpropagation gradient magnitudes across attention and activation layers. and activation function layers exhibit significant differ￾ences in gradient scales during backpropagation, which can lead to biased importance estimations and subopti￾mal pruning decisions when employing training-based search strategies. To be specific, we assign each layer in a ViT a learnable importance weight parameter and recor… view at source ↗
Figure 5
Figure 5. Figure 5: (b), after just 10 epochs of fine-tuning, the accuracy of both layer types converges to comparable levels, high￾lighting the rapid recovery capability of activation function layers following pruning. Importantly, this single-layer analysis should not be inter￾preted as evidence that attention-only pruning is optimal. It only characterizes local pruning behavior when all other lay￾ers remain intact. In mult… view at source ↗
Figure 6
Figure 6. Figure 6: Overview of HetDPT: a two-stage depth compression pipeline. removal (Step 2). 1) Step 1 utilizes a model accuracy predic￾tor, to help establish the optimal quantities of attention and activation function layers to prune based on the accuracy recovered after fine-tuning, directly implementing Princi￾ple 2. 2) Importantly, Step 1 focuses solely on determining the pruning quantities for each layer type, avoid… view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of HetDPT and single-layer-type pruning on ImageNet-1K for DeiT-S and DeiT-B. (a) Top-1 accuracy vs. speedup on DeiT-S. (b) Top-1 accuracy vs. speedup on DeiT-B. 4.1. Main Results Accuracy-speedup tradeoff analysis. We compare HetDPT against isolated pruning of attention layers and activation layers using NOSE (Lin et al., 2024). As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Orthogonality: HetDPT boosts inference efficiency by combining token pruning (a) GTP-ViT(b) ToMe. C. More Details About Polynomial Approximation of MAP Roadmap. This appendix details: (i) the data collection procedures used to build the regression set for the MAP; (ii) the pre-transformation that maps discrete layer counts to normalized variables; (iii) the polynomial approximation of MAP with Leave-2-out … view at source ↗
Figure 9
Figure 9. Figure 9: Visualization of cute dog: input image (left) and visualization of attention maps (right) of DeiT-Base. Attention maps with red bounding boxes are the attention layers to be removed. F. Theoretical Foundations of MAP This appendix provides a strengthened and fully rigorous theoretical foundation for Model Accuracy Predictor (MAP). Relative to the main text, we supply additional lemmas on the discreteness, … view at source ↗
Figure 10
Figure 10. Figure 10: Visualization of cute cat: input image (left) and visualization of attention maps (right) of DeiT-Base. Attention maps with red bounding boxes are the attention layers to be removed. where ◦ denotes the function composition operation. We aim to reduce computational overhead through heterogeneous depth pruning of attention layers and redundant GELU activation function layers. That is, f mˆ l (·) := Ll,2 ◦ … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

119 extracted references · 1 canonical work pages

  1. [1]

    Token merging: Your ViT but faster

    Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., and Hoffman, J. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023

    2023

  2. [2]

    Chavan, A., Shen, Z., Liu, Z., Liu, Z., Cheng, K.-T., and Xing, E. P. Vision transformer slimming: Multi-dimension searching in continuous optimization space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 4931--4941, June 2022

    2022

  3. [3]

    Transformer interpretability beyond attention visualization

    Chefer, H., Gur, S., and Wolf, L. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 782--791, 2021

    2021

  4. [4]

    R., Fan, Q., and Panda, R

    Chen, C.-F. R., Fan, Q., and Panda, R. CrossViT : Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 357--366, 2021 a

    2021

  5. [5]

    Diffrate: Differentiable compression rate for efficient vision transformers

    Chen, M., Shao, W., Xu, P., Lin, M., Zhang, K., Chao, F., Ji, R., Qiao, Y., and Luo, P. Diffrate: Differentiable compression rate for efficient vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 17164--17174, 2023 a

    2023

  6. [6]

    Chasing sparsity in vision transformers: An end-to-end exploration, 2021 b

    Chen, T., Cheng, Y., Gan, Z., Yuan, L., Zhang, L., and Wang, Z. Chasing sparsity in vision transformers: An end-to-end exploration, 2021 b

    2021

  7. [7]

    SparseViT : Revisiting activation sparsity for efficient high-resolution vision transformer

    Chen, X., Liu, Z., Tang, H., Yi, L., Zhao, H., and Han, S. SparseViT : Revisiting activation sparsity for efficient high-resolution vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 2061--2070, 2023 b

    2061

  8. [8]

    The stone-weierstrass theorem

    De Branges, L. The stone-weierstrass theorem. Proceedings of the American Mathematical Society, 10 0 (5): 0 822--824, 1959

    1959

  9. [9]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 248--255, 2009. doi:10.1109/CVPR.2009.5206848

    doi:10.1109/cvpr.2009.5206848 2009

  10. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021

    2021

  11. [14]

    and Chong, E

    Ganguli, T. and Chong, E. K. P. Activation-based pruning of neural networks. Algorithms, 17 0 (1): 0 48, 2024

    2024

  12. [16]

    Transformer in transformer

    Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., and Wang, Y. Transformer in transformer. Advances in Neural Information Processing Systems, 34: 0 15908--15919, 2021

    2021

  13. [18]

    and Xiao, L

    He, Y. and Xiao, L. Structured pruning for deep convolutional neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46 0 (5): 0 2900--2919, 2023

    2023

  14. [20]

    Learning multiple layers of features from tiny images

    Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  15. [21]

    EfficientFormer : Vision transformers at mobilenet speed

    Li, Y., Yuan, G., Wen, Y., Hu, J., Evangelidis, G., Tulyakov, S., Wang, Y., and Ren, J. EfficientFormer : Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems, 35: 0 12934--12949, 2022

    2022

  16. [22]

    Not all patches are what you need: Expediting vision transformers via token reorganizations, 2022

    Liang, Y., Ge, C., Tong, Z., Song, Y., Wang, J., and Xie, P. Not all patches are what you need: Expediting vision transformers via token reorganizations, 2022. URL https://arxiv.org/abs/2202.07800

    Pith/arXiv arXiv 2022

  17. [23]

    Mlp can be a good transformer learner

    Lin, S., Lyu, P., Liu, D., Tang, T., Liang, X., Song, A., and Chang, X. Mlp can be a good transformer learner. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 19489--19498, 2024. doi:10.1109/CVPR52733.2024.01843

    doi:10.1109/cvpr52733.2024.01843 2024

  18. [24]

    J., Bourdev, L

    Lin, T.-Y., Maire, M., Belongie, S. J., Bourdev, L. D., Girshick, R. B., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft COCO : Common objects in context. CoRR, abs/1405.0312, 2014. URL http://arxiv.org/abs/1405.0312

    Pith/arXiv arXiv 2014

  19. [25]

    TB-STC : Transposable block-wise N:M structured sparse tensor core

    Liu, J., Zeng, S., Zhao, J., Ding, L., Wang, Z., Li, J., Zhu, Z., Ning, X., Zhang, C., Wang, Y., et al. TB-STC : Transposable block-wise N:M structured sparse tensor core. In 2025 IEEE International Symposium on High Performance Computer Architecture, pp.\ 949--962. IEEE, 2025

    2025

  20. [26]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 10012--10022, 2021

    2021

  21. [28]

    Mao, H., Han, S., Pool, J., Li, W., Liu, X., Wang, Y., and Dally, W. J. Exploring the granularity of sparsity in convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp.\ 13--20, 2017

    2017

  22. [29]

    ShortGPT : Layers in large language models are more redundant than you expect

    Men, X., Xu, M., Zhang, Q., Yuan, Q., Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. ShortGPT : Layers in large language models are more redundant than you expect. In Findings of the Association for Computational Linguistics: ACL 2025, pp.\ 20192--20204, 2025

    2025

  23. [30]

    Speedup deep learning models on gpu by taking advantage of efficient unstructured pruning and bit-width reduction

    Pietro \'n , M., \.Z urek, D., and \'S nie \.z y \'n ski, B. Speedup deep learning models on gpu by taking advantage of efficient unstructured pruning and bit-width reduction. Journal of Computational Science, 67: 0 101971, 2023

    2023

  24. [31]

    DynamicViT : Efficient vision transformers with dynamic token sparsification

    Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., and Hsieh, C.-J. DynamicViT : Efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems, 2021

    2021

  25. [32]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models, 2024. URL https://arxiv.org/abs/2306.11695

    Pith/arXiv arXiv 2024

  26. [33]

    Going deeper with image transformers

    Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., and J \'e gou, H. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 32--42, 2021

    2021

  27. [34]

    Regression analysis

    Tyagi, K., Rane, C., Manry, M., et al. Regression analysis. In Artificial Intelligence and Machine Learning for EDGE Computing, pp.\ 53--63. Elsevier, 2022

    2022

  28. [35]

    PVT v2 : Improved baselines with pyramid vision transformer

    Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. PVT v2 : Improved baselines with pyramid vision transformer. Computational Visual Media, 8 0 (3): 0 415--424, 2022

    2022

  29. [36]

    Joint token pruning and squeezing towards more aggressive compression of vision transformers

    Wei, S., Ye, T., Zhang, S., Tang, Y., and Liang, J. Joint token pruning and squeezing towards more aggressive compression of vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 2092--2101, 2023

    2092

  30. [39]

    GTP-ViT : Efficient vision transformers via graph-based token propagation

    Xu, X., Wang, S., Chen, Y., Zheng, Y., Wei, Z., and Liu, J. GTP-ViT : Efficient vision transformers via graph-based token propagation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.\ 86--95, 2024 b

    2024

  31. [40]

    Evo-ViT : Slow-fast token evolution for dynamic vision transformer

    Xu, Y., Zhang, Z., Zhang, M., Sheng, K., Li, K., Dong, W., Zhang, L., Xu, C., and Sun, X. Evo-ViT : Slow-fast token evolution for dynamic vision transformer. Proceedings of the AAAI Conference on Artificial Intelligence, 36 0 (3): 0 2964--2972, 2022. doi:10.1609/aaai.v36i3.20202

    doi:10.1609/aaai.v36i3.20202 2022

  32. [41]

    Global vision transformer pruning with hessian-aware saliency

    Yang, H., Yin, H., Shen, M., Molchanov, P., Li, H., and Kautz, J. Global vision transformer pruning with hessian-aware saliency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 18547--18557, June 2023

    2023

  33. [42]

    Width & depth pruning for vision transformers

    Yu, F., Huang, K., Wang, M., Cheng, Y., Chu, W., and Cui, L. Width & depth pruning for vision transformers. Proceedings of the AAAI Conference on Artificial Intelligence, 36 0 (3): 0 3143--3151, 2022 a . doi:10.1609/aaai.v36i3.20222

    work page doi:10.1609/aaai.v36i3.20222 2022

  34. [43]

    Unified visual transformer compression

    Yu, S., Chen, T., Shen, J., Yuan, H., Tan, J., Yang, S., Liu, J., and Wang, Z. Unified visual transformer compression. In International Conference on Learning Representations, 2022 b

    2022

  35. [45]

    SAViT : Structure-aware vision transformer pruning via collaborative optimization

    Zheng, C., Li, Z., Zhang, K., Yang, Z., Tan, W., Xiao, J., Ren, Y., and Pu, S. SAViT : Structure-aware vision transformer pruning via collaborative optimization. In Advances in Neural Information Processing Systems, volume 35, pp.\ 9010--9023. Curran Associates, Inc., 2022

    2022

  36. [46]

    Scene parsing through ade20k dataset

    Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 5122--5130, 2017. doi:10.1109/CVPR.2017.544

    doi:10.1109/cvpr.2017.544 2017

  37. [47]

    Vision transformer pruning, 2021

    Zhu, M., Tang, Y., and Han, K. Vision transformer pruning, 2021. URL https://arxiv.org/abs/2104.08500

    Pith/arXiv arXiv 2021

  38. [48]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=

    Structured Pruning for Deep Convolutional Neural Networks: A Survey , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=. 2023 , publisher=

    2023

  39. [49]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages=

    Exploring the Granularity of Sparsity in Convolutional Neural Networks , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages=

  40. [50]

    arXiv preprint arXiv:2406.12837 , year=

    LayerMerge: Neural Network Depth Compression Through Layer Pruning and Merging , author=. arXiv preprint arXiv:2406.12837 , year=

    Pith/arXiv arXiv

  41. [51]

    International Conference on Machine Learning , pages=

    Efficient Latency-Aware CNN Depth Compression via Two-Stage Dynamic Programming , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  42. [52]

    arXiv preprint arXiv:2106.09309 , year=

    Layer Folding: Neural Network Depth Reduction Using Activation Linearization , author=. arXiv preprint arXiv:2106.09309 , year=

    Pith/arXiv arXiv

  43. [53]

    Langley , title=

    P. Langley , title=. Proceedings of the 17th International Conference on Machine Learning , address=. 2000 , pages=

    2000

  44. [54]

    T. M. Mitchell , title=. 1980 , address=

    1980

  45. [55]

    M. J. Kearns , title=

  46. [56]

    I , publisher=

    Machine Learning: An Artificial Intelligence Approach, Vol. I , publisher=. 1983 , address=

    1983

  47. [57]

    R. O. Duda and P. E. Hart and D. G. Stork , title=

  48. [58]

    Suppressed for Anonymity , author=

  49. [59]

    Newell and P

    A. Newell and P. S. Rosenbloom , title=. Cognitive Skills and Their Acquisition , pages=. 1981 , editor=

    1981

  50. [60]

    A. L. Samuel , title=. IBM Journal of Research and Development , year=

  51. [61]

    and Kaiser,

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser,. Attention Is All You Need , year=. Proceedings of the 31st International Conference on Neural Information Processing Systems , pages=

  52. [62]

    International Conference on Learning Representations , year=

    An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale , author=. International Conference on Learning Representations , year=

  53. [63]

    International Conference on Machine Learning , year=

    Training Data-Efficient Image Transformers and Distillation Through Attention , author=. International Conference on Machine Learning , year=

  54. [64]

    Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction Without Convolutions , year=

    Wang, Wenhai and Xie, Enze and Li, Xiang and Fan, Deng-Ping and Song, Kaitao and Liang, Ding and Lu, Tong and Luo, Ping and Shao, Ling , booktitle=. Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction Without Convolutions , year=

  55. [65]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  56. [66]

    2112.01526 , archivePrefix=

    Li, Yanghao and Wu, Chao-Yuan and Fan, Haoqi and Mangalam, Karttikeya and Xiong, Bo and Malik, Jitendra and Feichtenhofer, Christoph , year=. 2112.01526 , archivePrefix=

    Pith/arXiv arXiv

  57. [67]

    and Tan, Mingxing , booktitle=

    Dai, Zihang and Liu, Hanxiao and Le, Quoc V. and Tan, Mingxing , booktitle=

  58. [68]

    2021 , eprint=

    Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention , author=. 2021 , eprint=

    2021

  59. [69]

    2022 , eprint=

    MixFormer: Mixing Features Across Windows and Dimensions , author=. 2022 , eprint=

    2022

  60. [70]

    2107.00652 , archivePrefix=

    Dong, Xiaoyi and Bao, Jianmin and Chen, Dongdong and Zhang, Weiming and Yu, Nenghai and Yuan, Lu and Chen, Dong and Guo, Baining , year=. 2107.00652 , archivePrefix=

    Pith/arXiv arXiv

  61. [71]

    2204.03645 , archivePrefix=

    Ding, Mingyu and Xiao, Bin and Codella, Noel and Luo, Ping and Wang, Jingdong and Yuan, Lu , year=. 2204.03645 , archivePrefix=

    Pith/arXiv arXiv

  62. [72]

    2203.16768 , archivePrefix=

    Kim, Namyup and Kim, Dongwon and Lan, Cuiling and Zeng, Wenjun and Kwak, Suha , year=. 2203.16768 , archivePrefix=

    Pith/arXiv arXiv

  63. [73]

    2022 , eprint=

    Stratified Transformer for 3D Point Cloud Segmentation , author=. 2022 , eprint=

    2022

  64. [74]

    , title=

    Chavan, Arnav and Shen, Zhiqiang and Liu, Zhuang and Liu, Zechun and Cheng, Kwang-Ting and Xing, Eric P. , title=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , month=. 2022 , pages=

    2022

  65. [75]

    2021 , eprint=

    Vision Transformer Pruning , author=. 2021 , eprint=

    2021

  66. [76]

    International Conference on Learning Representations , year=

    Yu, Shixing and Chen, Tianlong and Shen, Jiayi and Yuan, Huan and Tan, Jianchao and Yang, Sen and Liu, Ji and Wang, Zhangyang , title=. International Conference on Learning Representations , year=

  67. [77]

    Zheng, Chuanyang and Li, Zheyang and Zhang, Kai and Yang, Zhi and Tan, Wenming and Xiao, Jun and Ren, Ye and Pu, Shiliang , booktitle=

  68. [78]

    MLP Can Be a Good Transformer Learner , year=

    Lin, Sihao and Lyu, Pumeng and Liu, Dongrui and Tang, Tao and Liang, Xiaodan and Song, Andy and Chang, Xiaojun , booktitle=. MLP Can Be a Good Transformer Learner , year=

  69. [79]

    arXiv preprint arXiv:2206.00843 , year=

    DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks , author=. arXiv preprint arXiv:2206.00843 , year=

    Pith/arXiv arXiv

  70. [80]

    2021 , eprint=

    Chasing Sparsity in Vision Transformers: An End-to-End Exploration , author=. 2021 , eprint=

    2021

  71. [81]

    2301.00774 , archivePrefix=

    Frantar, Elias and Alistarh, Dan , year=. 2301.00774 , archivePrefix=

    Pith/arXiv arXiv

  72. [82]

    2024 , eprint=

    A Simple and Effective Pruning Approach for Large Language Models , author=. 2024 , eprint=

    2024

  73. [83]

    2302.01172 , archivePrefix=

    Lu, Yucheng and Agrawal, Shivani and Subramanian, Suvinay and Rybakov, Oleg and De Sa, Christopher and Yazdanbakhsh, Amir , year=. 2302.01172 , archivePrefix=

    Pith/arXiv arXiv

  74. [84]

    Sabry and Yang, Xulei and Wu, Min and Li, Xiaoli and Lin, Weisi , year=

    Xu, Kaixin and Wang, Zhe and Chen, Chunyun and Geng, Xue and Lin, Jie and Aly, Mohamed M. Sabry and Yang, Xulei and Wu, Min and Li, Xiaoli and Lin, Weisi , year=. 2407.02068 , archivePrefix=

    Pith/arXiv arXiv

  75. [85]

    Beyond 2:4: Exploring

    Zhao, Kang and Yuan, Tao and Bao, Han and Su, Zhenfeng and Gao, Chang and Sun, Zhaofeng and Liang, Zichen and Jing, Liping and Chen, Jianfei , year=. Beyond 2:4: Exploring. 2410.16135 , archivePrefix=

    Pith/arXiv arXiv

  76. [86]

    Yuan, Li and Chen, Yunpeng and Wang, Tao and Yu, Weihao and Shi, Yujun and Tay, Francis E. H. and Feng, Jiashi and Yan, Shuicheng , journal=. Tokens-to-Token. 2021 , pages=

    2021

  77. [87]

    2022 , eprint=

    Not All Patches Are What You Need: Expediting Vision Transformers via Token Reorganizations , author=. 2022 , eprint=

    2022

  78. [88]

    Rao, Yongming and Zhao, Wenliang and Liu, Benlin and Lu, Jiwen and Zhou, Jie and Hsieh, Cho-Jui , booktitle=

  79. [89]

    Proceedings of the 36th International Conference on Machine Learning , series=

    Guan, Chaoyu and Wang, Xiting and Zhang, Quanshi and Chen, Runjin and He, Di and Xie, Xing , title=. Proceedings of the 36th International Conference on Machine Learning , series=

  80. [90]

    2019 , eprint=

    Mean Field Analysis of Neural Networks: A Law of Large Numbers , author=. 2019 , eprint=

    2019

Showing first 80 references.