Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SAPER prunes attention heads one by one, beating block pruning on FLOPs

desk verdict A useful pruning recipe with a nice visualization, but the headline efficiency claim rests on FLOPs numbers for RAPTOR that don't add up. read the letter →

arxiv 2608.00264 v1 pith:FSXC4QAR submitted 2026-07-31 cs.CV cs.AI

Kamil Książek , Piotr Suszyński , Micha{l} Jan W{l}odarczyk , Jacek Tabor , Przemys{l}aw Biecek This is my paper · ORCID
classification cs.CVcs.AI
keywords attentionheadpruningvisiontransformersDINOv2spectralclusteringLaplacemapsLapSumsofttop-kinterpretabilityFLOPsreduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the attention heads of a DINOv2 vision transformer are not uniformly redundant but fall into functional clusters that repeat across layers, so heads can be removed one at a time rather than in contiguous blocks. To exploit this, it introduces SAPER, a differentiable pruning method that learns a soft top-k mask over all heads and then fine-tunes or distills the surviving subset. On ImageNet-1K and CIFAR-100, SAPER keeps classification accuracy within a few points of the unpruned model while cutting FLOPs well below the unpruned model and below the block-recurrent RAPTOR baseline at matched head counts. A sympathetic reader would care because it suggests a practical path to smaller vision foundation models guided by what each head actually computes.

What carries the argument

The key machinery is threefold: Laplace maps, which render the Fiedler vector and next two eigenvectors of each head's normalized attention-graph Laplacian as an RGB image; spectral clustering of head activation statistics (Pearson correlation on maximum-attention features) to expose cross-depth functional redundancy; and the LapSum soft top-k selector, a differentiable relaxation of hard top-k that lets a continuous budget k be annealed to pick exactly k heads. The first two justify which heads are redundant; the third translates that into a trainable pruning decision.

What would settle it

Compute FLOPs for a DINOv2 ViT-B pruned to 24 heads using a standard FLOPs counter at the paper's input resolution, for both SAPER's selected heads and RAPTOR's recurrent block configuration. If RAPTOR's true FLOPs is below SAPER's (or even below the unpruned model's 46.33 GFLOPs), the paper's efficiency-advantage claim over RAPTOR is false. Alternatively, reproduce the 61.86 GFLOPs figure for RAPTOR's 24-head configuration; if it cannot be reproduced, the baseline comparison is not legitimate.

Watch

Extended reading notes

Core claim

The central discovery is that the normalized-Laplacian spectrum of an attention map gives a label-free signature of a head's function, and that these signatures cluster into a small number of groups that recur at different depths of the network. Building on this, SAPER uses the LapSum soft top-k operator to select any prespecified number of heads in an end-to-end differentiable way. With as few as 6 of 144 heads in DINOv2 ViT-B, distilled SAPER retains 72.97% ImageNet-1K top-1 accuracy (vs. 84.14% unpruned) at 30.05 GFLOPs; at 24 heads it reaches 80.58% at 32.17 GFLOPs, roughly half the FLOPs of the RAPTOR baseline at the same head count.

Load-bearing premise

The main quantitative claim that SAPER outperforms RAPTOR in FLOPs reduction assumes the FLOPs numbers for RAPTOR in Table 2 were computed under the same counting conventions and input settings as SAPER's; in particular, RAPTOR's pruned ViT-B at 24 heads is listed at 61.86 GFLOPs, which is higher than the unpruned model's 46.33 GFLOPs, so the comparison is only meaningful if the two sets of numbers are commensurable.

Editorial extensions

If this is right

  • A user can choose any desired head count between 1 and the full model, enabling a smooth accuracy-compute trade-off rather than block-multiple jumps.
  • Pruning to 24 heads cuts GFLOPs from 46.33 to 32.17 in ViT-B while losing under 4 points of ImageNet-1K accuracy in distilled mode.
  • Interpretability guides the selection: surviving heads are spread across early-layer 'convolutional' roles and late-layer object-level roles, so the pruned model keeps a diversity of functions.
  • Attention heads can stay frozen during fine-tuning or distillation, which reduces training cost and worked better than adjusting them.
  • The head-level clustering confirms and refines the layer-level block structure of ViTs, showing that functional recurrence happens within individual heads across depth.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the spectral signature transfers across architectures and datasets, the same clustering could pre-initialize pruning for new ViT variants without extra labeled data.
  • The clustering may serve as a testable proxy for functional redundancy: ablating heads from the same cluster one at a time should cause similar accuracy drops, which could be verified empirically.
  • Token-level pruning, which the paper mentions as future work, could be combined with head pruning to yield further FLOPs savings if the spectral lens extends to token importance.
  • The FLOPs advantage over RAPTOR at matched head counts depends on counting conventions; if RAPTOR's figures are measured with unrolled recurrent depth or a different input resolution, the gap may be overstated or reversed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SAPER, a differentiable attention-head pruning framework for DINOv2 vision transformers, together with two interpretability analyses: semantic clustering of attention heads based on Pearson correlation of per-image maximal CLS-attention values, and 'Laplace maps' that visualize attention heads through eigenvectors of the normalized Laplacian of the attention graph. The authors claim that SAPER achieves a favorable accuracy-efficiency trade-off, outperforming the RAPTOR baseline in FLOPs reduction while retaining competitive classification accuracy on ImageNet-1K and CIFAR-100. The paper reports experiments for ViT-S and ViT-B, comparing fine-tuning and knowledge-distillation modes against RAPTOR across head budgets k=6,12,24,36,48. The central quantitative claim is that SAPER reduces FLOPs by roughly a factor of two relative to RAPTOR at matching head counts (e.g., 32.17 vs. 61.86 GFLOPs for ViT-B at k=24).

Significance. If the efficiency numbers are correct, the paper offers a flexible alternative to block-level recurrent pruning, with detailed per-head budgets and an interpretability-oriented analysis of attention-head structure. The manuscript is transparent in reporting averages over multiple runs and extensive hyperparameter search details in the appendix. The spectral visualization of attention heads is a plausible novel contribution. However, the headline comparison with RAPTOR rests entirely on FLOPs values in Table 2 that are internally inconsistent: RAPTOR's reported pruned FLOPs exceed the unpruned DINOv2 baseline. This is a load-bearing issue for the abstract's central claim. In addition, the claimed connection between the interpretability analyses and the actual pruning mechanism is not demonstrated. These concerns make the current version unsuitable without substantial revision.

major comments (4)
  1. [§4.2, Table 2] The central claim that SAPER 'outperforms RAPTOR in FLOPs reduction' is unsupported because the RAPTOR FLOPs numbers in Table 2 are not credible under any consistent inference protocol. For ViT-B, the unpruned DINOv2 baseline is listed as 46.33 GFLOPs, but RAPTOR at k=24,36,48 is listed as 61.86 GFLOPs; for ViT-S, RAPTOR is listed as 16.15 GFLOPs at all k, while the unpruned model is 12.25 GFLOPs. A pruned model should not use more compute than the full model, and identical values across different k strongly suggest the numbers were not recomputed for the pruned architectures. The paper's own comparison — '32.17 vs. 61.86 GFLOPs for ViT-B at k=24' — and the abstract's efficiency claim depend on this number. The authors must either provide a reproducible FLOPs measurement protocol under identical conditions (input size, model definition, unrolling, inclusion of overhead) or withdraw the a
  2. [§3.2] The normalized Laplacian of an attention map is asserted to be a symmetric positive semi-definite matrix, but the attention map is a directed weighted graph (row-stochastic after softmax, or a logit matrix before softmax). For a directed graph, the standard normalized Laplacian is not symmetric PSD, so eigenvectors and the Fiedler vector are not generally well-defined real vectors. The paper does not describe any symmetrization step (e.g., L=(A+A^T)/2 or degree-normalized symmetrization). This needs to be fixed or the Laplace-map construction needs to be redefined, especially since the authors call the Fiedler vector the 'real-valued solution to the minimum cut problem,' which itself is only a relaxation.
  3. [§3.3, Eq. (2)] The paper's title and Section 2.4 describe Laplace maps as 'a signature that also serves as a pruning criterion,' and the introduction states that interpretability insights are leveraged by SAPER. However, the differentiable selection mechanism in Section 3.3 uses LapSum on a score vector r (Eq. 2), and the paper never defines r in terms of the spectral or clustering features. The scores appear to be learnable parameters optimized end-to-end. As written, the interpretability analyses of Sections 3.1–3.2 are not functionally connected to the pruning algorithm; they are exploratory studies. The authors should either specify how the spectral/semantic features enter the selection score, or temper the 'interpretability-guided' framing and the claim that Laplace maps serve as a pruning criterion.
  4. [§4.2, Table 1] The comparison with RAPTOR is also complicated by evaluation-protocol differences. For ImageNet-1K, RAPTOR accuracy is quoted from the original paper, while SAPER uses either a linear probe (fine-tune mode) or a kNN classifier (distillation mode). The authors do not state whether these protocols match RAPTOR's evaluation. At k=24 on ViT-B ImageNet-1K, RAPTOR reports 81.2 and SAPER distilled reports 80.58, so the headline is not that SAPER is more accurate, but that it uses fewer FLOPs. Given that the FLOPs values are questionable, the overall accuracy-efficiency advantage over RAPTOR is currently not demonstrated. A standardized comparison, ideally re-running RAPTOR under the same measurement protocol, is needed.
minor comments (5)
  1. [Figure 2] The captions use 'DB=' and 'Sil=' without defining them. Please spell out Davies-Bouldin index and Silhouette score in the caption or text.
  2. [§4.2] The text refers to 'layers L0-L7 and L8-L12'; a 12-layer ViT has layers L0 through L11, so the second interval should be L8-L11. The same issue appears in the appendix discussions of 'final two layers' and 'final three layers.'
  3. [Table 1] Several entries contain the typographical artifact '✗−✗' (e.g., RAPTOR24 ViT-S ImageNet-1K). This should be cleaned to a single '✗' or an explicit N/A.
  4. [§3.2] The statement that the Fiedler vector 'represents the real-valued solution to the minimum cut problem' is imprecise. It is the solution to a spectral relaxation of the normalized cut, not the exact solution. Please rephrase.
  5. [Appendix E] The hyperparameter tables are extensive, but the paper does not provide code or a repository link. Given the volume of tuning details, a public implementation would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SAPER's pruning results are measured post-training; the self-cited LapSum is a tool dependency, not an input-equivalent prediction.

full rationale

The paper's central claims are empirical: Table 1 reports ImageNet-1K/CIFAR-100 accuracies of SAPER after fine-tuning or distillation, and Table 2 reports FLOPs computed for the pruned architectures. These numbers are not derived from a fitted constant or from a prediction that reduces to an input; the selected-head scores are learned by gradient descent on the actual downstream loss, and accuracy is evaluated after hard-top-k retraining. The semantic clustering and Laplace maps are exploratory and do not seed or constrain the LapSum selection scores in a way that would make the accuracy outcome tautological. The one self-citation, LapSum [34] (Struski et al., including co-author Tabor), provides the differentiable top-k relaxation used as a component; citing it is a tool dependency, not the evidence for the central efficiency claim, and no 'uniqueness' or ansatz is imported from it. The apparent anomaly that RAPTOR's FLOPs exceed the unpruned model (e.g., 61.86 vs 46.33 GFLOPs in Table 2) is a measurement/comparability concern about the external baseline, not a circular derivation, and is outside the circularity definition.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities; Laplace maps are a visualization, and SAPER is a training procedure. The free parameters are the Optuna-tuned hyperparameters that the reported results depend on; they are fitted to validation accuracy and vary by dataset and model, which limits the generality of the reported trade-offs. The axioms include standard spectral methods plus ad hoc modeling choices about how attention maps are symmetrized and summarized, and a domain assumption inherited from RAPTOR's block-structure hypothesis.

free parameters (7)
  • LapSum alpha_init = e.g., 3.56261 (ViT-S fine-tune, ImageNet-1K); varies per scenario
    Initial negative temperature of the soft top-k; tuned with Optuna per dataset/model; directly affects which heads are selected.
  • LapSum alpha_final = e.g., 2.283e-3
    Final temperature after annealing; tuned per scenario.
  • alpha_anneal_epochs = e.g., 9
    Number of epochs over which alpha is annealed; tuned.
  • k_warmup_epochs = e.g., 4
    Epochs used to linearly reduce the retained head count k from all heads to the target; tuned.
  • backbone_lr = e.g., 2.884e-5
    Learning rate for trainable backbone parameters; tuned.
  • selector_lr = e.g., 4.489e-2
    Learning rate for the LapSum selector scores; tuned.
  • lr = e.g., 1.062e-3
    Learning rate for the linear classifier or student projections; tuned.
assumptions (5)
  • standard math Eigenvalues/eigenvectors of a graph Laplacian and the Fiedler vector approximate the minimum cut (Cheeger inequality).
    Used to justify Laplace maps in §3.2; this is a standard spectral graph theory result.
  • ad hoc to paper Attention maps can be treated as weighted undirected graphs to compute a symmetric positive semi-definite normalized Laplacian.
    Attention matrices are directed; the paper does not specify symmetrization or a directed Laplacian construction, yet claims a PSD Laplacian in §3.2.
  • ad hoc to paper Semantic head similarity can be captured by Pearson correlation of per-image maximum CLS attention to patches.
    Used in §3.1 for clustering; the maximum of the CLS attention vector is a strong summary statistic that is not justified as sufficient for functional similarity.
  • domain assumption The block-recurrent structure of ViTs hypothesized by RAPTOR is real and can be detected at head level.
    The paper builds on RAPTOR's findings (§2.3, §4.2) and uses its own clustering to 'confirm' the phases, but does not independently establish that the block structure exists beyond correlation patterns.
  • domain assumption Pruned heads are redundant across the whole dataset and can be removed without catastrophic loss after fine-tuning or distillation.
    Standard pruning premise; supported by experiments but not proven for all heads or datasets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers." pith.science (2026). https://pith.science/paper/FSXC4QAR

@misc{pith2026260800264,
  author       = {Pith},
  title        = {Pith review of: Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FSXC4QAR}},
  note         = {Machine review of arXiv:2608.00264}
}
read the original abstract

Vision foundation models, such as DINOv2, learn highly expressive representations but rely on massive, opaque architectures that demand substantial computational power and memory. To provide an interpretable-guided and efficient solution to this issue, we first propose a spectral analysis and new visualization technique for individual attention heads based on the Laplacian eigenvectors of their attention maps. Building upon recent observations regarding the block structure of Vision Transformers, we perform semantic clustering of attention heads and identify functional redundancies. Leveraging these insights, we introduce SAPER (Soft Attention PrunER), an end-to-end differentiable pruning framework based on the LapSum Soft Top-K approach. Extensive experiments on ImageNet-1K demonstrate that SAPER achieves a highly favorable accuracy-efficiency trade-off, outperforming the competitive RAPTOR baseline in FLOPs reduction while preserving strong classification performance.

Figures

Figures reproduced from arXiv: 2608.00264 by the authors.

Figure 1
Figure 1. The pipeline of SAPER . Green borders around a subset of heads correspond to Top-K heads selected with LapSum and retained for Hard Top-K fine-tuning/distillation. others) influencing target class identification are preserved while the remaining ones are pruned, according to layer-wise pruning rates. Visual explanation of masks is derived through the LRP-based method [1, 4]. Sequence reduction methods such as ToSA [… view at source ↗
Figure 2
Figure 2. Results of the ViT structure analysis with head activations on the ImageNet-1K dataset [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Most of the subspace area with ViT-B heads contains ViT-S heads in their close neighborhood. It suggests [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: DINOv2 Head Laplacian visualization. The (left) image displays the full Laplacian visualization map extracted from a DINOv2 ViT-B model over all layers and heads. On the (right) image a specific magnified region of the generated feature map (b) is compared alongside it…
Figure 5
Figure 5. Figure 5: Laplace visualization maps for selected ViT-S and ViT-B models trained on the ImageNet-1K dataset. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: The study on the influence of the number of retained heads on classification accuracy for ImageNet-1K and [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Laplace maps for the two selected samples of the ImageNet-1K dataset for all attention heads of ViT-S (left [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Laplace maps for the two selected samples of the ImageNet-1K dataset for all attention heads of ViT-S (left [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 17
Figure 17. Figure 17: 19 [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 9
Figure 9. Figure 9: Laplace visualization maps for selected ViT-S and ViT-B models trained on the ImageNet-1K dataset, for an [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Semantic clustering of ViT-S attention heads for the CIFAR-100 dataset (2, 3 and 4 clusters). We can see [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Semantic clustering of ViT-S attention heads for the ImageNet-1K dataset (2, 3 and 4 clusters). The resulting [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Semantic clustering of ViT-B attention heads for the CIFAR-100 dataset (2, 3 and 4 clusters). Separation of [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Semantic clustering of ViT-B attention heads for the ImageNet-1K dataset (2, 3 and 4 clusters). These [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: Extended results of the Principal Component Analysis performed on a common set of heads’ activations [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Classification accuracy for consecutive hyperparameters during ImageNet-1K optimization with Optuna. [PITH_FULL_IMAGE:figures/full_fig_p028_15.png]
Figure 16
Figure 16. Figure 16: Classification accuracy for consecutive hyperparameters during ImageNet-1K optimization with Optuna. [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: Classification accuracy for consecutive hyperparameters during ImageNet-1K optimization with Optuna. [PITH_FULL_IMAGE:figures/full_fig_p030_17.png]
Figure 18
Figure 18. Figure 18: Classification accuracy for consecutive hyperparameters during ImageNet-1K optimization with Optuna. The [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 3 canonical work pages

  1. [1]

    In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J

    Abnar, S., Zuidema, W.: Quantifying attention flow in transformers. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 4190–4197. Association for Computational Linguistics, Online (Jul 2020). https://doi.org/10.18653/v1/2020.acl- main.385,https://aclan...

    doi:10.18653/v1/2020.acl- 2020

  2. [2]

    In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)

    Abnar, S., Zuidema, W.: Quantifying attention flow in transformers. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). pp. 4190–4197. Association for Computational Linguistics (2020)

    2020

  3. [3]

    In: Proceedings of the 41st International Conference on Machine Learning (ICML)

    Achtibat, R., Hatefi, S.M.V ., Dreyer, M., Jain, A., Wiegand, T., Lapuschkin, S., Samek, W.: AttnLRP: Attention- aware layer-wise relevance propagation for transformers. In: Proceedings of the 41st International Conference on Machine Learning (ICML). Proceedings of Machine Learning Research, vol. 235, pp. 135–168. PMLR (2024)

    2024

  4. [4]

    PLOS ONE10(7), 1–46 (07 2015)

    Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE10(7), 1–46 (07 2015). https://doi.org/10.1371/journal.pone.0130140,https://doi.org/10.1371/journal.pone.0130140

    doi:10.1371/journal.pone.0130140 2015

  5. [5]

    arXiv preprint arXiv:2606.01189 (2026)

    Biecek, P., Longo, L., Zhou, J., Fel, T., Holzinger, A., Samek, W.: The case for model science: Verify, explore, steer, refine. arXiv preprint arXiv:2606.01189 (2026)

    arXiv 2026

  6. [6]

    Binkowski, J., Janiak, D., Sawczyn, A., Gabrys, B., Kajdanowicz, T.J.: Hallucination detection in LLMs using spectral features of attention maps (2025),https://openreview.net/forum?id=tm5JQTpBhj

    2025

  7. [7]

    In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV)

    Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 9630–9640 (2021)

    2021

  8. [8]

    Chavan, A., Shen, Z., Liu, Z., Liu, Z., Cheng, K.T., Xing, E.: Vision transformer slimming: Multi-dimension searching in continuous optimization space (2022)

    2022

Show all 41 references
  1. [9]

    In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Chefer, H., Gur, S., Wolf, L.: Transformer interpretability beyond attention visualization. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 782–791 (2021)

    2021

  2. [10]

    In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K

    Chuanyang, Z., Li, Z., Zhang, K., Yang, Z., Tan, W., Xiao, J., Ren, Y ., Pu, S.: SA Vit: Structure-aware vision trans- former pruning via collaborative optimization. In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K. (eds.) Advances in Neural Information Processing Systems (2022...

    2022

  3. [11]

    In: The Twelfth International Conference on Learning Representations (2024),https://openreview.net/forum?id=2dnO3LLiJ1

    Darcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. In: The Twelfth International Conference on Learning Representations (2024),https://openreview.net/forum?id=2dnO3LLiJ1

    2024

  4. [12]

    In: 2009 IEEE conference on computer vision and pattern recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. IEEE (2009)

    2009

  5. [13]

    org/abs/1702.08608

    Doshi-Velez, F., Kim, B.: Towards a rigorous science of interpretable machine learning (2017),https://arxiv. org/abs/1702.08608

    2017 arXiv

  6. [14]

    In: International Conference on Learning Representations (ICLR) (2021)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on L...

    2021

  7. [15]

    In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2026), https://openreview.net/forum?id=wy2FwyNYro

    Erel, Y ., Dünkel, O., Dabral, R., Golyanik, V ., Theobalt, C., Bermano, A.H.: Attention (as discrete-time markov) chains. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2026), https://openreview.net/forum?id=wy2FwyNYro

    2026

  8. [16]

    In: International Conference on Learning Representations (2019),https://openreview.net/forum?id=rJl-b3RcF7

    Frankle, J., Carbin, M.: The lottery ticket hypothesis: Finding sparse, trainable neural networks. In: International Conference on Learning Representations (2019),https://openreview.net/forum?id=rJl-b3RcF7

    2019

  9. [17]

    In: 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

    Huang, H.W., Chai, W., Chen, K.M., Yang, C.Y ., Hwang, J.N.: Tosa: Token merging with spatial awareness. In: 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 9654–9660. IEEE (2025)

    2025

  10. [18]

    In: The Fourteenth International Conference on Learning Representations (2026), https:// openreview.net/forum?id=gH3HhnfWLC

    Jacobs, M., Fel, T., Hakim, R., Brondetta, A., Ba, D.E., Keller, T.A.: Block recurrent dynamics in vision transformers. In: The Fourteenth International Conference on Learning Representations (2026), https:// openreview.net/forum?id=gH3HhnfWLC

    2026

  11. [19]

    Krizhevsky, A.: Learning multiple layers of features from tiny images. Tech. rep., University of Toronto (2009) 11 SAPER : Soft Attention PrunER

    2009

  12. [20]

    IEEE Transactions on Neural Networks and Learning Systems37(1), 357–370 (2026)

    Lee, E., Hwang, Y .: Amap: Automatic multihead attention pruning by similarity-based prun- ing indicator. IEEE Transactions on Neural Networks and Learning Systems37(1), 357–370 (2026). https://doi.org/10.1109/TNNLS.2025.3606750

    2026

  13. [21]

    In: 2024 IEEE Inter- national Conference on Consumer Electronics-Asia (ICCE-Asia)

    Lee, S., Kim, B.s.: Entropy-guided head importance for token pruning in vision transformers. In: 2024 IEEE Inter- national Conference on Consumer Electronics-Asia (ICCE-Asia). pp. 1–3 (2024). https://doi.org/10.1109/ICCE- Asia63397.2024.10773748

    2024

  14. [22]

    Transactions of the As- sociation for Computational Linguistics9, 1442–1459 (2021)

    Li, J., Cotterell, R., Sachan, M.: Differentiable subset pruning of transformer heads. Transactions of the As- sociation for Computational Linguistics9, 1442–1459 (2021). https://doi.org/10.1162/tacl_a_00436, https: //aclanthology.org/2021.tacl-1.86/

    2021 doi

  15. [23]

    Liu, N., Stewart, W.J.: Markov Chains and Spectral Clustering, pp. 87–98. Springer Berlin Heidel- berg, Berlin, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25575-5_8, https://doi.org/10.1007/ 978-3-642-25575-5_8

    2011 doi

  16. [24]

    In: Leen, T., Dietterich, T., Tresp, V

    Meila, M., Shi, J.: Learning segmentation by random walks. In: Leen, T., Dietterich, T., Tresp, V . (eds.) Advances in Neural Information Processing Systems. vol. 13. MIT Press (2000), https://proceedings.neurips.cc/ paper_files/paper/2000/file/069654d5ce089c13f642d19f09a3d1c0...

    2000

  17. [25]

    2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp

    Meng, L., Li, H., Chen, B.C., Lan, S., Wu, Z., Jiang, Y .G., Lim, S.N.: Adavit: Adaptive vision transformers for efficient image recognition. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 12299–12308 (2021),https://api.semanticscholar.org/Corpu...

    2022

  18. [26]

    (eds.) Advances in Neural Information Processing Systems

    Michel, P., Levy, O., Neubig, G.: Are sixteen heads really better than one? In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019), https://procee...

    2019

  19. [27]

    Transactions on Machine Learning Research (2024), https://openreview.net/forum?id=a68SUt6zFt, featured Certification

    Oquab, M., Darcet, T., Moutakanni, T., V o, H.V ., Szafraniec, M., Khalidov, V ., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y ., Li, S.W., Misra, I., Rabbat, M., Sharma, V ., Synnaeve, G., Xu, H., Jegou, H., Mai...

    2024

  20. [28]

    (eds.) Advances in Neural Information Processing Systems (2021),https://openreview.net/forum?id=R-616EWWKF5

    Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., Dosovitskiy, A.: Do vision transformers see like convolutional neural networks? In: Beygelzimer, A., Dauphin, Y ., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems (2021),https://openreview....

    2021

  21. [29]

    Schwartz, R., Dodge, J., Smith, N.A., Etzioni, O.: Green ai. Commun. ACM63(12), 54–63 (2020). https://doi.org/10.1145/3381831,https://doi.org/10.1145/3381831

    2020 doi

  22. [30]

    Transactions on Machine Learning Research (2025),https://openreview.net/forum?id=91H76m9Z94

    Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J.I., Biderman, S., Garriga-Alonso, A., Conmy, A., Nanda, N., Rumbelow, J.M., Wattenberg, M., Schoots, N., Miller, J., Saunders, W., Michaud, E.J., ...

    2025

  23. [31]

    Shen, C.: Adaptive mlp pruning for large vision transformers (2026),https://arxiv.org/abs/2603.08100

    2026

  24. [32]

    IEEE Transactions on Pattern Analysis and Machine Intelligence22(8), 888–905 (2000)

    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence22(8), 888–905 (2000)

    2000

  25. [33]

    In: Korhonen, A., Traum, D., Màrquez, L

    Strubell, E., Ganesh, A., McCallum, A.: Energy and policy considerations for deep learning in NLP. In: Korhonen, A., Traum, D., Màrquez, L. (eds.) Proceedings of the 57th Annual Meeting of the Associa- tion for Computational Linguistics. pp. 3645–3650. Association for Computat...

    2019 doi

  26. [34]

    In: The International Conference on Machine Learning (ICML) 2025 (2025)

    Struski, L., Bednarczyk, M.B., Podolak, I.T., Tabor, J.: Lapsum - one method to differentiate them all: Ranking, sorting and top-k selection. In: The International Conference on Machine Learning (ICML) 2025 (2025)

    2025

  27. [35]

    Uddin, M.H., Seymour, L., Baidya, S.: Heart-vit: Hessian-guided efficient dynamic attention and token pruning in vision transformer (2025),https://arxiv.org/abs/2512.20120

    2025

  28. [36]

    In: Guyon, I., Luxburg, U.V ., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polo- sukhin, I.: Attention is all you need. In: Guyon, I., Luxburg, U.V ., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Proce...

    2017

  29. [37]

    In: Korhonen, A., Traum, D., Màrquez, L

    V oita, E., Talbot, D., Moiseev, F., Sennrich, R., Titov, I.: Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In: Korhonen, A., Traum, D., Màrquez, L. (eds.) Proceedings of the 57th Annual Meeting of the Association for Comp...

    2019 doi

  30. [38]

    In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Wang, H., Dedhia, B., Jha, N.K.: Zero-tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers. In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16070–16079 (2024)

    2024

  31. [39]

    Science China Information Sciences66(7) (2023)

    Yu, H., Wu, J.: A unified pruning framework for vision transformers. Science China Information Sciences66(7) (2023). https://doi.org/10.1007/s11432-022-3646-6

    2023 doi

  32. [40]

    In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Yu, L., Xiang, W.: X-pruner: explainable pruning for vision transformers. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 24355–24363 (2023). https://doi.org/10.1109/CVPR52729.2023.02333

    2023

  33. [41]

    Zhang, Z., Zhang, J., Zhou, B., Duan, L., Chen, H.: D 2-vpr: A parameter-efficient visual-foundation-model- based visual place recognition method via knowledge distillation and deformable aggregation (2025), https: //arxiv.org/abs/2511.12528 13 SAPER : Soft Attention PrunER A ...

    2025

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.