Pith. sign in

REVIEW 2 major objections 5 minor 77 references

Token Cropr: Faster ViTs for Quite a Few Tasks

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Token Cropr learns task-relevant token rankings with auxiliary heads that are discarded after training, yielding 1.5-4x Vision Transformer speedups with small accuracy losses.

desk verdict Solid token pruning paper with a neat inference-time query aggregation trick; the segmentation headline lacks error bars and the abstract overstates the detection drop. read the letter →

arxiv 2412.00965 v1 pith:DV3AULZ5 submitted 2024-12-01 cs.CV cs.LG

Benjamin Bergner , Christoph Lippert , Aravindh Mahendran This is my paper · ORCID
classification cs.CVcs.LG
keywords tokenpruningvisiontransformerscross-attentionauxiliaryheadsinferencespeedupsemanticsegmentationobjectdetectionLastLayerFusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Token Cropr is a method for making Vision Transformers faster by removing image tokens that do not matter for the task at hand. Its central claim is that token importance can be learned end-to-end with small auxiliary prediction heads attached to a cross-attention router, and that the learned ranking still works after the auxiliary components are removed at inference. If that claim holds, image classification, semantic segmentation, object detection, and instance segmentation can all run 1.5-4x faster with only small accuracy penalties, and the overhead of the pruning mechanism itself drops to nearly that of a random pruner. The paper reports, as a best case, a 2x speedup on ADE20k semantic segmentation with a 0.1 median mIoU penalty, and 89.7% ImageNet top-1 accuracy at 2.1x speedup on a large ViT.

What carries the argument

The central object is the Cropr module, inserted after ViT blocks, with four parts: a scorer that computes cross-attention $A = QK(X)^\top$ between learnable queries and input token keys; a selector that keeps the Top-K tokens by summed attention scores; an aggregator that produces a weighted token for an auxiliary task head; and the head itself. A stop-gradient isolates the auxiliary branch from the backbone. The load-bearing identity for efficiency is $\sum_{n=1}^N (QK^\top)_n = (\sum_{n=1}^N Q_n)K^\top = qK^\top$: at inference the per-query attention matrix need not be materialized, so scoring collapses to one vector-matrix product. Last Layer Fusion then reinserts pruned tokens before the final block so dense tasks can still see all patches.

What would settle it

Fine-tune a ViT with Cropr, then at inference replace only the learned scorer with a random scorer at the same pruning rates on ImageNet-1k and ADE20k; if random selection matches or beats Cropr's accuracy at equal throughput, the learned ranking is not carrying the result.

Watch

Extended reading notes

Core claim

The paper's central claim is that token pruning for ViTs should be driven by a learned, task-specific notion of relevance rather than by heuristics such as self-attention scores or input variance. A Cropr module inserted after transformer blocks contains a cross-attention scorer that ranks tokens, a Top-K selector, an aggregator that forms a weighted token, and an auxiliary head that supplies task gradients; the aggregator and head are discarded after training. The paper establishes that the learned ranking transfers to inference, where the router collapses to a single precomputed query vector and a dot-product scoring step, giving throughput close to random pruning. Across classification, segmentation, detection, and instance segmentation, Cropr is claimed to beat random, variance, and attention-score baselines at comparable throughput and to close most of the accuracy gap to the unpruned model, especially as model size and image resolution grow.

Load-bearing premise

The learned token ranking stays valid after the auxiliary heads and aggregator are removed, so the Top-K selector at inference keeps choosing the tokens that matter.

Editorial extensions

If this is right

  • Image classification can run 1.6-1.9x faster with an accuracy drop of only 0.3-0.7 points on an MAE-pretrained ViT-L on ImageNet-1k.
  • Semantic segmentation on ADE20k with an EV A-02 backbone reaches 56.6 median mIoU versus 56.7 for the unpruned model while being 2.0x faster.
  • Object detection and instance segmentation on COCO with Cascade Mask R-CNN maintain 63.0 AP box versus 64.2 unpruned while achieving a 1.9x overall speedup and a 2.4x encoder speedup.
  • Larger models and higher-resolution inputs get larger speedups and smaller performance penalties, with the accuracy gap shrinking from -0.9 on ViT-B to -0.4 on ViT-H despite higher total pruning ratios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Pith inference: Because the learned scores are produced by a single precomputed query vector, the same training-time setup could attach to any transformer with a differentiable task head, including non-image sequence models; the paper itself notes language and audio as future work.
  • Pith inference: The scores the router produces at inference are effectively free task-relevance maps, so Cropr could double as a lightweight saliency or explainability signal without a separate attention-rollout or attribution pass.
  • Pith inference: The stop-gradient isolation of the auxiliary heads suggests a testable recipe for other efficiency tricks, such as learning per-image adaptive keep rates or merging decisions, without backprop interference with the backbone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents Cropr, a token-pruning module for Vision Transformers that scores tokens with a cross-attention router, keeps the top-K tokens, and uses an aggregator plus auxiliary task head to train the router. At inference the aggregator and auxiliary heads are removed, and the learned queries collapse into a single vector, leaving a linear scorer plus a Top-K selector. Last Layer Fusion (LLF) reinserts pruned tokens before the final transformer block to support dense tasks. The method is evaluated on ImageNet classification with MAE and EVA-02 backbones, ADE20k semantic segmentation, and COCO object detection and instance segmentation, with comparisons to random pruning, variance pruning, attention Top-K, and several prior token-pruning and token-merging methods. Reported speedups range from 1.5x to 4.1x with moderate accuracy drops, including a headline 0.1 median mIoU drop at 2.0x speedup on ADE20k.

Significance. If the results are reliable, Cropr is a useful practical contribution: it is among the few token-pruning methods demonstrated across classification, segmentation, and detection, and its inference-time overhead is very small. The paper benefits from a broad set of baselines, a clean ablation study of module design, fusion strategies and pruning schedules, and an honest discussion of limitations in App. B. The algebraic reduction in Eqs. (6)-(7) is correct and makes the inference-time collapse transparent. The main unresolved risks are the statistical support for the headline segmentation result and the alignment between the training-time attention-based objective and the inference-time collapsed score.

major comments (2)
  1. [Sec. 4.2 / Fig. 5] The central segmentation claim of a 0.1 median mIoU drop at 2.0x speedup is reported only as a median over 5 seeds, with no error bars, per-seed points, or significance test. Since all pruning baselines also benefit from LLF and the differences among methods are at the level of one mIoU point, the 'negligible performance penalty' in the abstract is not yet statistically supported. Please report the full distribution (e.g., box plot or mean plus standard deviation) and, if feasible, a paired test against the no-pruning and random-pruning baselines.
  2. [Secs. 3.1, 3.2 and 3.4] For semantic segmentation the inference-time scorer is the sum of raw attention logits over all queries, a = sum_n Q_n K^T, while the training signal flows through row-wise softmax of the full attention matrix inside the aggregator (Eq. 5). No loss directly optimizes the summed score for Top-K selection, and Top-K itself is non-differentiable. The stop-gradient does not remove this issue: it only prevents gradients from entering the encoder, while the auxiliary loss still shapes the router. For classification and detection N=1, so the collapse is trivial; the concern is specifically the N=h*w segmentation case. The current evidence for alignment is indirect. I request a direct diagnostic, e.g., train a segmentation variant with the collapsed score as the only scoring mechanism, compare Cropr against a randomly initialized but fixed linear scorer qK^T with the same Top-K and LLF, or report the correlation between the inference score a and the aggregator's learned softmax attention weights. Without such evidence, the dense-task results could be mostly attributable to LLF plus a heuristic score.
minor comments (5)
  1. [Abstract / Table 3] The abstract's 'small drops in performance' should be quantified: Table 3 shows APbox dropping from 64.2 to 63.0 and APmask from 55.4 to 54.0, which is larger than the phrase suggests; please state the exact drops in the abstract or conclusion.
  2. [Table 1] The column header 'Acc. 1000 im/s' appears to be a rendering artifact; please separate the accuracy and throughput columns clearly.
  3. [Sec. 3.4 / Fig. 3b] The claim that inference-time throughput is 'close to that of a random pruner' should be supported by exact throughput numbers in the figure or caption; the current bar chart is hard to read because the values are not labeled consistently.
  4. [Sec. 3.4] State explicitly that the aggregated query q is a fixed vector after training, so the entire inference-time scorer reduces to a fixed linear readout of the token embeddings; this makes the method's capacity and limitations clearer.
  5. [App. F] The t-SNE visualizations plot only the top-1 scoring pruned token per block; the interpretation that LLF 'synchronizes' tokens is suggestive rather than conclusive, and the text should note this limitation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Cropr's claims are empirical, benchmarked externally, and the only self-citations are contextual rather than load-bearing.

full rationale

The paper's derivation chain is not circular. Cropr's central assertion—that auxiliary prediction heads trained with a stop-gradient and cross-attention aggregation can be discarded at inference while the remaining router still selects useful tokens—is an empirical claim supported by held-out evaluations on ImageNet-1k, ADE20k, and COCO (Tables 1-3, Fig. 5). The inference scoring function a = qK^T of Eqs. (6)-(7) is not defined in terms of the reported accuracy or speedup; it is trained to support auxiliary losses, and the reported numbers are measured after training rather than derived from the training objective. No fitted parameter is renamed as a prediction, and no uniqueness theorem or load-bearing result is imported from prior work by the same authors. The only self-citations are contextual references to trends in model scale and high-resolution recognition ([13] and [3]), and neither supports a central claim. The train-inference mismatch in segmentation (sum-of-queries collapse, stop-gradient, non-differentiable Top-K) is a legitimate correctness risk about whether the learned ranking transfers, but it is not circularity: the inference score is not equivalent by construction to the training loss or to the reported metric. The paper states limitations in App. B and compares against random, variance, and attention baselines, which makes the benchmark comparisons externally falsifiable. Accordingly, no circular step is present and the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a set of empirically motivated design choices (pruning rates, schedules, LLF, stop-gradient) and on the empirical validity of the cross-attention scoring as a task-relevance signal. No theoretical guarantees are provided.

free parameters (5)
  • Pruning rate R per module = e.g., 8 (ImageNet ViT-L), 40 (ADE20k 512px), 40 (EV A-02-L 448px), 16 (ViT-B)
    R controls the accuracy-throughput trade-off and is manually set per task/model/resolution. The paper explores R in App. E (Tab. 10) and treats it as a hyperparameter.
  • Pruning schedule (module insertion points) = after every block for classification; after blocks 5,8,11,14,20 for COCO; etc.
    The schedule determines total pruning ratio (TPR) and is manually designed per task (Sec. 3.5, Sec. 4.3). The paper acknowledges schedule design as a limitation in App. B.
  • Use of Last Layer Fusion (LLF) = enabled for dense tasks, disabled for classification sometimes
    LLF is a design choice that is critical for dense tasks; ablation in Tab. 5 shows it is needed. It is a component chosen by the authors.
  • Number of queries N in scorer = 1 for classification/detection, h*w for segmentation
    N is a design choice affecting the capacity of the score signal and the training cost. The paper specifies it per task in Sec. 3.2.
  • Curriculum over R (segmentation) = linear from 1 to 40 over first 32 epochs
    A training heuristic introduced to facilitate learning; not a derived quantity.
assumptions (4)
  • domain assumption The score a = sum_n A_n computed from cross-attention with learned queries reflects task-relevant token importance.
    Sec. 3.1, Eq. (3)-(4). The whole method relies on this; validated empirically, not theoretically.
  • ad hoc to paper Applying stop-gradient before scoring/aggregation isolates auxiliary heads from backbone and improves learning.
    Sec. 3.1, Tab. 4d shows it helps by 0.3 accuracy; it is an empirical design choice specific to this paper.
  • domain assumption Last Layer Fusion enables the final ViT block to integrate pruned tokens without a performance penalty, even for dense tasks.
    Sec. 3.3, Tab. 5; empirically validated but not theoretically guaranteed.
  • domain assumption The throughput is measured with AMP on A100 and optimized batch size; results generalize to other settings.
    Sec. 4, App. B acknowledges hardware dependence; the speedup numbers depend on this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Token Cropr: Faster ViTs for Quite a Few Tasks." pith.science (2026). https://pith.science/paper/DV3AULZ5

@misc{pith2026241200965,
  author       = {Pith},
  title        = {Pith review of: Token Cropr: Faster ViTs for Quite a Few Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DV3AULZ5}},
  note         = {Machine review of arXiv:2412.00965}
}
read the original abstract

The adoption of Vision Transformers (ViTs) in resource-constrained applications necessitates improvements in inference throughput. To this end several token pruning and merging approaches have been proposed that improve efficiency by successively reducing the number of tokens. However, it remains an open problem to design a token reduction method that is fast, maintains high performance, and is applicable to various vision tasks. In this work, we present a token pruner that uses auxiliary prediction heads that learn to select tokens end-to-end based on task relevance. These auxiliary heads can be removed after training, leading to throughput close to that of a random pruner. We evaluate our method on image classification, semantic segmentation, object detection, and instance segmentation, and show speedups of 1.5 to 4x with small drops in performance. As a best case, on the ADE20k semantic segmentation benchmark, we observe a 2x speedup relative to the no-pruning baseline, with a negligible performance penalty of 0.1 median mIoU across 5 seeds.

Figures

Figures reproduced from arXiv: 2412.00965 by the authors.

Figure 1
Figure 1. Cross-attention pruning (Cropr) modules successively prune less relevant tokens, retaining only the most discriminative ones for deeper layers. Our method accelerates ViTs while main￾taining high performance and is applicable to many vision tasks, from classification to segmentation and detection. The example castle images illustrate the pruning process. The heatmap visual￾izes which tokens were pruned at each block… view at source ↗
Figure 2
Figure 2. Cropr module during training. The router scores and sep [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Cropr module during inference. (a) The aggregation [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Performance-throughput tradeoff plot for different [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Semantic segmentation results on ADE20k. Cropr per [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Bounding box and instance segmentation predictions [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Performance-throughput trade-off plot for different im [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Effect of sequence length M on throughput for different image sizes. Annotations denote speedups. A mere reduction of 1 token, instead of giving a negligible speedup, results in significant throughput drops. Both the x and y-axis are log scaled. Numerical precision and…
Figure 10
Figure 10. Figure 10: Throughput ablations for FP32, AMP, and AMP with [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: t-SNE projections of tokens extracted right before the [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 69 canonical work pages

  1. [1]

    Quantifying attention flow in transformers

    Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4190–4197, Online, 2020. Association for Computa- tional Linguistics. 2

    work page 2020

  2. [2]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2021. 13

    work page 2021

  3. [3]

    Iterative patch selection for high-resolution image recognition

    Benjamin Bergner, Christoph Lippert, and Aravindh Ma- hendran. Iterative patch selection for high-resolution image recognition. In ICLR, 2022. 1

    work page 2022

  4. [4]

    Flexivit: One model for all patch sizes

    Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. In CVPR, pages 14496–14506, 2023. 1

    work page 2023

  5. [5]

    Soft-nms–improving object detection with one line of code

    Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. Soft-nms–improving object detection with one line of code. In ICCV, pages 5561–5569, 2017. 13

    work page 2017

  6. [6]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In ICLR, 2023. 1, 2, 5

    work page 2023

  7. [7]

    Learned thresholds token merging and pruning for vision transformers

    Maxim Bonnaerens and Joni Dambre. Learned thresholds token merging and pruning for vision transformers. In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023. 2

    work page 2023

  8. [8]

    Cascade r-cnn: High quality object detection and instance segmentation

    Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High quality object detection and instance segmentation. IEEE TPAMI, 43(5):1483–1498, 2019. 4

    work page 2019

Show all 77 references
  1. [9]

    Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs. IEEE TPAMI, 40(4):834–848,

  2. [10]

    Electra: Pre-training text encoders as discrimina- tors rather than generators

    K Clark. Electra: Pre-training text encoders as discrimina- tors rather than generators. In ICLR, 2020. 13

    2020

  3. [11]

    Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V

    Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V . Le. Randaugment: Practical automated data augmentation with a reduced search space. In CVPR Workshops, pages 3008–3017, 2020. 13

    2020

  4. [12]

    Flashattention: Fast and memory-efficient exact at- tention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christo- pher R´e. Flashattention: Fast and memory-efficient exact at- tention with io-awareness. NeurIPS, 35:16344–16359, 2022. 2, 14

    2022

  5. [13]

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdul- mohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschan- nen, Anurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, M...

  6. [14]

    Scaling vision transformers to 22 billion pa- rameters

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdul- mohsin, et al. Scaling vision transformers to 22 billion pa- rameters. In ICML, pages 7480–7512. PMLR, 2023. 6

    2023

  7. [15]

    Davit: Dual attention vision transform- ers

    Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transform- ers. In ECCV, pages 74–92. Springer, 2022. 6

    2022

  8. [16]

    Heatvit: Hardware-efficient adaptive token pruning for vision transformers

    Peiyan Dong, Mengshu Sun, Alec Lu, Yanyue Xie, Ken- neth Liu, Zhenglun Kong, Xin Meng, Zhengang Li, Xue Lin, Zhenman Fang, et al. Heatvit: Hardware-efficient adaptive token pruning for vision transformers. In 2023 IEEE Inter- national Symposium on High-Performance Computer Arc...

    2023

  9. [17]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  10. [18]

    Study on den- sity peaks clustering based on k-nearest neighbors and prin- cipal component analysis

    Mingjing Du, Shifei Ding, and Hongjie Jia. Study on den- sity peaks clustering based on k-nearest neighbors and prin- cipal component analysis. Knowledge-Based Systems, 99: 135–145, 2016. 1, 2, 5

    2016

  11. [19]

    Eva-02: A visual representation for neon genesis

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xin- long Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 149:105171,

  12. [20]

    Adaptive token sampling for efficient vision transformers

    Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and J¨urgen Gall. Adaptive token sampling for efficient vision transformers. In ECCV, pages 396–414. Springer, 2022. 1, 2, 5

    2022

  13. [21]

    Simple copy-paste is a strong data augmentation method for instance segmentation

    Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In CVPR, pages 2918–2928, 2021. 13

    2021

  14. [22]

    Multi-scale hybrid vision transformer and sinkhorn tokenizer for sewer defect classifi- cation

    Joakim Bruslund Haurum, Meysam Madadi, Sergio Es- calera, and Thomas B Moeslund. Multi-scale hybrid vision transformer and sinkhorn tokenizer for sewer defect classifi- cation. Automation in Construction, 144:104614, 2022. 2, 5

    2022

  15. [23]

    Tay- lor, and Thomas B

    Joakim Bruslund Haurum, Sergio Escalera, Graham W. Tay- lor, and Thomas B. Moeslund. Which tokens to use? in- vestigating token reduction in vision transformers. In ICCV, pages 773–783, 2023. 2, 5

    2023

  16. [24]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In ICCV, pages 2961–2969, 2017. 4

    2017

  17. [25]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000–16009, 2022. 5, 12, 13

    2022

  18. [26]

    Deep networks with stochastic depth

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. InECCV, pages 646–661. Springer, 2016. 13 9

    2016

  19. [27]

    Mask scoring r-cnn

    Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang. Mask scoring r-cnn. In CVPR, pages 6409–6418, 2019. 13

    2019

  20. [28]

    Perceiver IO: A general architecture for structured inputs & outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Kop- pula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J Henaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver IO: A general a...

    2022

  21. [29]

    Transformers in vision: A survey.ACM Comput

    Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey.ACM Comput. Surv., 54(10s), 2022. 1

    2022

  22. [30]

    Token fusion: Bridging the gap between token pruning and token merging

    Minchul Kim, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. Token fusion: Bridging the gap between token pruning and token merging. In WACV, pages 1383– 1392, 2024. 2

    2024

  23. [31]

    Learned token pruning for transformers

    Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 784–794, 2022. 2

    2022

  24. [32]

    Spvit: Enabling faster vision transformers via latency-aware soft token pruning

    Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In ECCV, pages 620–640. Springer, 2022. 2

    2022

  25. [33]

    Fractalnet: Ultra-deep neural networks without residuals

    Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In ICLR, 2017. 4

    2017

  26. [34]

    Sait: Sparse vision transformers through adaptive token pruning

    Ling Li, David Thorsley, and Joseph Hassoun. Sait: Sparse vision transformers through adaptive token pruning. arXiv preprint arXiv:2210.05832, 2022. 2

    2022 arXiv

  27. [35]

    Not all patches are what you need: Expediting vision transformers via token reorganizations

    Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. In ICLR, 2022. 1, 2, 5

    2022

  28. [36]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. Springer, 2014. 7

    2014

  29. [37]

    Revisiting token pruning for object detection and instance segmentation

    Yifei Liu, Mathias Gehrig, Nico Messikommer, Marco Can- nici, and Davide Scaramuzza. Revisiting token pruning for object detection and instance segmentation. In WACV, pages 2658–2668, 2024. 2

    2024

  30. [38]

    Dynamic token-pass transformers for semantic segmentation

    Yuang Liu, Qiang Zhou, Jing Wang, Zhibin Wang, Fan Wang, Jun Wang, and Wei Zhang. Dynamic token-pass transformers for semantic segmentation. In WACV, pages 1827–1836, 2024. 2, 7

    2024

  31. [39]

    Beyond attentive tokens: Incorporating to- ken importance and diversity for efficient vision transform- ers

    Sifan Long, Zhen Zhao, Jimin Pi, Shengsheng Wang, and Jingdong Wang. Beyond attentive tokens: Incorporating to- ken importance and diversity for efficient vision transform- ers. In CVPR, pages 10334–10343, 2023. 2

    2023

  32. [40]

    SGDR: Stochastic gradi- ent descent with warm restarts

    Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradi- ent descent with warm restarts. In ICLR, 2017. 13

    2017

  33. [41]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 13

    2019

  34. [42]

    Token pooling in vision transformers for image classification

    Dmitrii Marin, Jen-Hao Rick Chang, Anurag Ranjan, An- ish Prabhu, Mohammad Rastegari, and Oncel Tuzel. Token pooling in vision transformers for image classification. In WACV, pages 12–21, 2023. 1, 2, 5

    2023

  35. [43]

    Scaling open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. NeurIPS, 36,

  36. [44]

    Ia-redˆ2: Interpretability-aware redundancy reduction for vision trans- formers

    Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva. Ia-redˆ2: Interpretability-aware redundancy reduction for vision trans- formers. In NeurIPS, pages 24898–24911. Curran Asso- ciates, Inc., 2021. 2

    2021

  37. [45]

    Beit v2: Masked image modeling with vector-quantized visual tokenizers

    Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208.06366, 2022. 6

    2022 arXiv

  38. [46]

    Acceleration of stochastic approximation by averaging

    Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838–855, 1992. 13

    1992

  39. [47]

    Rabe and Charles Staats

    Markus N. Rabe and Charles Staats. Self-attention does not need o(n2) memory. arXiv preprint arXiv:2112.05682,

    arXiv

  40. [48]

    Designing network design spaces

    Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollar. Designing network design spaces. In CVPR, 2020. 6

    2020

  41. [49]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. NeurIPS, 34:13937–13949, 2021. 2, 5

    2021

  42. [50]

    Learn- ing to merge tokens in vision transformers

    Cedric Renggli, Andr ´e Susano Pinto, Neil Houlsby, Basil Mustafa, Joan Puigcerver, and Carlos Riquelme. Learn- ing to merge tokens in vision transformers. arXiv preprint arXiv:2202.12015, 2022. 2, 5

    2022 arXiv

  43. [51]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Chal- lenge. IJCV, 115(3):211–252, 2015. 5

    2015

  44. [52]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 8430–8439, 2019. 7

    2019

  45. [53]

    Deep inside convolutional networks: Visu- alising image classification models and saliency maps.arXiv preprint arXiv:1312.6034, 2013

    Karen Simonyan. Deep inside convolutional networks: Visu- alising image classification models and saliency maps.arXiv preprint arXiv:1312.6034, 2013. 2

    2013 arXiv

  46. [54]

    Segmenter: Transformer for semantic segmenta- tion

    Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmenta- tion. In ICCV, pages 7262–7272, 2021. 4, 6, 12

    2021

  47. [55]

    Rethinking the inception ar- chitecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception ar- chitecture for computer vision. In CVPR, pages 2818–2826,

  48. [56]

    Dynamic token pruning in plain vision transformers for semantic segmentation

    Quan Tang, Bowen Zhang, Jiajun Liu, Fagui Liu, and Yifan Liu. Dynamic token pruning in plain vision transformers for semantic segmentation. In ICCV, pages 777–786, 2023. 2, 7

    2023

  49. [57]

    Patch slimming for ef- 10 ficient vision transformers

    Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for ef- 10 ficient vision transformers. In CVPR, pages 12165–12174,

  50. [58]

    Maxvit: Multi-axis vision transformer

    Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In ECCV, pages 459–479. Springer, 2022. 6

    2022

  51. [59]

    Learning a parametric embedding by preserving local structure

    Laurens Van Der Maaten. Learning a parametric embedding by preserving local structure. In Artificial intelligence and statistics, pages 384–391. PMLR, 2009. 15

    2009

  52. [60]

    Visualizing data using t-SNE

    Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9 (86):2579–2605, 2008. 4

    2008

  53. [61]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, et al. Attention is all you need. NeurIPS, 30(1):261–272, 2017. 1, 3

    2017

  54. [62]

    Image as a foreign language: Beit pretraining for vision and vision- language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhil- iang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for vision and vision- language tasks. In CVPR, pages 19175–19186, 2023. 6

    2023

  55. [63]

    Solo: A simple framework for instance segmen- tation

    Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Solo: A simple framework for instance segmen- tation. IEEE TPAMI, 44(11):8587–8601, 2021. 13

    2021

  56. [64]

    Joint token pruning and squeezing towards more ag- gressive compression of vision transformers

    Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jiajun Liang. Joint token pruning and squeezing towards more ag- gressive compression of vision transformers. InCVPR, pages 2092–2101, 2023. 2

    2023

  57. [65]

    Con- vnext v2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Con- vnext v2: Co-designing and scaling convnets with masked autoencoders. In CVPR, pages 16133–16142, 2023. 6

    2023

  58. [66]

    Ppt: Token pruning and pooling for efficient vision transformers

    Xinjian Wu, Fanhu Zeng, Xiudong Wang, and Xinghao Chen. Ppt: Token pruning and pooling for efficient vision transformers. arXiv preprint arXiv:2310.01812, 2023. 2

    2023 arXiv

  59. [67]

    Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V . Le. Self-training with noisy student improves imagenet clas- sification. In CVPR, 2020. 6

    2020

  60. [68]

    Evo-vit: Slow-fast token evolution for dynamic vision transformer

    Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In AAAI, pages 2964–2972, 2022. 2

    2022

  61. [69]

    A-vit: Adaptive tokens for efficient vision transformer

    Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In CVPR, pages 10809–10818,

  62. [70]

    Metaformer baselines for vision

    Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. Metaformer baselines for vision. IEEE TPAMI, 2023. 6

    2023

  63. [71]

    Cutmix: Regu- larization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu- larization strategy to train strong classifiers with localizable features. In ICCV, pages 6023–6032, 2019. 13

    2019

  64. [72]

    Visualizing and understanding convolutional net- works

    MD Zeiler. Visualizing and understanding convolutional net- works. In ECCV, 2014. 2

    2014

  65. [73]

    Not all tokens are equal: Human-centric visual analysis via token clustering transformer

    Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang, and Xiaogang Wang. Not all tokens are equal: Human-centric visual analysis via token clustering transformer. In CVPR, pages 11101–11111, 2022. 2

    2022

  66. [74]

    Scaling vision transformers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lu- cas Beyer. Scaling vision transformers. In CVPR, pages 12104–12113, 2022. 6

    2022

  67. [75]

    Dauphin, and David Lopez-Paz

    Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion. In ICLR, 2018. 13

    2018

  68. [76]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pages 633–641, 2017. 6

    2017

  69. [77]

    Self-slimmed vision trans- former

    Zhuofan Zong, Kunchang Li, Guanglu Song, Yali Wang, Yu Qiao, Biao Leng, and Yu Liu. Self-slimmed vision trans- former. In ECCV, pages 432–448. Springer, 2022. 2, 5 11 Appendix A. Broader Impact Our method significantly increases the throughput of ViTs, making it well suited fo...

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.