Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Acceleration

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that token pruning and token merging in vision transformers are both special cases of a token transformation matrix, and that a many-to-many transformation—where every original token can contribute to several retained…

desk verdict Broad experiments and a clean unifying framing, but the many-to-many claim collapses on inspection: at the optimal temperature the method is effectively token-exclusive merging. read the letter →

arxiv 2506.05709 v1 pith:WGENAK2L submitted 2025-06-06 cs.CV

Fanhu Zeng , Deli Yu , Zhenglun Kong , Hao Tang This is my paper
classification cs.CV
keywords visiontransformertokenpruningmergingtransformationmatrixmany-to-manyaggregationtraining-freeaccelerationattention-basedselectiondenseprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that every token-reduction scheme for vision transformers—pruning and merging alike—is just a choice of token transformation matrix $W$ with $Y = WX$, and that previously used matrices were unnecessarily restrictive because each original token could go to only one survivor. The proposed Token Transforming framework instead builds a many-to-many matrix: the $M$ most informative tokens, chosen by column sums of the self-attention map, act as anchors, and every original token is distributed fractionally across those anchors by thresholded cosine similarity followed by softmax normalization. Because the construction uses only attention statistics and similarities, no training is needed, and the paper reports $\times 1.5$ throughput acceleration on DeiT-S with a 0.1% accuracy drop, with extensions to segmentation, detection, depth, and multimodal generation. If true, this makes token compression a plug-in matrix multiplication rather than a learned module, and it suggests that the accuracy loss in prior methods comes from the exclusive assignment rather than from the act of reducing tokens.

What carries the argument

The load-bearing object is the transformation matrix $W \in \mathbb{R}^{M \times N}$ in $Y = WX$. The paper observes that pruning sets $W$ to a selection of rows of the identity matrix, giving a one-to-one mapping, while merging sets it to a block matrix in which each column has a single nonzero block, giving a many-to-one mapping; both forbid a token from contributing to more than one output. Token Transforming instead constructs $W$ by using column-sum attention saliency $H_j = \sum_i A_{ij}$ to name the $M$ anchors, gated cosine similarity $\mathrm{GSim}(i,j)$ from each anchor to every original token, column-wise softmax assignment normalization with temperature $\tau$, and row normalization. The attention-scaling vector $s_i = \sum_j m_{ij}$ is then added to the logits of the next self-attention to compensate for variable token mass. All of the paper's accuracy claims ride on this particular construction of $W$ retaining more information than exclusive matrices.

What would settle it

Run the same DeiT-S/0.7 pipeline with the $M$ anchors selected by the lowest column-sum attention instead of the highest, leaving the threshold, temperature, and all other settings unchanged; if accuracy stays near the reported 79.7%, the saliency criterion is not what preserves accuracy, and if it drops sharply, the criterion is confirmed as the operative mechanism.

Watch

Extended reading notes

Core claim

The central claim is that the information lost by existing token-reduction methods is a consequence of token-exclusive assignment, not of token reduction itself. The evidence offered is a generalized construction: after a transformer block's self-attention, the paper selects $M$ informative tokens from $N$ by sorting column sums $H_j = \sum_i A_{ij}$, then writes a transformation matrix $W$ whose entries are cosine similarities between informative and original tokens, gated by a threshold $\kappa$, softmax-normalized along columns so each original token can contribute to several output tokens, and row-normalized so each output token is a weighted average. The resulting transformed tokens feed the pretrained feed-forward network and later blocks, with an extra multiplicative scaling term $s$ added inside the next softmax attention to reflect how much original-token mass each transformed token carries. The paper reports that this many-to-many matrix retains more class-token fidelity than competing pruning and merging methods at matched FLOPs, in both training-free and fine-tuned settings.

Load-bearing premise

The method's accuracy depends on whether the few tokens that receive the most attention from all other tokens, when used as anchors and mixed with all original tokens by cosine similarity, retain what the unmodified network relies on; if they do not, the training-free accuracy gain collapses.

Editorial extensions

If this is right

  • Token pruning and token merging become two endpoints of a single design space, so any new form of $W$ defines a new compression method and token reduction turns into a matrix-construction problem.
  • Because the anchor selection ignores the class token, the same module can be dropped into dense-prediction backbones such as Segmenter and DPT, which do not have a class token.
  • At a 0.7 keep ratio on DeiT-S, the paper reports a 34.8% FLOP reduction with only a 0.1% drop in training-free accuracy, and 79.9% after 30 epochs of fine-tuning, showing the many-to-many matrix is also a useful starting point for further tuning.
  • The reported results on ViT-AugReg and ViT-H indicate the benefit persists as the backbone grows, with up to a 44.5% FLOP reduction while staying within about half a point of the uncompressed model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the matrix view suggests optimizing $W$ itself under a budget constraint, for example maximizing an information-retention proxy while fixing $M$, rather than treating cosine similarity and threshold as fixed choices.
  • The continuous scaling factor $s$ is a soft analogue of the integer token size used in some merging methods, hinting that the same bookkeeping could be exported to other sequence models that aggregate tokens, such as hierarchical or chunked transformers.
  • A direct testable extension would be to measure how well the reduced token set reconstructs the original tokens, for example minimal $L_2$ reconstruction error at matched budgets; the paper claims maximal information retention but does not report such a reconstruction metric.
  • Because the method is non-parametric and class-token-free, it could be applied with input-dependent compression ratios, for example choosing the keep ratio from the entropy of the attention map, which the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a unified view of token reduction in vision transformers as a linear transformation Y = W X, where token pruning and token merging are special cases of the transformation matrix W. It then introduces a specific construction of W: M informative tokens are selected by column-summing the self-attention map, a gated cosine-similarity matrix between informative and full tokens is computed, and a column softmax with temperature τ followed by a row normalization produces the transformation coefficients. A scaling adaptation term is added to the next attention block, and a nearest-neighbor token-recovery module is used for dense prediction tasks. The authors claim that this 'many-to-many' non-exclusive transformation preserves the most token information and enables training-free acceleration, reporting, among other results, ×1.5 throughput on DeiT-S with only a 0.1% accuracy drop. Experiments are carried out on image classification, segmentation, detection, depth estimation, and multimodal language model generation.

Significance. The unification of pruning and merging as diagonal and block-wise transformation matrices is a clean conceptual framing, and the experimental breadth is a genuine strength: the method is evaluated across several ViT variants and multiple dense prediction tasks, with consistent improvements over ToMe and Hourglass in most settings. If the central mechanism claim were supported, this would be a useful plug-and-play training-free compression module. However, the paper's own ablation contradicts the mechanism: the best accuracy is obtained at a temperature where the assignment matrix is effectively one-hot per column, i.e., token-exclusive merging, not many-to-many transformation. I also do not find the internal concern about Eq. (7) persuasive, since the added term, read as a per-key bias, is not a row-wise constant; nonetheless, the notation needs clarification. The empirical scheme may still be a competitive training-free merging method based on column-sum anchor selection, but the paper as written overclaims its novelty and does not demonstrate the information-retention property that is central to its title and abstract.

major comments (3)
  1. [Sec. 3.3, Eq. (5), Table 6] The optimal operating point is token-exclusive, which contradicts the paper's central claim. In Eq. (5), the softmax is computed with multiplier τ; at τ = 150, for any two anchors whose cosine similarities differ by δ = 0.05, the softmax weight ratio is exp(150 × 0.05) > 1000, so each column of m is effectively one-hot. The subsequent row normalization in Eq. (6) then makes W a hard, exclusive many-to-one assignment of each full token to exactly one transformed token, which is precisely the token-exclusive merging the paper claims to supersede. Table 6 reports 74.5% accuracy at τ = 1 (the regime with genuinely many-to-many column distributions) versus 79.7% at τ = 150. The paper provides no measurement of effective assignment entropy or of the number of columns with multiple significant entries at the configurations used in Tables 1–5. The claim that non-exclusive many-to-many transformation drives the reported gains is therefore not established and is contradicted by the chosen best configuration.
  2. [Sec. 4.7, Table 7] The ablation for assignment normalization reports an unexplained collapse to 5.7% accuracy when assignment normalization is removed while scaling adaptation is kept. According to Eqs. (5)–(6), omitting the column softmax would still leave the row normalization W_ij = m_ij / Σ_j m_ij, so each transformed token would be a convex combination of full tokens with nonnegative weights summing to one. Such an output should not degrade DeiT-S to near chance level. The authors need to specify exactly how W was computed in this variant, verify the number, and reconcile the prose (which cites '77.9% vs 79.6%') with the table (which shows 77.9% for the normalization-only variant and 79.7% for the full method). This result is load-bearing because it is used to justify the necessity of Eq. (5).
  3. [Sec. 3.2, Eq. (2), Fig. 2] The claim that column sums H_j identify tokens whose transformed mixtures 'reserve token information to the most' is not backed by an information-retention measurement. Fig. 2 plots class-token L2 error against accuracy, which is a proxy for downstream classification, not a measure of retained token information. Table 8 shows that the column-sum criterion outperforms uniform and class-attention selection empirically, but that is not a demonstration that information retention is maximized. The authors should either provide a direct measurement of retained information (e.g., reconstruction error, attention-entropy preservation, or a token-recovery accuracy measure) or soften the information-theoretic phrasing of the central claim.
minor comments (5)
  1. [Sec. 3.4, Eq. (7)] Please clarify the broadcast direction. If log(s) is added as a per-key bias (the same row vector added to every row, so log(s_j) is added to column j for all queries), the term is not a row-wise constant and does affect the softmax output. As written, 'repeating row vector along column axis' is ambiguous and invites the incorrect reading that the term is a row-wise constant and hence inert.
  2. [Table 1] The 'Training-Free' column is internally inconsistent: ToMe is marked ✗ for DeiT-S but ✓ for ViT-AugReg-S/Ti/B and ViT-H, although ToMe is a training-free method in all these rows. Please correct the markers or define exactly what the column denotes.
  3. [Abstract and Table 1] The abstract claims a 40% FLOPs reduction with a 0.1% accuracy drop, but the training-free DeiT-S row in Table 1 reduces FLOPs by 34.8% (3.0 vs 4.6 GFLOPs), while the 43.5% row (2.6 GFLOPs) is marked as not training-free. Please align the numbers and clarify which result corresponds to the abstract claim.
  4. [Sec. 4.7] The hyperparameters κ, τ, and the reserving ratio are tuned on the validation set. The paper should state for each downstream task whether the same values were used without re-tuning, since 'training-free' otherwise requires qualification as 'no weight updates but with validation-set hyperparameter selection.'
  5. [Fig. 2] The plot lacks axis labels and quantitative values for the L2-loss axis; please add them so the claimed trend between class-token error and accuracy can be verified.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the transformation matrix is computed from inputs and validated against external benchmarks; the only self-citation (M2M-TAG) is not load-bearing.

full rationale

The paper's derivation is self-contained. Equation (1) defines token reduction as Y = W X, and Eqs. (2)-(6) deterministically construct W from the attention map and token similarities; the transformed tokens are then evaluated on external benchmarks (DeiT, Segmenter, DPT, DINO, LLaVA) where accuracy is not constructed by W. The hyperparameters kappa and tau are validation-set choices (Table 6), not fitted parameters renamed as predictions, so there is no fitted-input-called-prediction issue. The central claim that many-to-many assignment 'reserves token information to the most' is a heuristic assertion backed by Fig. 2 and ablations, not a theorem, but it is not circular: the paper does not assume the target accuracy in defining W. The only self-reference is the bibliography entry [53] (M2M-TAG by two of the present authors); it is not used in the body as a load-bearing premise, and the algorithm is fully specified in Eqs. (2)-(8). The temperature behavior in Table 6 (tau=1 gives 74.5% while tau=150 gives 79.7%) raises a consistency question about whether the optimal operating point is truly many-to-many, but that is a correctness/robustness concern, not circularity. No step reduces by construction to the paper's own inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or architectural entities. Its central assumptions are the informativeness of attention column sums and the sufficiency of linear mixtures for training-free transfer. Two hyperparameters are tuned on validation data, and the token reserving ratio is a user-chosen operating point.

free parameters (3)
  • Gate similarity threshold kappa = 0.5 (best in ablation; default not stated)
    Filters out low cosine-similarity tokens in Eq. (3). Tuned on ImageNet validation in Sec. 4.7, Table 6.
  • Softmax temperature tau = 100 (best in ablation; default not stated)
    Controls sharpness of assignment normalization in Eq. (5). Tuned on ImageNet validation in Sec. 4.7, Table 6.
  • Token reserving ratio = 0.7 default, 0.6 for aggressive compression
    User-chosen compression budget in Table 1. It is an operating point rather than a fitted constant, but it is selected by the authors and determines the accuracy-FLOPs trade-off.
assumptions (4)
  • domain assumption Column sums of the self-attention map identify the most informative tokens.
    Used to define informative tokens in Sec. 3.2, Eq. (2). No analysis is given for dense prediction settings or for later layers after repeated transformation.
  • domain assumption Cosine-similarity weighting with threshold and softmax normalization preserves task-relevant information when many tokens are mixed into fewer anchors.
    This is the core premise of the transformation matrix W in Eqs. (3)-(6). The paper calls it information-preserving but provides no formal bound or measurement of information retention.
  • domain assumption Pretrained transformer layers tolerate the transformed token distribution without any fine-tuning.
    The training-free claim depends on the pretrained subsequent attention and FFN layers still working well after the token mixture. This is asserted but not justified beyond the reported experiments.
  • domain assumption Nearest-neighbor mapping from transformed tokens back to original token positions is sufficient for dense prediction tasks.
    Used in Sec. 3.5, Eqs. (9)-(10). The recovery step copies the nearest transformed token to each original position, which is a strong assumption about representational alignment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Acceleration." pith.science (2026). https://pith.science/paper/WGENAK2L

@misc{pith2026250605709,
  author       = {Pith},
  title        = {Pith review of: Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Acceleration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WGENAK2L}},
  note         = {Machine review of arXiv:2506.05709}
}
abstract

Vision transformers have been widely explored in various vision tasks. Due to heavy computational cost, much interest has aroused for compressing vision transformer dynamically in the aspect of tokens. Current methods mainly pay attention to token pruning or merging to reduce token numbers, in which tokens are compressed exclusively, causing great information loss and therefore post-training is inevitably required to recover the performance. In this paper, we rethink token reduction and unify the process as an explicit form of token matrix transformation, in which all existing methods are constructing special forms of matrices within the framework. Furthermore, we propose a many-to-many Token Transforming framework that serves as a generalization of all existing methods and reserves the most information, even enabling training-free acceleration. We conduct extensive experiments to validate our framework. Specifically, we reduce 40% FLOPs and accelerate DeiT-S by $\times$1.5 with marginal 0.1% accuracy drop. Furthermore, we extend the method to dense prediction tasks including segmentation, object detection, depth estimation, and language model generation. Results demonstrate that the proposed method consistently achieves substantial improvements, offering a better computation-performance trade-off, impressive budget reduction and inference acceleration.

Figures

Figures reproduced from arXiv: 2506.05709 by the authors.

Figure 1
Figure 1. Comparison of different token reduction between original (left column) and remaining (right column) tokens. (a) Pruning methods collect tokens from corresponding one token (one-to-one). (b) Merging meth￾ods exclusively merge multiple original tokens into one remaining to￾ken (many-to-one). (c) Token Transforming integrates original tokens into remaining ones by matrix transformation (many-to-many). kens in a dynamic… view at source ↗
Figure 2
Figure 2. Relationship between the error of class token and the [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Detail structure of our proposed Token Transforming framework. The module is inserted between the self-attention layer and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison with previous methods under different FLOPs in a training-free manner on different tasks. For fair comparison, [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Informative tokens and the heatmap of transformation [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Continual Learning for Generative AI: From LLMs to MLLMs and Beyond

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.

Reference graph

Works this paper leans on

56 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    To- 8 ken merging: Your vit but faster.arXiv preprint arXiv:2210.09461, 2022

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. To- 8 ken merging: Your vit but faster.arXiv preprint arXiv:2210.09461, 2022. 1, 2, 3, 4, 5, 7

    arXiv 2022

  2. [2]

    Crossvit: Cross-attention multi-scale vision transformer for image classification

    Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. InProceedings of the IEEE/CVF in- ternational conference on computer vision, pages 357–366,

  3. [3]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016. 5

    2016

  4. [4]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 5

    2009

  5. [5]

    An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020. 1, 2

    arXiv 2010

  6. [6]

    Adaptive token sampling for efficient vision transformers

    Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and J ¨urgen Gall. Adaptive token sampling for efficient vision transformers. InComputer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XI, pages 396–414. Spri...

    work page 2022

  7. [7]

    Dynamic channel pruning: Feature boosting and suppression.arXiv preprint arXiv:1810.05331,

    Xitong Gao, Yiren Zhao, Łukasz Dudziak, Robert Mullins, and Cheng-zhong Xu. Dynamic channel pruning: Feature boosting and suppression.arXiv preprint arXiv:1810.05331,

    arXiv

  8. [8]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017. 5

    2017

Show all 56 references
  1. [9]

    Dynamic neural networks: A sur- vey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7436–7456, 2021

    Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A sur- vey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7436–7456, 2021. 1

    2021

  2. [10]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000– 16009, 2022. 5

    2022

  3. [11]

    Distill- ing the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- ing the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015. 1

    2015 arXiv

  4. [12]

    All tokens matter: Token labeling for training better vision transform- ers.Advances in neural information processing systems, 34: 18590–18602, 2021

    Zi-Hang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transform- ers.Advances in neural information processing systems, 34: 18590–18602, 2021. 2, 6

    2021

  5. [13]

    Spvit: Enabling faster vision transformers via latency-aware soft token pruning

    Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. InEuropean conference on computer vision, pages 620–640. Springer, 2022. 1

    2022

  6. [14]

    Peeling the onion: Hierarchical reduction of data redundancy for efficient vision transformer training

    Zhenglun Kong, Haoyu Ma, Geng Yuan, Mengshu Sun, Yanyue Xie, Peiyan Dong, Xin Meng, Xuan Shen, Hao Tang, Minghai Qin, et al. Peeling the onion: Hierarchical reduction of data redundancy for efficient vision transformer training. InProceedings of the AAAI Conference on Artifici...

    2023

  7. [15]

    Token reduction should go beyond effi- ciency in generative models–from vision, language to mul- timodality.arXiv preprint arXiv:2505.18227, 2025

    Zhenglun Kong, Yize Li, Fanhu Zeng, Lei Xin, Shvat Mes- sica, Xue Lin, Pu Zhao, Manolis Kellis, Hao Tang, and Marinka Zitnik. Token reduction should go beyond effi- ciency in generative models–from vision, language to mul- timodality.arXiv preprint arXiv:2505.18227, 2025. 2

    2025

  8. [16]

    Mpvit: Multi-path vision transformer for dense prediction

    Youngwan Lee, Jonghee Kim, Jeffrey Willette, and Sung Ju Hwang. Mpvit: Multi-path vision transformer for dense prediction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7287– 7296, 2022. 1

    2022

  9. [17]

    Vidtome: Video token merging for zero-shot video editing

    Xirui Li, Chao Ma, Xiaokang Yang, and Ming-Hsuan Yang. Vidtome: Video token merging for zero-shot video editing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 1

    2024

  10. [18]

    Exploring plain vision transformer backbones for object de- tection

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object de- tection. InEuropean Conference on Computer Vision, pages 280–296. Springer, 2022. 1

    2022

  11. [19]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pages 292–305, 2023. 5

    2023

  12. [20]

    Expediting large-scale vision transformer for dense predic- tion without fine-tuning.Advances in Neural Information Processing Systems, 35:35462–35477, 2022

    Weicong Liang, Yuhui Yuan, Henghui Ding, Xiao Luo, Wei- hong Lin, Ding Jia, Zheng Zhang, Chao Zhang, and Han Hu. Expediting large-scale vision transformer for dense predic- tion without fine-tuning.Advances in Neural Information Processing Systems, 35:35462–35477, 2022. 1, 3, 6, 7, 8

    2022

  13. [21]

    Not all patches are what you need: Expediting vision transformers via token reorganiza- tions.arXiv preprint arXiv:2202.07800, 2022

    Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganiza- tions.arXiv preprint arXiv:2202.07800, 2022. 1, 2, 3, 5, 6, 8

    2022 arXiv

  14. [22]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  15. [23]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024. 7

    2024

  16. [24]

    Adaptive sparse vit: Towards learnable adaptive token pruning by fully exploiting self-attention.arXiv preprint arXiv:2209.13802,

    Xiangcheng Liu, Tianyi Wu, and Guodong Guo. Adaptive sparse vit: Towards learnable adaptive token pruning by fully exploiting self-attention.arXiv preprint arXiv:2209.13802,

    arXiv

  17. [25]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 2, 7

    2021

  18. [26]

    Beyond attentive tokens: Incorporating to- ken importance and diversity for efficient vision transform- ers

    Sifan Long, Zhen Zhao, Jimin Pi, Shengsheng Wang, and Jingdong Wang. Beyond attentive tokens: Incorporating to- ken importance and diversity for efficient vision transform- ers. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 10334–10343,

  19. [27]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521,

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521,

  20. [28]

    Importance estimation for neural net- work pruning

    Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Fro- sio, and Jan Kautz. Importance estimation for neural net- work pruning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264– 11272, 2019. 1

    2019

  21. [29]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by gen- erative pre-training. 2018. 2

    2018

  22. [30]

    Vi- sion transformers for dense prediction

    Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. InProceedings of the IEEE/CVF international conference on computer vision, pages 12179–12188, 2021. 1, 5, 7

    2021

  23. [31]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification.Advances in neural information processing systems, 34:13937–13949,

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification.Advances in neural information processing systems, 34:13937–13949,

  24. [32]

    Beyond fixa- tion: Dynamic window visual transformer

    Pengzhen Ren, Changlin Li, Guangrun Wang, Yun Xiao, Qing Du, Xiaodan Liang, and Xiaojun Chang. Beyond fixa- tion: Dynamic window visual transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11987–11997, 2022. 2

    2022

  25. [33]

    Tokenlearner: Adaptive space-time tokenization for videos.Advances in Neural In- formation Processing Systems, 34:12786–12797, 2021

    Michael Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: Adaptive space-time tokenization for videos.Advances in Neural In- formation Processing Systems, 34:12786–12797, 2021. 1

    2021

  26. [34]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models.arXiv preprint arXiv:2403.15388,

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models.arXiv preprint arXiv:2403.15388,

  27. [35]

    Indoor segmentation and support inference from rgbd images

    Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. InComputer Vision–ECCV 2012: 12th Eu- ropean Conference on Computer Vision, Florence, Italy, Oc- tober 7-13, 2012, Proceedings, Part V 12, pages 746–760. ...

    2012

  28. [36]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 5

    2019

  29. [37]

    How to train your vit? data, augmentation, and regularization in vision transformers

    Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. 2022. 5

    2022

  30. [38]

    Segmenter: Transformer for semantic segmenta- tion

    Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmenta- tion. InProceedings of the IEEE/CVF international confer- ence on computer vision, pages 7262–7272, 2021. 1

    2021

  31. [39]

    Chip: Channel independence- based pruning for compact neural networks.Advances in Neural Information Processing Systems, 34:24604–24616,

    Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman Aliari Zonouz, and Bo Yuan. Chip: Channel independence- based pruning for compact neural networks.Advances in Neural Information Processing Systems, 34:24604–24616,

  32. [40]

    Revisiting unreasonable effectiveness of data in deep learning era

    Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhi- nav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. InProceedings of the IEEE international conference on computer vision, pages 843–852, 2017. 2

    2017

  33. [41]

    Patch slimming for ef- ficient vision transformers

    Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for ef- ficient vision transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12165–12174, 2022. 2, 6

    2022

  34. [42]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. InInternational conference on machine learning, pages 10347–10357. PMLR, 2021. 1, 2, 5

    2021

  35. [43]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 2

    2017

  36. [44]

    Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition.Advances in Neural Information Processing Systems, 34:11960–11973,

    Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition.Advances in Neural Information Processing Systems, 34:11960–11973,

  37. [45]

    Qsfm: Model pruning based on quantified similarity between feature maps for ai on edge.IEEE Internet of Things Journal, 9(23):24506–24515,

    Zidu Wang, Xuexin Liu, Long Huang, Yunqing Chen, Yufei Zhang, Zhikang Lin, and Rui Wang. Qsfm: Model pruning based on quantified similarity between feature maps for ai on edge.IEEE Internet of Things Journal, 9(23):24506–24515,

  38. [46]

    Joint token pruning and squeezing towards more aggressive compression of vision transformers

    Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jia- jun Liang. Joint token pruning and squeezing towards more aggressive compression of vision transformers. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2092–2101, 2023. 1, 2, 3, 5

    2023

  39. [47]

    Ppt: Token pruning and pooling for efficient vision transformers.arXiv preprint arXiv:2310.01812, 2023

    Xinjian Wu, Fanhu Zeng, Xiudong Wang, and Xinghao Chen. Ppt: Token pruning and pooling for efficient vision transformers.arXiv preprint arXiv:2310.01812, 2023. 1

    2023 arXiv

  40. [48]

    Evo-vit: Slow-fast token evolution for dynamic vision transformer

    Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. InProceedings of the AAAI Conference on Artificial Intelligence, pages 2964–2972, 2022. 1, 5

    2022

  41. [49]

    Global vision transformer pruning with hessian-aware saliency

    Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision transformer pruning with hessian-aware saliency. InPro- 10 ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18547–18557, 2023. 1

    2023

  42. [50]

    A-vit: Adaptive tokens for efficient vision transformer

    Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10809–10818, 2022. 2

    2022

  43. [51]

    Tokens-to-token vit: Training vision transformers from scratch on imagenet

    Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. InProceedings of the IEEE/CVF in- ternational conference on computer vision, pages 558–567,

  44. [52]

    Vision trans- former with progressive sampling

    Xiaoyu Yue, Shuyang Sun, Zhanghui Kuang, Meng Wei, Philip HS Torr, Wayne Zhang, and Dahua Lin. Vision trans- former with progressive sampling. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 387–396, 2021. 6

    2021

  45. [53]

    M2m-tag: Training-free many- to-many token aggregation for vision transformer accelera- tion

    Fanhu Zeng and Deli Yu. M2m-tag: Training-free many- to-many token aggregation for vision transformer accelera- tion. InWorkshop on Machine Learning and Compression, NeurIPS 2024. 2

    2024

  46. [54]

    Parameter efficient merging for multimodal large language models with complementary parameter adaptation.arXiv preprint arXiv:2502.17159, 2025

    Fanhu Zeng, Haiyang Guo, Fei Zhu, Li Shen, and Hao Tang. Parameter efficient merging for multimodal large language models with complementary parameter adaptation.arXiv preprint arXiv:2502.17159, 2025. 7

    2025

  47. [55]

    Dino: Detr with improved denoising anchor boxes for end-to-end object detection.arXiv preprint arXiv:2203.03605, 2022

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection.arXiv preprint arXiv:2203.03605, 2022. 1, 7

    2022 arXiv

  48. [56]

    Semantic under- standing of scenes through the ade20k dataset.International Journal of Computer Vision, 127:302–321, 2019

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset.International Journal of Computer Vision, 127:302–321, 2019. 5 11

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.