Pith. sign in

REVIEW 4 major objections 6 minor 46 references

ToFe: Lagged Token Freezing and Reusing for Efficient Vision Transformer Inference

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ToFe halves the computational cost of LV-ViT while keeping Top-1 accuracy within two points of the full model by freezing unimportant tokens and reusing them later.

desk verdict A genuinely new freeze-and-reuse mechanism for ViT token reduction, with solid ablations and a real accuracy-per-FLOP win—but the headline 'better trade-off' overreaches once throughput is in the picture. read the letter →

arxiv 2507.16260 v1 pith:HTKLSDZY submitted 2025-07-22 cs.CV cs.LG

Haoyue Zhang , Jie Zhang , Song Guo This is my paper
classification cs.CVcs.LG
keywords VisionTransformerInferenceAccelerationComputationalCostTokenReductionFreezingandReusingSelectionGumbel-Softmax
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing token-reduction methods for vision transformers discard or merge low-attention tokens permanently, so tokens that look unimportant in early blocks can never contribute later even when deeper blocks would find them useful. ToFe instead freezes those tokens temporarily—they skip several transformer blocks—and revives them at a later stage if they become important, while a lightweight 2-layer MLP approximator estimates the residual update the skipped blocks would have produced. A computation-budget-aware loss is added to the training objective so the freezing and reuse decisions are optimized jointly with the backbone under a specified FLOPs target. On ImageNet, ToFe cuts LV-ViT-S and LV-ViT-M computational cost by about 50% with less than a 2% Top-1 accuracy drop, beating irreversible token-reduction baselines at the same budget. The paper thereby argues that reversible token reduction—lagged freezing and reusing—yields a better accuracy-efficiency trade-off than pruning or merging.

What carries the argument

The load-bearing mechanism is the freeze-then-reuse cycle governed by a binary decision mask. Before block $l_s$ the selector computes $z_s = \text{Softmax}(\text{MLP}(X))$ and applies Gumbel-Softmax to produce a binary mask $M_s$; only tokens with $M_s = 1$ pass through the transformer blocks, while the rest are frozen. The frozen tokens are later recovered by the token approximator, a bottleneck 2-layer MLP that learns the residual $\Delta X = X_{l+1} - X_l$ as a function of the frozen token's current features, relying on the empirically observed high cosine similarity of token features across consecutive blocks. To make the discrete selection trainable, attention is masked with $G_{i,j} = 1$ if $i=j$ or $M^s_j = 1$, so frozen tokens neither attend nor are attended to, while the forward pass can be written as $X \leftarrow M_s \odot \text{ViTBlocks}(X, M_s) + (1-M_s) \odot (X + \Delta X)$. A computation-budget-aware loss $L_{\text{FLOPs}}$ measures the squared gap between the batch-average actual FLOPs and a target budget, and is combined with classification and approximation losses; the selector and approximator are trained jointly with the backbone.

What would settle it

On ImageNet with LV-ViT-S, record the relative feature change $\varepsilon = \|X_{l+1} - X_l\| / \|X_l\|$ for the tokens ToFe freezes at the first reduction stage. The central claim predicts $\varepsilon$ is small for most frozen tokens; if a sizable fraction (e.g., more than 20%) of frozen tokens have $\varepsilon > 0.5$, the MLP cannot recover their skipped updates and ToFe's under-2% accuracy drop should fail to reproduce.

Watch

Extended reading notes

Core claim

The paper's central discovery claim is that the information loss caused by token reduction in vision transformers is largely reversible if reduction is recast as a freeze-and-reuse operation. At each reduction stage, a lightweight token selector (an MLP followed by Gumbel-Softmax) outputs a binary mask that splits patch tokens into kept and frozen sets; the kept tokens are forwarded through the transformer blocks, while the frozen tokens skip those blocks and are updated by a bottleneck 2-layer MLP approximator that predicts the skipped residual ($X_{\text{approx}} = X_{\text{freeze}} + \text{Approx}_s(X_{\text{freeze}})$). Later, frozen tokens can be reinserted into the sequence, so a token that was inattentive to the [CLS] token in a shallow block is not permanently lost. The whole system is trained end-to-end with a FLOPs budget loss $L_{\text{FLOPs}} = (\text{average actual FLOPs} - \text{target FLOPs})^2$ alongside classification and approximation losses, making the per-stage keep ratios adaptive to the input and to the device budget. The achieved result—roughly 50% FLOPs reduction on LV-ViT with under 2% accuracy drop, and consistent gains over EViT, DynamicViT, AS-ViT, ToMe, PPT, LTMP and IdleViT—is presented as evidence that lagged reuse captures information that irreversible reduction throws away.

Load-bearing premise

The load-bearing premise is that tokens which are frozen for several blocks change only slightly during those blocks, so a lightweight 2-layer MLP can approximate the updates they miss; if that premise fails for a class of images, the reused tokens carry stale features and accuracy drops.

Editorial extensions

If this is right

  • At a fixed 50% FLOPs budget on LV-ViT-S and LV-ViT-M, ToFe stays within 2% of full-model Top-1 accuracy, while baselines EViT, DynamicViT, AS-ViT and IdleViT lose between 2.2 and 19.8 accuracy points.
  • Token reusing is the source of the gain: ablating it (using $M_s \leftarrow M_{s-1} \odot M_s$) costs 2.7% Top-1 accuracy on LV-ViT-S at identical FLOPs.
  • A 3-layer MLP token selector outperforms a more complex DynamicViT-style selector by over 4 points under the same computation budget, and a bottleneck 2-layer MLP approximator beats identity, depth-wise convolution, and full transformer-block alternatives.
  • Computation-budget-aware training automatically determines per-stage keep ratios that beat hand-set ratios at equal FLOPs.
  • Instance-adaptive and batch-adaptive inference variants let the same trained model serve both single-image and batched throughput settings with negligible accuracy loss across batch sizes 1 to 128.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because ToFe stores frozen tokens in memory until their reuse stage, practical deployments would likely trade some activation memory for the compute savings; measuring peak memory alongside FLOPs would sharpen the cost picture.
  • The freeze-reuse pattern is a token-level analogue of checkpointing or recomputation, suggesting a natural extension where the number of blocks a token skips is itself learned rather than fixed at stage boundaries.
  • The approximator's assumption that frozen token features change little across consecutive blocks implies ToFe's advantage would shrink on tasks where the visual content at a fixed spatial location changes rapidly with depth, such as video or fine-grained object-part recognition; applying ToFe there would stress-test the assumption.
  • The FLOPs budget is enforced as a soft squared loss, so a harder per-sample budget controller might preserve accuracy on easy images while guaranteeing worst-case latency on edge devices.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ToFe, a token-reduction framework for efficient Vision Transformer inference. Rather than irreversibly pruning or merging low-importance tokens, ToFe temporarily freezes them at each stage and reuses a subset in later stages. A lightweight MLP token selector with Gumbel-Softmax produces binary keep/freeze decisions, and a 2-layer bottleneck MLP approximates the missing updates of frozen tokens. The framework is trained end-to-end with a self-distillation approximation loss and a computation-budget-aware FLOPs loss, and it supports both instance-adaptive and batch-adaptive inference. Experiments on DeiT-S/B and LV-ViT-S/M at roughly half the backbone FLOPs report Top-1 accuracy drops of about 1.3--2.3 points, claiming a better accuracy-complexity trade-off than prior token-reduction methods.

Significance. The central idea is interesting and timely: treating token reduction as reversible freezing rather than irreversible pruning is a genuine departure from most prior work, and the ablation in Table II (ToFe without reusing 79.2 vs. with reusing 81.9 at the same FLOPs) provides clean evidence that the reuse mechanism itself contributes substantially to the accuracy gain. The computation-budget-aware formulation and the batch-adaptive inference mode are also practical contributions, and the comparison set is broad. If the results hold, the method would give a useful accuracy-FLOPs operating point for edge deployment. However, the paper's headline claim of a better performance-complexity trade-off is only partially supported by the evidence, because throughput--which the paper itself lists as a primary complexity metric--is consistently lower than IdleViT at matched FLOPs. The absence of error bars is also a concern for the smallest reported accuracy gaps.

major comments (4)
  1. [§VI-B, Table I; §VII] The claim of a better performance-complexity trade-off is only supported on the FLOPs axis. In Table I, at matched GFLOPs, ToFe has lower throughput than IdleViT in every configuration (LV-ViT-S: 3265.1 vs. 3365.7 imgs/s; LV-ViT-M: 1964.2 vs. 2013.8; DeiT-S: 5477.3 vs. 5603.7; DeiT-B: 1689.1 vs. 1895.2), while the accuracy advantage is 2.4 points on LV-ViT-S but only 0.1--0.2 points on DeiT-S/B. Since §VI.A names throughput as a primary complexity metric and the abstract and conclusion promise a better trade-off, the paper should report a latency breakdown and a throughput-accuracy trade-off analysis, or explicitly limit the claim to FLOPs.
  2. [§IV-C, Eq. (10)] The approximation loss is normalized by N_s^b = sum_i M^s_{i,b}, which is the number of kept tokens, but the loss term is computed on the frozen tokens via the factor (1 - M^s_b). Unless the keep ratio is exactly 0.5, this denominator gives an incorrect normalization of the frozen-token error and contradicts the text that defines N_s^b as "the total number of the frozen tokens." The denominator should be sum_i (1 - M^s_{i,b}). Please correct this and verify that the main results are unchanged after retraining with the corrected normalization.
  3. [§V, Eqs. (16)--(18); Algorithm 1] The training procedure uses a masked-attention surrogate in which frozen tokens remain in the sequence but are prevented from interacting, whereas inference physically removes and later reinserts tokens via Eqs. (6)--(7). This train/inference mismatch is not quantified or discussed. The authors should provide a control experiment that trains with the actual inference schedule (or at least measures accuracy with and without the masked surrogate) to establish that the learned selectors transfer to the deployed skipping mechanism.
  4. [Table I; §VI-A] All reported accuracies appear to come from single runs without error bars or multiple seeds. Several head-to-head differences supporting the central claim are small (DeiT-S: 77.7 vs. 77.6 for IdleViT; DeiT-B: 79.6 vs. 79.4), so "consistently outperforms" is not statistically supported at those operating points. Please report mean and standard deviation over at least three seeds for the main comparisons, or identify the comparisons where the observed difference exceeds expected run-to-run noise.
minor comments (6)
  1. [§V] There is a typo in "instance-adaptive ToFe for sigle-instance inference": "sigle" should be "single."
  2. [§VI-A] The sentence "We employ a three-stage token reduction strategy analogous to existing methods" is repeated verbatim in consecutive sentences.
  3. [§IV-B, Eq. (9)] The module name is misspelled as "GumbleSoftmax" in Eq. (9) and in the text; it should be "GumbelSoftmax."
  4. [§III-A] The phrase "proportional to the quadratic number of the input tokens" should read "proportional to the square of the number of input tokens" or "quadratic in the number of input tokens."
  5. [§V, Algorithm 1] In instance-adaptive inference, line 8 obtains the mask according to Eqs. (8)--(9), but Eq. (9) includes Gumbel noise; if sampling is used at inference the output becomes stochastic. The algorithm should specify a deterministic argmax or zero temperature for deployment.
  6. [§VI-B, Table I] The footnote says different compression ratios were set to achieve "approximately 100% acceleration of the backbone models," but the reported GFLOPs correspond to roughly 50% FLOPs reduction; the phrasing is ambiguous and should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the selector and approximator are trained against an external teacher and a user-specified FLOPs budget, not against the paper's own conclusions.

full rationale

The paper's central components are a learned token selector (Eq. 8-9) and a learned token approximator (Eq. 5-7), optimized with classification loss, an approximation loss against the original pretrained teacher model (Eq. 10), and a computation-budget loss (Eq. 14). The approximation target is the teacher's per-token output, an external reference, not a quantity derived from ToFe's own final accuracy, so the self-distillation setup is not circular. The 50% FLOPs reduction is not a predicted output but a user-specified training constraint (targetFLOPs in Eq. 13-14); reporting the resulting FLOPs is reporting the experimental setting, not deriving the claim from itself. The motivating premise that token features change little across adjacent blocks is supported by the paper's own cosine-similarity measurements (Fig. 4b/c) and by an external citation to Vig and Belinkov [41], not by a self-citation chain. The only self-citations [39,40] are background references and are not load-bearing for any derivation. The skeptical concern that Table I shows ToFe is slower than IdleViT at matched FLOPs is a correctness and claims-support issue about throughput, not a circularity of the derivation chain. Overall, no step reduces by construction to its own inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests mainly on the empirical validity of the approximator and on the train/inference equivalence of masking. No new physical entities are introduced; the token selector and approximator are trained modules with hand-chosen hyperparameters.

free parameters (3)
  • Loss weights lambda_cls, lambda_apr, lambda_FLOPs = 1.0, 2.0, 5.0
    Chosen by hand in Sec. VI-A; they balance classification, approximation, and FLOPs-budget objectives and directly affect the accuracy/complexity trade-off.
  • targetFLOPs computation budget = e.g., 3.3 GFLOPs for LV-ViT-S, 2.0 GFLOPs for DeiT-S
    User-specified budget in Eq. (13)-(15); determines the optimized keep ratios and the reported operating point.
  • Token reduction stage locations = Blocks 4, 7, 10 for DeiT; 5, 9, 13 for LV-ViT
    Hand-selected in Sec. VI-A; all results depend on where freezing stages are inserted.
assumptions (4)
  • domain assumption Frozen token features change only slightly across several consecutive transformer blocks.
    Load-bearing premise for the lightweight approximator, stated in Sec. III-B and visualized in Fig. 4b/c; if false, reuse propagates stale features.
  • standard math A 2-layer MLP can approximate the residual update of frozen tokens (Universal Approximation Theorem).
    Invoked in Sec. IV-B [43]; existence result, does not guarantee the small MLP used here is sufficient.
  • ad hoc to paper Masked attention (Eqs. 17-18) and the masked forward (Eq. 16) are a faithful differentiable surrogate for actual token skipping at inference.
    Training removes frozen tokens via masks while inference physically skips blocks; the paper does not analyze this gap.
  • domain assumption The original backbone is a good teacher for the approximation loss (self-distillation).
    Used in Eq. (10); assumes teacher outputs are the right regression target for frozen-token estimates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ToFe: Lagged Token Freezing and Reusing for Efficient Vision Transformer Inference." pith.science (2026). https://pith.science/paper/HTKLSDZY

@misc{pith2026250716260,
  author       = {Pith},
  title        = {Pith review of: ToFe: Lagged Token Freezing and Reusing for Efficient Vision Transformer Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HTKLSDZY}},
  note         = {Machine review of arXiv:2507.16260}
}
read the original abstract

Although vision transformers (ViT) have shown remarkable success in various vision tasks, their computationally expensive self-attention hinder their deployment on resource-constrained devices. Token reduction, which discards less important tokens during forward propagation, has been proposed to enhance the efficiency of transformer models. However, existing methods handle unimportant tokens irreversibly, preventing their reuse in subsequent blocks. Considering that transformers focus on different information among blocks, tokens reduced in early blocks might be useful later. Furthermore, to adapt transformer models for resource-constrained devices, it is crucial to strike a balance between model performance and computational overhead. To address these challenges, in this paper, we introduce a novel Token Freezing and Reusing (ToFe) framework, where we identify important tokens at each stage and temporarily freeze the unimportant ones, allowing their lagged reusing at a later stage. Specifically, we design a prediction module for token identification and an approximate module for recovery of the frozen tokens. By jointly optimizing with the backbone through computation budget-aware end-to-end training, ToFe can adaptively process the necessary tokens at each block, thereby reducing computational cost while maintaining performance. Extensive experiments demonstrate that ToFe reduces the computational cost of LV-ViT model by 50% with less than 2% drop in Top-1 accuracy, achieving a better trade-off between performance and complexity compared to state-of-the-art methods.

Figures

Figures reproduced from arXiv: 2507.16260 by the authors.

Figure 1
Figure 1. Comparison of (a) Token pruning [15], [17], (b) Token Re [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An illustration of a typical transformer model with [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Visualization of [CLS] Token Attention. We use DeiT-S model and [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: (a) Visualization of the [CLS] attention values in 2-th, 5-th, 7-th, 9-th and 10-th blocks of DeiT-S model. The red parts of the heat map represent areas [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Overview of our proposed ToFe Framework. The ToFe framework enhances vanilla vision transformer models (e.g., ViT [37], DeiT [5], LV-ViT [23]) [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Trade-off between FLOPs and Top-1 accuracy of different token [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Comparison of trade-offs between model complexity and accuracy [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Visualization of tokens used at each stage. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 10
Figure 10. Figure 10: Visualization of token usage for samples with different complexities. [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 29 canonical work pages

  1. [1]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  2. [2]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  3. [3]

    (2022) Introducing chatgpt

    OpenAI. (2022) Introducing chatgpt. [Online]. Available: https: //openai.com/index/chatgpt/

    work page 2022

  4. [4]

    (2023) Github copilot: Your ai pair programmer

    Github. (2023) Github copilot: Your ai pair programmer. [Online]. Available: https://github.com/features/copilot

    work page 2023

  5. [5]

    Training data-efficient image transformers & distillation through attention,

    H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J ´egou, “Training data-efficient image transformers & distillation through attention,” in International conference on machine learning . PMLR, 2021, pp. 10 347–10 357

    2021

  6. [6]

    Tinymim: An empirical study of distilling mim pre-trained models,

    S. Ren, F. Wei, Z. Zhang, and H. Hu, “Tinymim: An empirical study of distilling mim pre-trained models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 3687–3697

    work page 2023

  7. [7]

    Llm-pruner: On the structural pruning of large language models,

    X. Ma, G. Fang, and X. Wang, “Llm-pruner: On the structural pruning of large language models,” Advances in neural information processing systems, vol. 36, pp. 21 702–21 720, 2023

    2023

  8. [8]

    Width & depth pruning for vision transformers,

    F. Yu, K. Huang, M. Wang, Y . Cheng, W. Chu, and L. Cui, “Width & depth pruning for vision transformers,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 3, 2022, pp. 3143– 3151

    work page 2022

Show all 46 references
  1. [9]

    Towards accurate post-training quantization for vision transformer,

    Y . Ding, H. Qin, Q. Yan, Z. Chai, J. Liu, X. Wei, and X. Liu, “Towards accurate post-training quantization for vision transformer,” in Proceedings of the 30th ACM international conference on multimedia , 2022, pp. 5380–5388

    2022

  2. [10]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,

    W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 568–578

    2021

  3. [11]

    Co-scale conv-attentional image transformers,

    W. Xu, Y . Xu, T. Chang, and Z. Tu, “Co-scale conv-attentional image transformers,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 9981–9990

    2021

  4. [12]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022

    2021

  5. [13]

    Tokens-to-token vit: Training vision transformers from scratch on imagenet,

    L. Yuan, Y . Chen, T. Wang, W. Yu, Y . Shi, Z.-H. Jiang, F. E. Tay, J. Feng, and S. Yan, “Tokens-to-token vit: Training vision transformers from scratch on imagenet,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 558–567

    2021

  6. [14]

    Not all patches are what you need: Expediting vision transformers via token reorganizations,

    Y . Liang, C. Ge, Z. Tong, Y . Song, J. Wang, and P. Xie, “Not all patches are what you need: Expediting vision transformers via token reorganizations,” arXiv preprint arXiv:2202.07800 , 2022

    2022 arXiv

  7. [15]

    Dynam- icvit: Efficient vision transformers with dynamic token sparsification,

    Y . Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsieh, “Dynam- icvit: Efficient vision transformers with dynamic token sparsification,” Advances in neural information processing systems, vol. 34, pp. 13 937– 13 949, 2021

    2021

  8. [16]

    Token merging: Your vit but faster,

    D. Bolya, C.-Y . Fu, X. Dai, P. Zhang, C. Feichtenhofer, and J. Hoffman, “Token merging: Your vit but faster,” in The Eleventh International Conference on Learning Representations , 2023

    2023

  9. [17]

    Adaptive sparse vit: towards learnable adaptive token pruning by fully exploiting self-attention,

    X. Liu, T. Wu, and G. Guo, “Adaptive sparse vit: towards learnable adaptive token pruning by fully exploiting self-attention,” in Proceed- ings of the Thirty-Second International Joint Conference on Artificial Intelligence, 2023, pp. 1222–1230

    2023

  10. [18]

    A simple romance between multi-exit vision transformer and token reduction,

    D. Liu, M. Kan, S. Shan, and C. Xilin, “A simple romance between multi-exit vision transformer and token reduction,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  11. [19]

    Synergistic patch pruning for vision transformer: Unifying intra-& inter-layer patch importance,

    Y . Zhang, L. Wei, and N. Freris, “Synergistic patch pruning for vision transformer: Unifying intra-& inter-layer patch importance,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  12. [20]

    Diffrate: Differentiable compression rate for efficient vision transformers,

    M. Chen, W. Shao, P. Xu, M. Lin, K. Zhang, F. Chao, R. Ji, Y . Qiao, and P. Luo, “Diffrate: Differentiable compression rate for efficient vision transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 17 164–17 174

    2023

  13. [21]

    Beyond attentive tokens: Incorporating token importance and diversity for efficient vision transformers,

    S. Long, Z. Zhao, J. Pi, S. Wang, and J. Wang, “Beyond attentive tokens: Incorporating token importance and diversity for efficient vision transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 10 334–10 343

    2023

  14. [22]

    Joint token pruning and squeezing towards more aggressive compression of vision transformers,

    S. Wei, T. Ye, S. Zhang, Y . Tang, and J. Liang, “Joint token pruning and squeezing towards more aggressive compression of vision transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2092–2101

    2023

  15. [23]

    Token labeling: Training a 85.5% top-1 accuracy vision transformer with 56m parameters on imagenet,

    Z. Jiang, Q. Hou, L. Yuan, D. Zhou, X. Jin, A. Wang, and J. Feng, “Token labeling: Training a 85.5% top-1 accuracy vision transformer with 56m parameters on imagenet,” arXiv preprint arXiv:2104.10858 , vol. 3, no. 6, p. 7, 2021

    2021 arXiv

  16. [24]

    [Online]

    (2021) Visualizer. [Online]. Available: https://github.com/luo3300612/ Visualizer

    2021

  17. [25]

    Categorical reparameterization with gumbel-softmax,

    E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” arXiv preprint arXiv:1611.01144 , 2016

    2016 arXiv

  18. [26]

    The concrete distribution: A continuous relaxation of discrete random variables,

    C. J. Maddison, A. Mnih, and Y . W. Teh, “The concrete distribution: A continuous relaxation of discrete random variables,” arXiv preprint arXiv:1611.00712, 2016

    2016 arXiv

  19. [27]

    All tokens matter: Token labeling for training better vi- sion transformers,

    Z.-H. Jiang, Q. Hou, L. Yuan, D. Zhou, Y . Shi, X. Jin, A. Wang, and J. Feng, “All tokens matter: Token labeling for training better vi- sion transformers,” Advances in neural information processing systems , vol. 34, pp. 18 590–18 602, 2021

    2021

  20. [28]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255

    2009

  21. [29]

    Crossvit: Cross-attention multi- scale vision transformer for image classification,

    C.-F. R. Chen, Q. Fan, and R. Panda, “Crossvit: Cross-attention multi- scale vision transformer for image classification,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 357– 366

    2021

  22. [30]

    Condi- tional positional encodings for vision transformers,

    X. Chu, Z. Tian, B. Zhang, X. Wang, and C. Shen, “Condi- tional positional encodings for vision transformers,” arXiv preprint arXiv:2102.10882, 2021

    2021 arXiv

  23. [31]

    Designing network design spaces,

    I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Doll ´ar, “Designing network design spaces,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 10 428–10 436

    2020

  24. [32]

    Efficientnet: Rethinking model scaling for con- volutional neural networks,

    M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for con- volutional neural networks,” in International conference on machine learning. PMLR, 2019, pp. 6105–6114

    2019

  25. [33]

    High-performance large-scale image recognition without normalization,

    A. Brock, S. De, S. L. Smith, and K. Simonyan, “High-performance large-scale image recognition without normalization,” in International conference on machine learning . PMLR, 2021, pp. 1059–1071

    2021

  26. [34]

    Ia- red2: Interpretability-aware redundancy reduction for vision transform- ers,

    B. Pan, R. Panda, Y . Jiang, Z. Wang, R. Feris, and A. Oliva, “Ia- red2: Interpretability-aware redundancy reduction for vision transform- ers,” Advances in Neural Information Processing Systems , vol. 34, pp. 24 898–24 911, 2021

    2021

  27. [35]

    Evo-vit: Slow-fast token evolution for dynamic vision transformer,

    Y . Xu, Z. Zhang, M. Zhang, K. Sheng, K. Li, W. Dong, L. Zhang, C. Xu, and X. Sun, “Evo-vit: Slow-fast token evolution for dynamic vision transformer,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 3, 2022, pp. 2964–2972

    2022

  28. [36]

    Token fusion: Bridging the gap between token pruning and token merging,

    M. Kim, S. Gao, Y .-C. Hsu, Y . Shen, and H. Jin, “Token fusion: Bridging the gap between token pruning and token merging,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2024, pp. 1383–1392

    2024

  29. [37]

    An image is worth 16x16 words: Trans- formers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Trans- formers for image recognition at scale,” in International Conference on Learni...

    2021

  30. [38]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi...

    2019

  31. [39]

    Edge learning: The enabling technology for distributed big data analytics in the edge,

    J. Zhang, Z. Qu, C. Chen, H. Wang, Y . Zhan, B. Ye, and S. Guo, “Edge learning: The enabling technology for distributed big data analytics in the edge,” ACM Computing Surveys (CSUR) , vol. 54, no. 7, pp. 1–36, 2021

    2021

  32. [40]

    Otas: An elastic transformer serving system via token adaptation,

    J. Chen, W. Xu, Z. Hong, S. Guo, H. Wang, J. Zhang, and D. Zeng, “Otas: An elastic transformer serving system via token adaptation,” arXiv preprint arXiv:2401.05031 , 2024

    2024 arXiv

  33. [41]

    Analyzing the structure of attention in a transformer language model,

    J. Vig and Y . Belinkov, “Analyzing the structure of attention in a transformer language model,” arXiv preprint arXiv:1906.04284 , 2019

    1906 arXiv

  34. [42]

    Xception: Deep learning with depthwise separable convolu- tions,

    F. Chollet, “Xception: Deep learning with depthwise separable convolu- tions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1251–1258. 12

    2017

  35. [43]

    Approximation by superpositions of a sigmoidal function,

    G. Cybenko, “Approximation by superpositions of a sigmoidal function,” Mathematics of control, signals and systems , vol. 2, no. 4, pp. 303–314, 1989

    1989

  36. [44]

    Learned thresholds token merging and pruning for vision transformers,

    M. Bonnaerens and J. Dambre, “Learned thresholds token merging and pruning for vision transformers,” arXiv preprint arXiv:2307.10780 , 2023

    2023 arXiv

  37. [45]

    Ppt: Token pruning and pooling for efficient vision transformers,

    X. Wu, F. Zeng, X. Wang, and X. Chen, “Ppt: Token pruning and pooling for efficient vision transformers,” arXiv preprint arXiv:2310.01812 , 2023

    2023 arXiv

  38. [46]

    No token left behind: Efficient vision transformer via dynamic token idling,

    X. Xu, C. Li, Y . Chen, X. Chang, J. Liu, and S. Wang, “No token left behind: Efficient vision transformer via dynamic token idling,” in Australasian Joint Conference on Artificial Intelligence. Springer, 2023, pp. 28–41

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.