Pith. sign in

REVIEW 3 major objections 6 minor 41 references

Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that token merging in Vision Transformers should be driven by a separate learned embedding, and that training this embedding through a continuous relaxation of the merging step improves the speed-accuracy trade-off of…

desk verdict A modest but consistent improvement over ToMe via learned decoupled merging embeddings; the main open question is whether the soft training surrogate really transfers to hard inference as claimed. read the letter →

arxiv 2412.10569 v1 pith:FAOQDBSJ submitted 2024-12-13 cs.CV cs.AI

Dong Hoon Lee , Seunghoon Hong This is my paper · ORCID
classification cs.CVcs.AI
keywords VisionTransformerstokenmergingreductiondecoupledembeddingcontinuousrelaxationefficientinferenceimageclassificationsemanticsegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Token merging makes Vision Transformers faster by combining similar tokens, but existing methods decide what to merge using the same intermediate features the ViT uses for encoding. This paper claims those features are the wrong tool: the merging policy needs its own representation, learned specifically for merging. DTEM adds a per-block linear projection that maps detached ViT features into a small similarity space, and trains that projection by replacing the hard matching step with a soft, differentiable version. If the claim is right, a frozen pre-trained ViT can be sped up by training only this small module, with consistent accuracy gains over ToMe across classification, captioning, and segmentation. The paper also notes that it has so far been validated only on vision tasks and does not reduce training-time compute.

What carries the argument

The central object is the decoupled embedding module, $Z = f(X; \phi_l)$, one affine layer per block that maps the block input to a low-dimensional similarity space ($d'=64$ for ViT-S/B, $128$ for ViT-L) and defines token similarity by cosine distance. The training mechanism is a continuous relaxation of Bipartite Soft Matching: a soft top-$r$ selection built from temperature-softmax steps with suppression of already-selected source nodes (Eqs. 6-7), normalized to a soft adjacency matrix (Eq. 8), followed by soft merging updates that reweight features and effective token sizes (Eqs. 9-10). This relaxation is what allows gradients from the task loss to reach the merging policy, and as $\tau \to 0$ it converges to the discrete matching used at inference. The paper also uses detached ViT features as input to the embedding module, so the module alone is trained by the relaxed merging signal.

What would settle it

Train DTEM modularly on ImageNet at $\tau=0.1$, then measure agreement between the token pairs selected by hard bipartite matching at inference and the pairs receiving the largest soft weights during training; if the agreement is low, or if lowering the training temperature to $0.01$ does not raise both agreement and final accuracy, the surrogate is not faithful to the inference-time merging policy.

Watch

Extended reading notes

Core claim

The central claim is that a decoupled, per-block token embedding—computed by a single linear layer from detached ViT features and used only to compute cosine similarities—learns a better merging policy than the ViT's own keys or token embeddings. To make this learnable, DTEM replaces ToMe's hard bipartite soft matching with a temperature-softmax relaxation of sequential top-r selection, normalized into a soft adjacency matrix, and merges tokens with asynchronous weighted updates. At inference the soft operators are swapped for hard bipartite soft matching, so the method keeps ToMe's speed while changing which tokens get merged. In modular training on frozen DeiT-S/B and MAE-B/L models, DTEM reports +0.15 to +0.47 accuracy over ToMe at 35% FLOP reduction and +0.47 to +1.64 at 50% reduction; in end-to-end fine-tuning on DeiT-S it reaches 79.85% top-1 at 2.9 GFLOPs, a 37.2% FLOP cut. The same recipe improves CIDEr scores by +2.3 to +6.0 in captioning and mIoU by +0.32 to +1.3 in segmentation at matched compute.

Load-bearing premise

The load-bearing premise is that the softened merging decisions used during training are a faithful stand-in for the hard merging decisions made at inference; the paper shows this only as the smoothing temperature goes to zero, while training runs at a nonzero temperature ($\tau = 0.1$).

Editorial extensions

If this is right

  • A frozen pre-trained ViT can be made faster without fine-tuning its weights: only the per-block embedding modules are trained, and the paper reports gains starting from 0.31% of ImageNet and after one epoch.
  • A single DTEM model trained at reduction rate $r$ can be evaluated at any lower rate, so one training run covers multiple inference FLOPs budgets.
  • Because the embedding module adds less than 1% of parameters and FLOPs, the accuracy gain is nearly pure policy improvement rather than extra capacity.
  • The decoupled-policy recipe transfers across tasks: classification, image captioning, and semantic segmentation all show better accuracy at matched compute than merging with backbone features.
  • In end-to-end fine-tuning, DTEM improves on both ToMe and EViT at equal FLOPs and also matches longer-trained pruning methods at 100 epochs with only the task loss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The soft-to-hard transfer gap is the main extension risk: because training runs at $\tau=0.1$ while inference is hard matching, the reported gains could shrink if the learned embedding overfits the relaxed operator.
  • The same decoupling idea could be dropped into other token-reduction mechanisms—clustering-based merging or importance-based pruning—by replacing only the similarity source with a trained detached projection.
  • The visualization evidence suggests the learned embedding encodes a foreground/background preference; that signal could be reused as a saliency map or combined with a semantic loss to make merging more content-aware.
  • Jointly learning which block drops how many tokens (a reduction profile) together with the decoupled embedding is a natural next step that the paper identifies as orthogonal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Decoupled Token Embedding for Merging (DTEM), a method for token reduction in Vision Transformers. Instead of computing token similarity from the ViT's intermediate features, DTEM learns a lightweight per-block linear embedding module whose cosine similarities drive the merging policy. Because the grouping and merging operators are discrete, the module is trained through a continuous relaxation of ToMe's bipartite soft matching: soft grouping via a differentiable top-k operator and soft merging via weighted averaging with size tracking. At inference, the hard BSM operator is used. DTEM is evaluated on ImageNet classification (modular and end-to-end), COCO captioning, and ADE20K segmentation, claiming consistent improvements over ToMe and other baselines at matched FLOPs, typically +0.1 to +0.5 accuracy points with larger gains at higher reduction rates. The paper includes ablations showing the decoupled embedding is essential, and that a single model trained at a high reduction rate transfers to lower rates.

Significance. If the results hold, DTEM is a useful and simple add-on for making frozen ViTs faster with a small trained module, and it demonstrates a practical way to train through a continuous relaxation of token merging. The contribution is incremental but clearly positive: the gains are small in classification but more substantial in captioning and segmentation. Strengths: code is released; the experimental coverage is broad (three tasks, five backbones, both modular and end-to-end training); and the ablation in Table 6 directly shows that naive soft merging on raw keys hurts, which supports the core design. The main risks are the unverified soft-to-hard transfer of the relaxation, the lack of error bars for small classification differences, and the training-status imbalance in headline comparisons to ToMe.

major comments (3)
  1. [Secs. 3.2-3.4] The paper's central novelty is learning the decoupled embedding through the continuously relaxed grouping and merging operators (Eqs. 6-10), but the only justification that this surrogate matches the deployed hard BSM is an asymptotic statement as tau -> 0. Training uses tau = 0.1 and additionally scales the similarity matrix by 0.1 (Sec. 4.1, Implementation details), so the effective softmax temperature is near 1, far from the asymptotic regime. The manuscript never measures the agreement between the soft adjacency matrix E~ and the hard BSM adjacency E' at the trained embeddings. Without such a measurement (or a comparison to a straight-through estimator of BSM), the claim that the soft relaxation is what confers the improvement is not established. Please add an experiment quantifying the soft-to-hard gap, e.g., the proportion of selected edges that coincide, or a training variant that optimizes hard BSM directly.
  2. [Sec. 4.1, Tables 1-3, Fig. 2] The headline classification gains are small: in Figure 2 the improvements over ToMe are 0.12-0.2% for DeiT-S, and in Table 3, 0.12-0.23% over ToMe at matched FLOPs. No multiple seeds, confidence intervals, or significance tests are reported anywhere in the paper. Given that the reported gains are of the same order as typical run-to-run variance for ImageNet fine-tuning, the central claim of 'consistently improves' is not statistically supported as presented. Please report at least three seeds with standard deviations for the key comparisons (Tables 1, 3, and Figure 2), and state whether the improvements are significant.
  3. [Table 1 and Sec. 4.1] Table 1 compares DTEM against ToMe and EViT in the 'off-the-shelf frozen' setting, but ToMe and EViT as used there are training-free, while DTEM has been trained on the target task (ImageNet) for 30 epochs. The improvement may therefore reflect the benefit of task-specific training of the merging policy rather than the decoupled-embedding design per se. The controls in Table 8 (ToMe + Gumbel-Softmax + soft merging) are the right type of comparison, but they are relegated to an ablation and use a different training setup. Please include a trained ToMe-based alternative (e.g., ToMe+GS or a trained linear similarity head) in the main comparison table, or clearly state the training budgets and frame Table 1 as 'trained vs. training-free'.
minor comments (6)
  1. [Eq. (7)] The notation in Eq. (7) is ambiguous: the update log(1 - sum_j a^t_ij) suppresses rows with any soft mass, but the text says it suppresses 'the entire outbounding edges from the softly selected nodes in A'; please clarify that the sum is over j in B and that the update applies to all rows i, not only the selected ones.
  2. [Table 14] The appendix uses 'dTPS' while the main text and reference [33] use 'eTPS'; please unify the abbreviation.
  3. [A.1] In the appendix, the sentence 'In Table 12, we report Figure 4 results across a broader reduction range' should read 'Table 4 results'.
  4. [Sec. 3.3 and A.2] The description of how zero-size tokens are handled differs between Sec. 3.3 ('such tokens will be excluded from the subsequent merging process by Eq. 9') and Appendix A.2, which says the r tokens with minimum effective size are successively excluded each block; please reconcile these statements.
  5. [Table 1] The column header 'Reduction' is described as 'roughly represents the decreases in FLOPs', but the FLOPs differ across methods at the same nominal reduction (e.g., DeiT-S 35%: ToMe 3.02 GFLOPs vs DTEM 2.91 GFLOPs); consider reporting exact FLOPs-reduction percentages per method rather than a single nominal value.
  6. [Sec. 4.2] The captioning comparisons are at slightly different reduction levels (e.g., ToMe 32% vs DTEM 31%); since CIDEr is sensitive to the exact token count, please match reduction levels more closely or interpolate the baseline curve.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DTEM's learned embeddings are trained under a relaxed surrogate and evaluated on held-out external benchmarks against independent baselines.

full rationale

The paper's central claim is that a lightweight decoupled embedding module, trained through a continuously relaxed token-merging surrogate, improves hard token merging at inference. This is not circular: the embeddings are trained with the task loss on training splits, and the reported gains are measured on held-out validation/test sets (ImageNet-1k validation, COCO test, ADE20K) against independent baselines (ToMe, EViT, etc.). The soft grouping (Eqs. 6-8) is constructed as a relaxation of BSM using an external differentiable top-k operator [36], and soft merging (Eqs. 9-10) is a weighted-average pooling; neither equation is defined in terms of the reported accuracy numbers. The inference procedure replaces the relaxation with hard BSM, so the final evaluation does not use the soft adjacency as its own metric. The skeptic's concern about soft-to-hard transfer at tau=0.1 is a legitimate robustness/validation question about whether the training surrogate matches the deployment operator, but it is not a circularity: the paper does not define 'improvement' as the training objective itself, and the hard-inference results are external measurements. The paper's own limitations statement (A.4) and the asymptotic-equivalence assertion in Sec. 3.4 identify support gaps, but these are empirical validation gaps rather than circular derivations. Hyperparameter choices (tau, embedding dimension, training reduction rate) are tuned in the conventional sense and do not amount to fitting the reported held-out numbers. No load-bearing self-citation, uniqueness theorem, or ansatz-smuggling-via-citation is present. The paper is self-contained against external benchmarks, so the appropriate circularity finding is none.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claim relies on several hand-tuned hyperparameters and on the assumption that the soft relaxation is a faithful surrogate for hard BSM inference. No new physical entities are introduced. The decoupled embedding module is a learned construct with empirical validation.

free parameters (5)
  • Temperature scale tau = 0.1
    Controls softmax sharpness in soft grouping (Eq. 6); tuned on validation, ablated in Table 16 with best range 0.1 to 0.3.
  • Similarity scaling factor = 0.1 (applied to S before soft grouping)
    Implementation detail in Appendix A.2; acts as a second temperature-like scale and is hand-tuned without a separate ablation.
  • Decoupled embedding dimension d' = 64 for ViT-S/B, 128 for ViT-L
    Chosen by hand; ablation in Fig. 3(a) shows 64 gives a good trade-off, but gains are similar across 32 to 128.
  • Training reduction rate r_train = 16 (DeiT-S/B), 8 (ViT-L), 13 (GIT), 48 (AugReg ViT-S)
    Set to the maximum inference reduction; Table 9 shows lower training rates generalize but with slightly lower accuracy. Hand-selected.
  • End-to-end ViT update reduction rate and alternation ratio = r=13 for ViT updates; 1 embedding update per 9 ViT updates
    Introduced in Sec. 3.4 and Appendix A.2; tuned to save compute, with no ablation shown for the alternation ratio.
assumptions (5)
  • domain assumption Bipartite Soft Matching (ToMe) with proportional attention is an effective and faithful token merging framework.
    The method is built on ToMe's BSM as the target operator (Secs. 2 and 3.2). If BSM itself were flawed, DTEM inherits the flaw.
  • ad hoc to paper The Xie-Ermon relaxed top-k operator (Eqs. 6-8) is a valid differentiable approximation of hard top-k matching at the chosen temperature.
    Only asymptotic convergence as tau to 0 is argued; no proof or empirical analysis shows the approximation is accurate at tau=0.1, which is the value used in training.
  • domain assumption A detached linear projection of intermediate features contains enough information to learn a good merging policy.
    Appendix A.2 detaches the input to the embedding module, so the success of the method depends on the linear module extracting merging-relevant structure from frozen features.
  • domain assumption The task loss (classification, captioning, or segmentation) provides sufficient training signal to shape a good merging policy without auxiliary merging supervision.
    Central to the method; no direct supervision on merging quality is used, only the downstream task loss.
  • ad hoc to paper Soft merging with retained tokens and size updates (Eqs. 9-10) correctly simulates hard merging for training purposes.
    The asynchronous updates and stop-gradient clipping are heuristic; convergence properties and bias relative to hard merging are not analyzed.
invented entities (1)
  • Decoupled token embedding module (per-block learned projection Z=f(X;phi)) independent evidence
    purpose: Produces a dedicated similarity space for token merging, separate from the ViT's task features
    Validated by ablations (Table 6), the Kendall correlation analysis (Table 7), and cross-task experiments. It is a learned component rather than a physical entity, but evidence supports its role.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers." pith.science (2026). https://pith.science/paper/FAOQDBSJ

@misc{pith2026241210569,
  author       = {Pith},
  title        = {Pith review of: Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FAOQDBSJ}},
  note         = {Machine review of arXiv:2412.10569}
}
read the original abstract

Recent token reduction methods for Vision Transformers (ViTs) incorporate token merging, which measures the similarities between token embeddings and combines the most similar pairs. However, their merging policies are directly dependent on intermediate features in ViTs, which prevents exploiting features tailored for merging and requires end-to-end training to improve token merging. In this paper, we propose Decoupled Token Embedding for Merging (DTEM) that enhances token merging through a decoupled embedding learned via a continuously relaxed token merging process. Our method introduces a lightweight embedding module decoupled from the ViT forward pass to extract dedicated features for token merging, thereby addressing the restriction from using intermediate features. The continuously relaxed token merging, applied during training, enables us to learn the decoupled embeddings in a differentiable manner. Thanks to the decoupled structure, our method can be seamlessly integrated into existing ViT backbones and trained either modularly by learning only the decoupled embeddings or end-to-end by fine-tuning. We demonstrate the applicability of DTEM on various tasks, including classification, captioning, and segmentation, with consistent improvement in token merging. Especially in the ImageNet-1k classification, DTEM achieves a 37.2% reduction in FLOPs while maintaining a top-1 accuracy of 79.85% with DeiT-small. Code is available at \href{https://github.com/movinghoon/dtem}{link}.

Figures

Figures reproduced from arXiv: 2412.10569 by the authors.

Figure 1
Figure 1. Comparison of our method with conventional token merging. Contrary to prior works that merge tokens directly based on intermediate features in ViT, our method leverages a decoupled embedding to extract features tailored for token merging. The embedding module is trained via continuous relaxation of grouping and merging operators, i.e., soft grouping and merging, respectively, that allow differentiation. achieve impr… view at source ↗
Figure 2
Figure 2. Classification results under differ￾ent FLOPs and throughputs. All methods are end-to-end trained [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Ablation study on decoupled embed￾ding module design: (a) decoupled embedding dimension and (b) number of hidden layers. Results [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Image classification results on data and train efficiency: (a) dataset size and (b) training epochs. In the experiments, DTEM is modularly trained on DeiT-S model, while ToMe undergoes end-to-end training [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 6
Figure 6. Figure 6: More visualization of merged tokens. We apply a reduction profile with [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 35 canonical work pages

  1. [1]

    Anderson, B

    P. Anderson, B. Fernando, M. Johnson, and S. Gould. Spice: Semantic propositional image caption evaluation. In European conference on computer vision (ECCV), 2016

    work page 2016

  2. [2]

    Bolya, C.-Y

    D. Bolya, C.-Y . Fu, X. Dai, P. Zhang, C. Feichtenhofer, and J. Hoffman. Token merging: Your vit but faster. In International Conference on Learning Representations (ICLR), 2023

    work page 2023

  3. [3]

    Bolya and J

    D. Bolya and J. Hoffman. Token merging for fast stable diffusion.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023

    work page 2023

  4. [4]

    Carion, F

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision (ECCV), 2020

    work page 2020

  5. [5]

    M. Chen, W. Shao, P. Xu, M. Lin, K. Zhang, F. Chao, R. Ji, Y . Qiao, and P. Luo. Diffrate: Dif- ferentiable compression rate for efficient vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023

    work page 2023

  6. [6]

    X. Chen, H. Fang, T.-Y . Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015

    arXiv 2015

  7. [7]

    Contributors

    M. Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and bench- mark. https://github.com/open-mmlab/mmsegmentation, 2020

    work page 2020

  8. [8]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (CVPR), 2009

    work page 2009

Show all 41 references
  1. [9]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning R...

    2021

  2. [10]

    Fayyaz, S

    M. Fayyaz, S. A. Koohpayegani, F. R. Jafari, S. Sengupta, H. R. V . Joze, E. Sommerlade, H. Pirsiavash, and J. Gall. Adaptive token sampling for efficient vision transformers. In European Conference on Computer Vision (ECCV), 2022

    2022

  3. [11]

    K. He, X. Chen, S. Xie, Y . Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2022

    2022

  4. [12]

    J. H. Heo, A. Fayyazi, M. Nazemi, and M. Pedram. A fast training-free compression framework for vision transformers. arXiv preprint arXiv:2303.02331, 2023

    2023 arXiv

  5. [13]

    Jiang, Q

    Z.-H. Jiang, Q. Hou, L. Yuan, D. Zhou, Y . Shi, X. Jin, A. Wang, and J. Feng. All tokens matter: Token labeling for training better vision transformers. Advances in neural information processing systems (NeurIPS), 2021

    2021

  6. [14]

    Karpathy and L

    A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2015

    2015

  7. [15]

    Kirillov, E

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, et al. Segment anything.arXiv preprint arXiv:2304.02643, 2023

    2023 arXiv

  8. [16]

    D. Li, J. Li, H. Le, G. Wang, S. Savarese, and S. C. Hoi. Lavis: A library for language-vision intelligence. arXiv preprint arXiv:2209.09019, 2022

    2022 arXiv

  9. [17]

    J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023

    2023 arXiv

  10. [18]

    Liang, C

    Y . Liang, C. GE, Z. Tong, Y . Song, J. Wang, and P. Xie. EVit: Expediting vision transformers via token reorganizations. In International Conference on Learning Representations (ICLR), 2022. 11

    2022

  11. [19]

    D. Liu, M. Kan, S. Shan, and X. CHEN. A simple romance between multi-exit vision transformer and token reduction. In International Conference on Learning Representations (ICLR), 2024

    2024

  12. [20]

    S. Long, Z. Zhao, J. Pi, S. Wang, and J. Wang. Beyond attentive tokens: Incorporating token importance and diversity for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  13. [21]

    Marin, J.-H

    D. Marin, J.-H. R. Chang, A. Ranjan, A. Prabhu, M. Rastegari, and O. Tuzel. Token pooling in vision transformers for image classification. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023

    2023

  14. [22]

    L. Meng, H. Li, B.-C. Chen, S. Lan, Z. Wu, Y .-G. Jiang, and S.-N. Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  15. [23]

    B. Pan, R. Panda, Y . Jiang, Z. Wang, R. Feris, and A. Oliva. IA-RED$^2$: Interpretability-aware redundancy reduction for vision transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  16. [24]

    Papineni, S

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (ACL), 2002

    2002

  17. [25]

    Y . Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  18. [26]

    Renggli, A

    C. Renggli, A. S. Pinto, N. Houlsby, B. Mustafa, J. Puigcerver, and C. Riquelme. Learning to merge tokens in vision transformers. arXiv preprint arXiv:2202.12015, 2022

    2022 arXiv

  19. [27]

    A. P. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research (TMLR), 2022

    2022

  20. [28]

    Strudel, R

    R. Strudel, R. Garcia, I. Laptev, and C. Schmid. Segmenter: Transformer for semantic segmen- tation. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), 2021

    2021

  21. [29]

    Touvron, M

    H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou. Training data- efficient image transformers & distillation through attention. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedi...

    2021

  22. [30]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems (NeurIPS), 2017

    2017

  23. [31]

    Vedantam, C

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2015

    2015

  24. [32]

    J. Wang, Z. Yang, X. Hu, L. Li, K. Lin, Z. Gan, Z. Liu, C. Liu, and L. Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022

    2022 arXiv

  25. [33]

    S. Wei, T. Ye, S. Zhang, Y . Tang, and J. Liang. Joint token pruning and squeezing towards more aggressive compression of vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  26. [34]

    Wightman

    R. Wightman. Pytorch image models. https://github.com/rwightman/ pytorch-image-models, 2019. 12

    2019

  27. [35]

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Transformers: State-of-the-art natu...

    2020

  28. [36]

    S. M. Xie and S. Ermon. Reparameterizable subset sampling via continuous relaxations. In International Joint Conference on Artificial Intelligence (IJCAI), 2019

    2019

  29. [37]

    J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  30. [38]

    Y . Xu, Z. Zhang, M. Zhang, K. Sheng, K. Li, W. Dong, L. Zhang, C. Xu, and X. Sun. Evo- vit: Slow-fast token evolution for dynamic vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022

    2022

  31. [39]

    H. Yin, A. Vahdat, J. M. Alvarez, A. Mallya, J. Kautz, and P. Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  32. [40]

    W. Zeng, S. Jin, W. Liu, C. Qian, P. Luo, W. Ouyang, and X. Wang. Not all tokens are equal: Human-centric visual analysis via token clustering transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  33. [41]

    B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2017. 13 A Appendix A.1 More Results Full Classification Results Table 10 and Tabl...

    2017

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.