Pith. sign in

REVIEW 4 major objections 6 minor 78 references

Representation Shift: Unifying Token Compression with FlashAttention

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes Representation Shift, a training-free token-importance metric that measures L2 change through a layer, making it compatible with FlashAttention and generalizable beyond transformers.

desk verdict A simple, plausible training-free token-importance heuristic that is FlashAttention-compatible, but the evaluation overclaims: throughput gains are mostly kernel, the metric is tuned on test benchmarks, and 'training-free' breaks for CNNs. read the letter →

arxiv 2508.00367 v1 pith:JYUAVHJ7 submitted 2025-08-01 cs.CV

Joonmyung Choi , Sanghyeok Lee , Byungoh Ko , Eunseo Kim , Jihyung Kil , Hyunwoo J. Kim This is my paper · ORCID
classification cs.CV
keywords tokencompressionpruningFlashAttentionrepresentationshiftvisiontransformervideo-textretrievaltraining-freemodel-agnostic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Token pruning usually needs an attention map to decide which tokens to keep, but FlashAttention deliberately never materializes that map. This paper proposes Representation Shift, a training-free importance score that measures how far each token's representation moves (L2 distance) through a layer, and uses it to drop the least-changed tokens. If correct, the score makes token compression compatible with FlashAttention and with any architecture, since it does not depend on attention at all. The reported payoff is up to 5.5x throughput on UMT video-text retrieval and around 4x on video QA, with small accuracy losses, plus working extensions to ResNets and state space models.

What carries the argument

The central object is the per-token scalar \(s_i = \|\mathrm{MLP}(\mathrm{LN}(x'_i)) - x'_i\|_2\), the L2 distance between a token's representation before and after the MLP (multilayer perceptron) sublayer. It is a token importance score that requires only the MLP input and output, so it sidesteps the attention map entirely; the pruning decision then removes the tokens with the smallest shifts at selected layers.

What would settle it

Prune tokens by representation shift and by random choice at the same ratio on a held-out set of tasks; if random pruning matches or beats shift-based pruning in accuracy at the same speed, the claimed importance signal is not real. A sharper check is to correlate shift rankings with per-token leave-one-out accuracy impact across layers and show the correlation is consistently positive.

Watch

Extended reading notes

Core claim

The central claim is that the magnitude of a token's representation change across a layer is a reliable proxy for its informativeness. Formally, for input tokens \(x\) and a layer transformation \(F\), the importance score is \(s=\$\Delta$ x=D(F(x),x)\), with \(D\) chosen as the L2 norm and \(F\) as the MLP sublayer after ablation. The hypothesis is that networks amplify critical tokens (foreground content, task-relevant patches) while barely updating redundant ones; the paper supports this with pruning experiments, top/bottom-50% retention tests, and visualizations. Because the score needs only the layer's input and output, it is available under FlashAttention, which never exposes attention maps, and it transfers to CNNs and state space models by measuring the shift at the corresponding stage.

Load-bearing premise

The load-bearing premise is that a token's L2 representation change through a layer is a valid measure of its importance; if that correlation fails for some layer, model, or task, the pruning signal no longer identifies what to keep.

Editorial extensions

If this is right

  • Token pruning no longer needs attention maps, so it can be stacked on FlashAttention; the reported result is up to 5.5x throughput on UMT-L video-text retrieval and about 4x on video QA with modest accuracy loss.
  • On ImageNet classification with DeiT-T/S/B, representation-shift pruning beats attention-based class-token scoring at the same pruning ratio.
  • The criterion transfers to non-attention architectures: line-wise and token-wise pruning of ResNet and token pruning of Vision Mamba (ViM-T) preserve accuracy while raising throughput.
  • Existing training-free token merging methods such as vid-TLDR can swap their attention-based importance scores for representation shift and then run under FlashAttention.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the metric makes no architectural assumptions, it should apply to decoder-only language models running FlashAttention, where training-free token pruning is currently blocked by the same attention-map unavailability; testing it on long-context LLMs is a natural next step.
  • Beyond the paper: the authors' own ablations show that the best operation (MLP) and distance (L2) are empirical choices, so the heuristic may need per-architecture tuning rather than being universally automatic.
  • Beyond the paper: the same shift signal could serve as a post-hoc saliency map for interpretability, since the visualizations suggest it highlights foreground objects; this could be tested against human-annotated segmentation.
  • Beyond the paper: the speedups are GPU-throughput numbers, and end-to-end latency on other hardware (CPU, mobile, or different GPUs) may not scale the same way, so deploy-target measurements would be needed to confirm the gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Representation Shift, a token-importance metric defined as the L2 distance between a token's representation before and after a network layer (defaulting to the MLP block in Transformers). The authors claim this metric is training-free, model-agnostic, and independent of attention maps, making it compatible with FlashAttention and applicable to CNNs and SSMs. They evaluate token pruning with this metric on video-text retrieval (UMT), video QA, ImageNet classification with DeiT, ResNet, and ViM, reporting large throughput gains and competitive accuracy.

Significance. If the claims are supported, the contribution is practically valuable: a simple importance score that works without attention maps would enable training-free token pruning on FlashAttention-accelerated models and on non-transformer architectures. The paper includes a useful reliability check (Table 8) and provides code. However, the experimental protocol currently prevents a clean attribution of the reported gains to the proposed metric, so the significance is conditional on additional ablations.

major comments (4)
  1. [Section 3.3, Figure 5] The operation choice (MLP) and distance metric (L2) are selected by ablations on ImageNet/DeiT-S and MSRVTT/UMT-B, which are the same benchmarks used for the main results in Tables 2 and 5. Consequently, the reported improvements over attention-based scoring are partially selection artifacts and the 'model-agnostic' claim is not independently validated. Please either fix the configuration a priori, evaluate on held-out benchmarks, or report the selection procedure as model selection with appropriate caveats.
  2. [Tables 2, 4, 5] The throughput comparison between 'Attn' and 'Ours' conflates the importance metric with the attention kernel, since 'Attn' uses standard attention while 'Ours' uses FlashAttention. Table 1 shows that FlashAttention alone already provides 1.5x (DeiT-S) and 2.7x (UMT-B) throughput gains, so the additional speedups in the main tables cannot be attributed to representation shift. Please report an ablation with representation-shift pruning under standard attention (or attention-based pruning with materialized attention under FlashAttention) to compare the metrics at matched kernels and FLOPs.
  3. [Section 4.2, ResNet experiments] The authors state that after token compression in CNNs 'we finetune the model for 100 epochs, including 10 cooldown epochs' to adapt to the changed resolution. This contradicts the paper's central claim of a 'training-free' method (abstract and Section 1). Please clarify the scope of the training-free claim, or provide CNN results that do not require finetuning.
  4. [Table 2] On UMT-B, the proposed method is not consistently better than attention-based pruning; for MSVD the R@1 drops from 60.3 (Attn) to 57.7 (Ours) and for ActivityNet from 54.2 to 50.3. The text claims 'competitive or even better performance' but this is not supported on these benchmarks. Please discuss these failure cases and present per-dataset results honestly, or restrict the claim to the settings where the advantage holds.
minor comments (6)
  1. [Abstract] The phrase 'speedups of up to 5.5% and 4.4%' should read '5.5x and 4.4x' (the Introduction and Tables report multiplicative speedups).
  2. [Section 4.1] The harmonic mean of V2T and T2V results is mentioned but the individual V2T/T2V scores are not shown; please clarify how the reported R@1 values are aggregated.
  3. [Table 3 caption] The comparison is described as 'vid-TLDR +Ours', but it is unclear whether the baseline vid-TLDR runs with standard attention and what the '+Ours' exactly replaces; please specify the kernel and importance-score settings for both columns.
  4. [Section 4.2] The claimed 'negligible overhead' of computing representation shift is not quantified; please include a timing or FLOP breakdown of the scoring step.
  5. [Table 7] The ViM experiment covers only one model and dataset; the conclusion that representation shift generalizes to SSMs would benefit from additional models or tasks.
  6. [References, [50]] Reference [50] is cited for line-wise pruning in CNNs, but that work is for Transformers without retraining; the difference in protocol (finetuning) should be acknowledged.

Circularity Check

1 steps flagged · score 4.0 of 10

Design choices for representation shift are selected on the evaluation benchmarks, making the reported advantage of MLP+L2 partially self-confirming; core FlashAttention compatibility remains independent.

  1. fitted input called prediction [Section 3.3 (Figure 5) and Section 4 (Tables 2 and 5)]
    "In our experiments, we evaluate the impact of operation choice by pruning tokens based on the representation shift computed using the L2 norm for each candidate operation. ... Based on these findings, we adopt the representation shift at MLP as our primary measure for token importance."

    The operation choice (MLP vs Attn vs All) and distance metric (L2 vs L1 vs Cos) are selected by evaluating on ImageNet/DeiT-S and MSRVTT/UMT-B in Figure 5, exactly the same datasets and models used for the main results in Tables 2 and 5. The conclusion that the MLP+L2 configuration is the best variant is therefore not an independent prediction; its superiority on those benchmarks is, by construction, the outcome of the selection procedure rather than a fresh confirmation. The central FlashAttention-compatibility claim and the comparisons to attention-based pruning are not forced by this selection, so the circularity is partial.

full rationale

There is no mathematical derivation chain in the paper: representation shift is an empirical heuristic, not a first-principles result, so most alleged circularity does not apply. The metric s = ||F(x) - x|| is defined independently of attention maps, which genuinely makes it FlashAttention-compatible by construction, but that compatibility is a definitional property rather than a circular derivation. No load-bearing self-citation was found: the authors' prior vid-TLDR is used as a baseline and extension target, not as justification for the central claim. The throughput comparison between 'Attn' and 'Ours' confounds the importance metric with the FlashAttention kernel, but since FlashAttention is exact, the accuracy comparison at matched FLOPs remains attributable to the pruning decision. The CNN experiments fine-tune for 100 epochs, which weakens the 'training-free' description, but that is an overclaim rather than circularity. The one genuine circularity concern is that the operational choices of MLP and L2 were selected on the same benchmarks later used to demonstrate the method's effectiveness, making the variant-comparison claim partially self-confirming. This does not invalidate the core compatibility or generalization claims, so the overall circularity score is moderate.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the heuristic that representation shift correlates with token importance, plus a few design choices (MLP, L2, pruning layers and ratios) that are selected empirically on the evaluation benchmarks. No new physical or mathematical entities are introduced.

free parameters (4)
  • Pruning ratio = 20% for DeiT and video retrieval, 10% for video QA
    Chosen by hand per task; no analysis of sensitivity or optimality.
  • Pruned layers = First three layers for UMT; layers [1,4,7] for DeiT; first two stages for ResNet
    Selected based on the same benchmarks used for final evaluation, not on a held-out validation set.
  • Operation for representation shift = MLP for transformers, stage output for ResNet
    Selected via Figure 5a ablation on ImageNet and MSRVTT, the same datasets for the main results.
  • Distance metric = L2 norm
    Selected via Figure 5b ablation on the same benchmarks.
assumptions (4)
  • domain assumption Higher representation shift implies higher token importance
    Central hypothesis in Section 3.2; not derived or formally justified.
  • ad hoc to paper MLP representation shift is the best operation for importance scoring
    Based on a limited ablation in Figure 5a on two datasets, not on a theoretical argument.
  • ad hoc to paper L2 distance is the best distance metric for representation shift
    Based on Figure 5b, which evaluates only L1, L2, and cosine on the same two datasets.
  • domain assumption Token pruning without finetuning preserves accuracy for transformers and SSMs
    Assumed throughout; no analysis of error accumulation or distribution shift after pruning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Representation Shift: Unifying Token Compression with FlashAttention." pith.science (2026). https://pith.science/paper/JYUAVHJ7

@misc{pith2026250800367,
  author       = {Pith},
  title        = {Pith review of: Representation Shift: Unifying Token Compression with FlashAttention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JYUAVHJ7}},
  note         = {Machine review of arXiv:2508.00367}
}
read the original abstract

Transformers have demonstrated remarkable success across vision, language, and video. Yet, increasing task complexity has led to larger models and more tokens, raising the quadratic cost of self-attention and the overhead of GPU memory access. To reduce the computation cost of self-attention, prior work has proposed token compression techniques that drop redundant or less informative tokens. Meanwhile, fused attention kernels such as FlashAttention have been developed to alleviate memory overhead by avoiding attention map construction and its associated I/O to HBM. This, however, makes it incompatible with most training-free token compression methods, which rely on attention maps to determine token importance. Here, we propose Representation Shift, a training-free, model-agnostic metric that measures the degree of change in each token's representation. This seamlessly integrates token compression with FlashAttention, without attention maps or retraining. Our method further generalizes beyond Transformers to CNNs and state space models. Extensive experiments show that Representation Shift enables effective token compression compatible with FlashAttention, yielding significant speedups of up to 5.5% and 4.4% in video-text retrieval and video QA, respectively. Code is available at https://github.com/mlvlab/Representation-Shift.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 70 canonical work pages

  1. [1]

    Yolov12: A breakdown of the key architectural features

    Mujadded Al Rabbani Alif and Muhammad Hussain. Yolov12: A breakdown of the key architectural features. arXiv:2502.14740, 2025. 2

    arXiv 2025

  2. [2]

    Localizing mo- ments in video with natural language

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing mo- ments in video with natural language. In ICCV, pages 5803– 5812, 2017. 5, 6

    work page 2017

  3. [3]

    Long- former: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Long- former: The long-document transformer. arXiv:2004.05150,

    arXiv 2004

  4. [4]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. ICLR, 2023. 2, 3

    work page 2023

  5. [5]

    The pagerank citation ranking: bringing order to the web

    Sergey Brin. The pagerank citation ranking: bringing order to the web. ASIS, 1998. 3

    work page 1998

  6. [6]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR,

  7. [7]

    Efficientvit: Multi-scale linear attention for high-resolution dense prediction

    Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han. Efficientvit: Multi-scale linear attention for high-resolution dense prediction. ICCV, 2023. 2

    work page 2023

  8. [8]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In ECCV. Springer,

    work page

Show all 78 references
  1. [9]

    Scatterbrain: Unifying sparse and low- rank attention

    Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher R´e. Scatterbrain: Unifying sparse and low- rank attention. NeurIPS, 2021. 2

    2021

  2. [10]

    Collecting highly paral- lel data for paraphrase evaluation

    David Chen and William B Dolan. Collecting highly paral- lel data for paraphrase evaluation. InProceedings of the 49th annual meeting of the association for computational linguis- tics: human language technologies, 2011. 5, 6

    2011

  3. [11]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In CVPR,

  4. [12]

    Per- pixel classification is not all you need for semantic segmen- tation

    Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per- pixel classification is not all you need for semantic segmen- tation. NeurIPS, 2021. 1

    2021

  5. [13]

    vid-tldr: Training free token merging for light-weight video transformer

    Joonmyung Choi, Sanghyeok Lee, Jaewon Chu, Minhyuk Choi, and Hyunwoo J Kim. vid-tldr: Training free token merging for light-weight video transformer. In CVPR, 2024. 2, 3, 6, 7

    2024

  6. [14]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Do- han, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. ICLR, 2021. 2

    2021

  7. [15]

    Twins: Revisiting the design of spatial attention in vision transformers

    Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haib- ing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. NeurIPS, 2021. 1, 2

    2021

  8. [16]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christo- pher R ´e. Flashattention: Fast and memory-efficient exact attention with io-awareness. NeurIPS, 2022. 2, 3

    2022

  9. [17]

    Vision transformers need registers.ICLR, 2024

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers.ICLR, 2024. 8

    2024

  10. [18]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 3, 4, 7, 8

    2009

  11. [19]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. ICLR, 2021. 1, 2, 3

    2021

  12. [20]

    Levit: a vision transformer in convnet’s clothing for faster inference

    Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herv ´e J ´egou, and Matthijs Douze. Levit: a vision transformer in convnet’s clothing for faster inference. In ICCV, 2021. 2

    2021

  13. [21]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,

  14. [22]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In ICML, 2020. 2

    2020

  15. [23]

    Groupwise query special- ization and quality-aware multi-assignment for transformer- based visual relationship detection

    Jongha Kim, Jihwan Park, Jinyoung Park, Jinyoung Kim, Se- hyung Kim, and Hyunwoo J Kim. Groupwise query special- ization and quality-aware multi-assignment for transformer- based visual relationship detection. In CVPR, 2024. 1

    2024

  16. [24]

    Learned token pruning for transformers

    Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. In KDD, 2022. 2, 3

    2022

  17. [25]

    Re- former: The efficient transformer

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Re- former: The efficient transformer. ICLR, 2020. 1, 2

    2020

  18. [26]

    Video-text representation learning via differentiable weak temporal alignment

    Dohwan Ko, Joonmyung Choi, Juyeon Ko, Shinyeong Noh, Kyoung-Woon On, Eun-Sol Kim, and Hyunwoo J Kim. Video-text representation learning via differentiable weak temporal alignment. In CVPR, 2022. 1

    2022

  19. [27]

    Meltr: Meta loss transformer for learning to fine-tune video foun- dation models

    Dohwan Ko, Joonmyung Choi, Hyeong Kyu Choi, Kyoung- Woon On, Byungseok Roh, and Hyunwoo J Kim. Meltr: Meta loss transformer for learning to fine-tune video foun- dation models. In CVPR, 2023

    2023

  20. [28]

    Vidchain: Chain-of-tasks with metric- based direct preference optimization for dense video caption- ing

    Ji Soo Lee, Jongha Kim, Jeehye Na, Jinyoung Park, and Hyunwoo J Kim. Vidchain: Chain-of-tasks with metric- based direct preference optimization for dense video caption- ing. In AAAI, 2025. 1

    2025

  21. [29]

    Multi-criteria token fusion with one-step-ahead attention for efficient vision transformers

    Sanghyeok Lee, Joonmyung Choi, and Hyunwoo J Kim. Multi-criteria token fusion with one-step-ahead attention for efficient vision transformers. In CVPR, 2024. 2

    2024

  22. [30]

    Ef- ficientvim: Efficient vision mamba with hidden state mixer based state space duality

    Sanghyeok Lee, Joonmyung Choi, and Hyunwoo J Kim. Ef- ficientvim: Efficient vision mamba with hidden state mixer based state space duality. In CVPR, 2025. 2

    2025

  23. [31]

    Revealing single frame bias for video-and-language learning

    Jie Lei, Tamara L Berg, and Mohit Bansal. Revealing single frame bias for video-and-language learning. ACL, 2023. 5, 6

    2023

  24. [32]

    Unmasked teacher: Towards training-efficient video foundation models

    Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. In ICCV, 2023. 1, 2, 3, 4, 5, 6 9

    2023

  25. [33]

    Not all patches are what you need: Expediting vision transformers via token reorganiza- tions

    Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganiza- tions. ICLR, 2022. 2, 3, 4, 7

    2022

  26. [34]

    Efficientvit: Memory efficient vision transformer with cascaded group attention

    Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang, Han Hu, and Yixuan Yuan. Efficientvit: Memory efficient vision transformer with cascaded group attention. In CVPR,

  27. [35]

    Efficient training of visual trans- formers with small datasets

    Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, and Marco Nadai. Efficient training of visual trans- formers with small datasets. NeurIPS, 2021. 2

    2021

  28. [36]

    Vmamba: Visual state space model

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. NeurIPS, 2024. 2

    2024

  29. [37]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 1, 2

    2021

  30. [38]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022. 2

    2022

  31. [39]

    Beyond attentive tokens: Incorporating to- ken importance and diversity for efficient vision transform- ers

    Sifan Long, Zhen Zhao, Jimin Pi, Shengsheng Wang, and Jingdong Wang. Beyond attentive tokens: Incorporating to- ken importance and diversity for efficient vision transform- ers. In CVPR, 2023. 2, 3, 4, 7

    2023

  32. [40]

    Mobilevit: light- weight, general-purpose, and mobile-friendly vision trans- former

    Sachin Mehta and Mohammad Rastegari. Mobilevit: light- weight, general-purpose, and mobile-friendly vision trans- former. ICLR, 2022. 2

    2022

  33. [41]

    Adavit: Adaptive vision transformers for efficient image recognition

    Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In CVPR, 2022. 2, 3

    2022

  34. [42]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2023. 2

    2023

  35. [43]

    IA-RED 2: Interpretability-aware redundancy reduction for vision trans- formers

    Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva. IA-RED 2: Interpretability-aware redundancy reduction for vision trans- formers. NeurIPS, 2021. 2, 3

    2021

  36. [44]

    Deepvideo-r1: Video reinforcement fine-tuning via difficulty-aware regressive grpo

    Jinyoung Park, Jeehye Na, Jinyoung Kim, and Hyunwoo J Kim. Deepvideo-r1: Video reinforcement fine-tuning via difficulty-aware regressive grpo. arXiv preprint, 2025. 1

    2025

  37. [45]

    cosformer: Rethinking softmax in attention

    Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. ICLR,

  38. [46]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. NeurIPS,

  39. [47]

    Movie description

    Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie description. IJCV, 2017. 5, 6

    2017

  40. [48]

    Efficient content-based sparse attention with rout- ing transformers

    Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with rout- ing transformers. TACL, 2021. 1, 2

    2021

  41. [49]

    Segmenter: Transformer for semantic segmenta- tion

    Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmenta- tion. In ICCV, 2021. 1

    2021

  42. [50]

    Removing rows and columns of tokens in vision transformer enables faster dense prediction without retraining

    Diwei Su, Cheng Fei, and Jianxu Luo. Removing rows and columns of tokens in vision transformer enables faster dense prediction without retraining. In ECCV, 2024. 7

    2024

  43. [51]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. NeurIPS, 2022. 1

    2022

  44. [52]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. In ICML, 2021. 1, 2, 3, 4, 7, 8

    2021

  45. [53]

    Going deeper with im- age transformers

    Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Herv´e J´egou. Going deeper with im- age transformers. In ICCV, 2021. 1, 3

    2021

  46. [54]

    Maxvit: Multi-axis vision transformer

    Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In ECCV, 2022. 1, 2

    2022

  47. [55]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017. 1, 2

    2017

  48. [56]

    Zero- tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers

    Hongjie Wang, Bhishma Dedhia, and Niraj K Jha. Zero- tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers. In CVPR, 2024. 2, 3

    2024

  49. [57]

    Linformer: Self-attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv:2006.04768, 2020. 1, 2

    2006 arXiv

  50. [58]

    Pyra- mid vision transformer: A versatile backbone for dense pre- diction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyra- mid vision transformer: A versatile backbone for dense pre- diction without convolutions. In ICCV, 2021. 1, 2

    2021

  51. [59]

    Pvt v2: Improved baselines with pyramid vision transformer

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational visual media, 2022. 1, 2

    2022

  52. [60]

    Videocomposer: Compositional video synthesis with motion controllability

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Ji- uniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jin- gren Zhou. Videocomposer: Compositional video synthesis with motion controllability. NeurIPS, 2023. 2

    2023

  53. [61]

    End-to-end video instance segmentation with transformers

    Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In CVPR,

  54. [62]

    Internvideo: General video founda- tion models via generative and discriminative learning

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video founda- tion models via generative and discriminative learning. arXiv:2212.03191, 2022. 1

    2022 arXiv

  55. [63]

    Anchor detr: Query design for transformer-based object de- tection

    Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer-based object de- tection. AAAI, 2022. 1

    2022

  56. [64]

    Internvideo2: Scaling foundation models for mul- timodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for mul- timodal video understanding. In ECCV, 2024. 1, 2 10

    2024

  57. [65]

    Con- vnext v2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Con- vnext v2: Co-designing and scaling convnets with masked autoencoders. In CVPR, 2023. 2

    2023

  58. [66]

    Nystr¨omformer: A nystr¨om-based algorithm for approximat- ing self-attention

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystr¨omformer: A nystr¨om-based algorithm for approximat- ing self-attention. In AAAI, 2021. 1, 2

    2021

  59. [67]

    Video question answer- ing via gradually refined attention over appearance and mo- tion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answer- ing via gradually refined attention over appearance and mo- tion. In ACMMM, 2017. 5, 6

    2017

  60. [68]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016. 3, 4, 5, 6

    2016

  61. [69]

    A-vit: Adaptive tokens for efficient vision transformer

    Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In CVPR, 2022. 2, 3

    2022

  62. [70]

    Tokens-to-token vit: Training vision transformers from scratch on imagenet

    Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, 2021. 2

    2021

  63. [71]

    Efficient trans- former adaptation with soft token merging

    Xin Yuan, Hongliang Fei, and Jinoo Baek. Efficient trans- former adaptation with soft token merging. In CVPR, 2024. 2, 3

    2024

  64. [72]

    Shvit: Single-head vision transformer with memory efficient macro design

    Seokju Yun and Youngmin Ro. Shvit: Single-head vision transformer with memory efficient macro design. In CVPR,

  65. [73]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. NeurIPS, 2020. 1, 2

    2020

  66. [74]

    Exploring token pruning in vision state space models

    Zheng Zhan, Zhenglun Kong, Yifan Gong, Yushu Wu, Zi- chong Meng, Hangyu Zheng, Xuan Shen, Stratis Ioannidis, Wei Niu, Pu Zhao, et al. Exploring token pruning in vision state space models. NeurIPS, 2024. 7

    2024

  67. [75]

    Dino: Detr with improved denoising anchor boxes for end-to-end object detection

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. ICLR, 2023. 1

    2023

  68. [76]

    Rethinking semantic segmen- tation from a sequence-to-sequence perspective with trans- formers

    Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmen- tation from a sequence-to-sequence perspective with trans- formers. In CVPR, 2021. 1

    2021

  69. [77]

    Vision mamba: Efficient visual representation learning with bidirectional state space model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. ICML, 2024. 2, 7

    2024

  70. [78]

    Deformable detr: Deformable transformers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. ICLR, 2021. 1 11

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.