Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FFN blocks, not attention, dominate Vision Transformer latency, and a channel idle mechanism that leaves most FFN channels linear lets those blocks collapse into smaller ones at inference—yielding ~68% speed-ups on large ViTs with…

desk verdict RePaViT's channel idle trick is a real, scalable speed-up, but the advertised accuracy gains on large ViTs are confounded with a training-recipe change, so the paper needs a same-recipe vanilla baseline before the 'sometimes gains' claim stands. read the letter →

arxiv 2505.21847 v2 pith:WFDFF72F submitted 2025-05-28 cs.CV cs.AI

Xuwei Xu , Yang Li , Yudong Chen , Jiajun Liu , Sen Wang This is my paper
classification cs.CVcs.AI
keywords visiontransformerstructuralreparameterizationfeedforwardnetworkaccelerationchannelidlemechanismBatchNorminferencespeedupmodelscalingimageclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RePaViT argues that the feedforward network (FFN) blocks, not the attention blocks, are the real latency bottleneck of Vision Transformers, and that their share of runtime grows as models get larger. The proposal is a channel idle mechanism: during training, only a small fraction of an FFN's hidden channels pass through the activation function, while the rest stay linear. Because those idle channels form a linear pathway, the two large projection matrices, the two BatchNorms, and the residual shortcut can be algebraically merged after training into three much smaller linear projections at test time. On ViT-Large and ViT-Huge this yields 66.8% and 68.7% inference speed-ups with 1.7 and 1.1 points higher top-1 accuracy under the paper's training recipe; on small models accuracy drops, a limitation the authors acknowledge. If correct, this gives general-hardware acceleration for large visual foundation models without sparse kernels.

What carries the argument

The load-bearing object is the channel idle mechanism. In each FFN layer, the activation function is applied only to $\mu C$ of the $\rho C$ hidden channels, while the remaining $(\rho-\mu)C$ channels bypass it and pass through linearly. With LayerNorm replaced by BatchNorm (and an extra BatchNorm inserted before the second projection), every normalization on that linear route can be folded into adjacent weights, and the identity $\widetilde{W}=\widetilde{W}_{\text{In}}[:,\mu C+1:\rho C]\widetilde{W}_{\text{Out}}[\mu C+1:\rho C,:]+I$ merges the second projection with the residual shortcut into one $C\times C$ matrix. At test time the FFN shrinks to three dense layers ($C\times\mu C$, $\mu C\times C$, $C\times C$), cutting parameters and FLOPs by the factor $1-\theta+1/(2\rho)$; because every operation on the idle path is linear, the reparameterized output is exactly the original output by construction.

What would settle it

Train a vanilla ViT-Large (LayerNorm kept, no idle channels) with exactly the RePaViT recipe—Lamb optimizer, batch size 4096, 300 epochs, same augmentations—and measure its ImageNet-1k top-1 accuracy; if it reaches or exceeds 82.0%, the claimed accuracy benefit of the idle mechanism is refuted, even though the 66.8% speed-up claim would survive.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that FFN layers dominate ViT inference latency—53.8% of runtime in ViT-Large—and that this share grows with model size because FFN cost scales as $O(2\rho N C^2)$ while token counts stay roughly constant. It then shows that a channel idle mechanism, which sends only $\mu C$ of the $\rho C$ FFN hidden channels through the activation and leaves $(\rho-\mu)C$ channels idle, creates a linear pathway. Combined with BatchNorm substitution for LayerNorm, this pathway makes the FFN reparameterizable after training: the two dense projections, the normalizations, and the shortcut merge into three smaller linear weights $\widetilde{W}_{\text{In}}[:,1:\mu C]$, $\widetilde{W}_{\text{Out}}[1:\mu C,:]$, and $\widetilde{W}=\widetilde{W}_{\text{In}}[:,\mu C+1:\rho C]\widetilde{W}_{\text{Out}}[\mu C+1:\rho C,:]+I$. The parameter count falls from $2\rho C^2$ to $(2\mu+1)C^2$, which at $\rho=4$ and idle ratio $\theta=0.75$ is 37.5% of the original. The resulting RePaViT family scales favorably: larger models show larger speed-ups and smaller accuracy gaps, with RePa-ViT-Large and RePa-ViT-Huge reporting 66.8% and 68.7% faster inference and +1.7% and +1.1% top-1 accuracy. The paper presents this as the first structural reparameterization applied directly to FFN layers rather than to CNN components attached to ViTs.

Load-bearing premise

The comparison that supports the reported accuracy gains is not controlled—the vanilla ViT-Large and ViT-Huge baselines were not trained with the paper's own recipe (BatchNorm, Lamb, batch size 4096, 300 epochs)—so the claimed +1.7% and +1.1% gains could come from the stronger training recipe rather than from the channel idle mechanism itself.

Editorial extensions

If this is right

  • Large visual foundation models built on ViTs can be accelerated by roughly 25–69% on ordinary hardware with little or no accuracy loss; the paper reports 24.7% speed-up with +0.8% zero-shot accuracy on a CLIP ViT-B/16.
  • The speed-up scales with model size, from about 16.5% on DeiT-Tiny to 66.8% on ViT-Large, while the accuracy gap narrows from −2.7% to +1.7%, making the method increasingly effective as models grow.
  • Because the reparameterization is exact linear algebra, pre- and post-reparameterization accuracies are identical, so the inference-time acceleration is obtained for free once training is complete.
  • Dense prediction tasks inherit the gains: RePa-Swin-Base cuts RetinaNet latency by 18.7% at slightly higher AP, Mask R-CNN latency by 16.0%, and UperNet segmentation latency by 15.4% with a 1.2-point mIoU loss.
  • The idle ratio $\theta$ acts as a tunable accuracy–speed knob: $\theta=0.25$ beats the vanilla backbone on several models, while $\theta=1.0$ (no nonlinearity at all) collapses accuracy, showing that a minimum amount of activation is required.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because the paper does not retrain the vanilla baselines with its own recipe (BatchNorm, Lamb, batch size 4096, 300 epochs), the reported accuracy gains on ViT-Large and ViT-Huge could partly be a training-recipe effect; training a vanilla ViT-Large with the identical recipe would isolate the contribution of the idle mechanism.
  • Inference: the linear-pathway idea generalizes beyond FFNs—any block with a residual shortcut and an activation applied only to a channel subset (for example, mixture-of-experts routers or convolutional blocks with channel-wise activations) could be reparameterized the same way.
  • Inference: the paper fixes a single global idle ratio, but the accuracy cost on small models suggests an adaptive per-layer $\theta$ that keeps more nonlinearity in early layers might recover much of the small-model accuracy loss while retaining most of the speed-up.
  • Inference: the most decisive scalability test would be applying RePaViT to a much larger contrastive or generative vision backbone; if the +1–2% accuracy gains persist there, the idle mechanism is doing real regularization work rather than acting as pure compression.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes RePaViT, a method for accelerating Vision Transformers by structurally reparameterizing the FFN layers after training. The key idea is a 'channel idle mechanism' in which a fraction (default 75%) of the FFN's hidden channels bypass the activation function during training, creating a linear pathway that can be merged with adjacent BatchNorm layers and the shortcut into three smaller linear projections at inference. The paper reports latency profiling showing FFN layers dominate ViT latency, especially for large models, and presents experiments on DeiT, Swin, LV-ViT, and ViT-Large/Huge, claiming speed-ups of 16.5–68.7% with accuracy changes ranging from -2.8 to +1.7 points, including +1.7% and +1.1% on ViT-Large and ViT-Huge. It also includes comparisons to network pruning and the SLAB reparameterization method, plus dense prediction, DINO, and CLIP experiments.

Significance. The reparameterization algebra in Section 3.3 is exact and correctly applied to frozen BatchNorm statistics; the reported parameter and FLOP reductions are consistent with the measured throughput gains. The paper ships source code and the speed measurements are on a fixed hardware setup, which is a strength. If the accuracy-gain claim for large models were supported by a controlled comparison, this would be a significant result, as it is a rare example of post-training structural reparameterization applied directly to FFN layers of standard ViTs. However, the current evidence for accuracy gains is confounded by training-recipe differences, and the matched-recipe experiments in the appendix do not corroborate the gains.

major comments (3)
  1. [§4.2, Table 1] The accuracy gains for ViT-Large and ViT-Huge are confounded. The 'vanilla' rows for ViT-Large (80.3%) and ViT-Huge (80.3%) are taken from published results (Dosovitskiy et al., 2021), which were trained with a different recipe (e.g., Adam, smaller batch size, LayerNorm), whereas RePaViT rows use BatchNorm replacing LayerNorm, an extra BatchNorm before the second projection, Lamb optimizer, batch size 4096, and drop path 0.30 (Table 7). The abstract's phrase 'under the same training strategy' is therefore misleading. Without a same-recipe vanilla control, the +1.7% and +1.1% accuracy gains cannot be attributed to the channel idle mechanism.
  2. [§4.8, Tables 8 and 9] The only controlled comparisons with matched training recipes (DINO and CLIP) show accuracy drops or mixed results: DINO ViT-Small -2.6%, ViT-Base -1.2%; CLIP B/32 at θ=0.50 -0.3% and at θ=0.75 -3.9%; CLIP B/16 at θ=0.50 +0.8% and at θ=0.75 -1.7%. These results do not support a systematic accuracy gain, and the single +0.8% is within the noise of zero-shot evaluation. The claim of 'higher top-1 accuracies on larger models' should be either supported by same-recipe vanilla baselines on ViT-Large/Huge or removed/qualified.
  3. [Appendix C] The limitations section acknowledges the accuracy drop on small models but does not mention that the large-model accuracy gains are not verified with matched training recipes. Since the paper explicitly invites community validation, this missing control should be addressed, e.g., by training a vanilla ViT-Large with the same BatchNorm/Lamb/4096-batch/300-epoch recipe as a baseline and comparing against RePaViT trained identically.
minor comments (6)
  1. [§3.3, Eq. (6)] The symbol Y is used for the output of Eq. (2) and then reused for the input to the reparameterized FFN in Eq. (6), which is confusing. Please rename the input in Eq. (6) to X or add an explicit sentence clarifying the transition.
  2. [§3.4] The statement that 'the inference speed gain is more than the computational complexity reduction' is plausible due to removal of normalizations and shortcuts, but no end-to-end latency breakdown before and after reparameterization is provided to quantify this effect. Adding such a breakdown would strengthen the claim.
  3. [§4.2, Table 1] The pre-reparameterization rows (marked ×) are slower than the vanilla backbones; a one-sentence explanation that this is due to the added BatchNorm layers and/or the non-fused training-time structure would help readers interpret the table.
  4. [§4.5] There is a typo in the section title: 'Sensitivty' should be 'Sensitivity'.
  5. [Appendix C] The text refers to 'RePaFormers' but the model family is called RePaViT; please correct this inconsistency.
  6. [Figure 3] The absolute latencies are reported in milliseconds, but the measurement conditions (batch size, GPU, precision) are not specified in the caption. Please add these details for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: RePaViT's reparameterization is exact algebra on frozen BatchNorm statistics, and its efficiency and accuracy claims are measured against external benchmarks rather than derived from their own inputs.

full rationale

Walking the paper's derivation chain, the channel-idle mechanism is defined constructively in Eq. 2, and the reparameterization in Eqs. 3-6 is a direct algebraic consequence of folding frozen BatchNorm statistics into adjacent linear projections and merging the idle linear pathway with the shortcut. This is a construction, not a result that assumes its conclusion: the post-reparameterization model is derived from the training-time model, and the claimed equality of pre- and post-reparameterization accuracy is a property of exact linear algebra, not a fitted input. The speed-up figures are measured on hardware against vanilla backbones and external pruning baselines, so they do not reduce to the method's definitions. The channel idle ratio θ is tuned via sensitivity experiments (Table 4), but hyperparameter selection is not an instance of fitting a parameter and then calling the output a prediction. The paper cites Yao et al. (2021) for BatchNorm substitution and Ding et al. (2021b) for structural reparameterization, but these are external prior works, and the authors' own prior publications (Xu et al., 2023; Xu et al., 2024b) appear only as related work and are not load-bearing. Appendix C explicitly acknowledges accuracy degradation on tiny models, which is a limitation but not a circular step. The principal caveat is that the headline +1.7% and +1.1% accuracy gains on ViT-Large/Huge compare against vanilla baselines that were not retrained with the RePaViT recipe (BatchNorm, Lamb, batch size 4096, 300 epochs), so the accuracy-gain claim is confounded by the training recipe; however, this is a validity and attribution concern, not a circularity concern. No equation or fitted quantity in the paper reduces to its own input, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central method rests on standard BN folding plus the empirical premise that 75% channel idling is harmless. The accuracy attribution is an unproven assumption.

free parameters (2)
  • channel idle ratio theta = 0.75 default; 0.50 for DeiT-Tiny/Small and CLIP-B/32
    Chosen per model based on sensitivity experiments in Table 4; not fitted to data, but a design choice tuned on validation accuracy.
  • activated channel subset size mu = 1 (first C channels)
    Set to 1 by default; the position of activated channels is fixed to the first block rather than learned or selected.
assumptions (4)
  • standard math BatchNorm can be folded into preceding linear layer weights at inference using frozen statistics.
    Used in Eq. 3; standard inference-time normalization folding.
  • domain assumption The reparameterized model and the training model produce identical outputs for all inputs.
    Requires that the activation is applied elementwise to a fixed channel subset and BN statistics are frozen; stated in Section 3.3.
  • ad hoc to paper Keeping 75% of channels idle preserves representational capacity sufficiently for large models.
    This is the core empirical premise; validated only on the tested models, not theoretically guaranteed.
  • ad hoc to paper The increase in accuracy on large models is attributable to the channel idle mechanism rather than training recipe changes.
    The paper does not retrain vanilla baselines with the same recipe; this is the weakest experimental premise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers." pith.science (2026). https://pith.science/paper/WFDFF72F

@misc{pith2026250521847,
  author       = {Pith},
  title        = {Pith review of: RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WFDFF72F}},
  note         = {Machine review of arXiv:2505.21847}
}
read the original abstract

We reveal that feedforward network (FFN) layers, rather than attention layers, are the primary contributors to Vision Transformer (ViT) inference latency, with their impact signifying as model size increases. This finding highlights a critical opportunity for optimizing the efficiency of large-scale ViTs by focusing on FFN layers. In this work, we propose a novel channel idle mechanism that facilitates post-training structural reparameterization for efficient FFN layers during testing. Specifically, a set of feature channels remains idle and bypasses the nonlinear activation function in each FFN layer, thereby forming a linear pathway that enables structural reparameterization during inference. This mechanism results in a family of ReParameterizable Vision Transformers (RePaViTs), which achieve remarkable latency reductions with acceptable sacrifices (sometimes gains) in accuracy across various ViTs. The benefits of our method scale consistently with model sizes, demonstrating greater speed improvements and progressively narrowing accuracy gaps or even higher accuracies on larger models. In particular, RePa-ViT-Large and RePa-ViT-Huge enjoy 66.8% and 68.7% speed-ups with +1.7% and +1.1% higher top-1 accuracies under the same training strategy, respectively. RePaViT is the first to employ structural reparameterization on FFN layers to expedite ViTs to our best knowledge, and we believe that it represents an auspicious direction for efficient ViTs. Source code is available at https://github.com/Ackesnal/RePaViT.

Figures

Figures reproduced from arXiv: 2505.21847 by the authors.

Figure 1
Figure 1. RePaViT architecture. (a) represents the vanilla ViT block. (b) illustrates our channel idle mechanism for FFN layers during training, where only a subset of channels are activated while the rest bridge a linear pathway. (c) shows the reparameterized RePaViT block during testing, where the number of parameters and computational complexity are significantly reduced. 1. Introduction Vision Transformer (ViT) (Dosovitsk… view at source ↗
Figure 2
Figure 2. Performance comparison of RePaViTs and their vanilla backbones. RePaViTs (red circled) consistently achieve greater accelerations and smaller accuracy gaps when model sizes increase, showing the potential effectiveness in expediting large-scale ViTs. It is also worth noting that RePa-ViT-Large not only improves inference speed by more than 50% but also raises accuracy by 1.7%. token merging (Bolya et al., 2023; Zong… view at source ↗
Figure 3
Figure 3. Latency analysis. Visualization of the runtime latencies of patch embedding, MHSA and FFN layers. Notably, as the model size increases, the proportion of latency attributed to FFN layers also rises. Our method effectively reduces the latency of FFN layers and obtains increasingly better performance on larger models, demonstrating a scalable acceleration of FFN layers. 3.2. Channel Idle Mechanism for FFN Layers As [… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DeVIT: Low-Power Vision Transformer Acceleration Using Delta Computation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    DeVIT sorts quantized vision transformer weights into a differential chain and reuses the input-by-smallest-weight product, converting most weight multiplications into shift-add operations.

Reference graph

Works this paper leans on

75 extracted references · 60 canonical work pages · cited by 1 Pith paper

  1. [1]

    Token merging: Your vit but faster

    Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., and Hoffman, J. Token merging: Your vit but faster. In ICLR, 2023

    work page 2023

  2. [2]

    B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In NeurIPS, 2020

    work page 2020

  3. [3]

    Efficientvit: Multi-scale linear attention for high-resolution dense prediction

    Cai, H., Li, J., Hu, M., Gan, C., and Han, S. Efficientvit: Multi-scale linear attention for high-resolution dense prediction. In ICCV, 2023

    work page 2023

  4. [4]

    Emerging properties in self-supervised vision transformers

    Caron, M., Touvron, H., Misra, I., J \'e gou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In ICCV, 2021

    2021

  5. [5]

    C., and Lin, D

    Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., Zhang, Z., Cheng, D., Zhu, C., Cheng, T., Zhao, Q., Li, B., Lu, X., Zhu, R., Wu, Y., Dai, J., Wang, J., Shi, J., Ouyang, W., Loy, C. C., and Lin, D. MMDetection : Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019

    arXiv 1906

  6. [6]

    Mobile-former: Bridging mobilenet and transformer

    Chen, Y., Dai, X., Chen, D., Liu, M., Dong, X., Yuan, L., and Liu, Z. Mobile-former: Bridging mobilenet and transformer. In CVPR, 2022 a

    work page 2022

  7. [7]

    Improved feature distillation via projector ensemble

    Chen, Y., Wang, S., Liu, J., Xu, X., de Hoog, F., and Huang, Z. Improved feature distillation via projector ensemble. In NeurIPS, 2022 b

    work page 2022

  8. [8]

    Reproducible scaling laws for contrastive language-image learning

    Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023

    work page 2023

Show all 75 references
  1. [9]

    MMSegmentation : Openmmlab semantic segmentation toolbox and benchmark

    Contributors, M. MMSegmentation : Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation, 2020

    2020

  2. [10]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Dao, T., Fu, D., Ermon, S., Rudra, A., and R \'e , C. Flashattention: Fast and memory-efficient exact attention with io-awareness. In NeurIPS, 2022

    2022

  3. [11]

    P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al

    Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al. Scaling vision transformers to 22 billion parameters. In ICML, 2023

    2023

  4. [12]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, 2009

    2009

  5. [13]

    Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks

    Ding, X., Guo, Y., Ding, G., and Han, J. Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks. In ICCV, 2019

    2019

  6. [14]

    Diverse branch block: Building a convolution as an inception-like unit

    Ding, X., Zhang, X., Han, J., and Ding, G. Diverse branch block: Building a convolution as an inception-like unit. In CVPR, 2021 a

    2021

  7. [15]

    Repvgg: Making vgg-style convnets great again

    Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., and Sun, J. Repvgg: Making vgg-style convnets great again. In CVPR, 2021 b

    2021

  8. [16]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021

    2021

  9. [17]

    A., Jafari, F

    Fayyaz, M., Koohpayegani, S. A., Jafari, F. R., Sengupta, S., Joze, H. R. V., Sommerlade, E., Pirsiavash, H., and Gall, J. Adaptive token sampling for efficient vision transformers. In ECCV, 2022

    2022

  10. [18]

    Levit: a vision transformer in convnet's clothing for faster inference

    Graham, B., El-Nouby, A., Touvron, H., Stock, P., Joulin, A., J \'e gou, H., and Douze, M. Levit: a vision transformer in convnet's clothing for faster inference. In ICCV, 2021

    2021

  11. [19]

    Slab: Efficient transformers with simplified linear attention and progressive re-parameterized batch normalization

    Guo, J., Chen, X., Tang, Y., and Wang, Y. Slab: Efficient transformers with simplified linear attention and progressive re-parameterized batch normalization. In ICML, 2024

    2024

  12. [20]

    M., and Salzmann, M

    Guo, S., Alvarez, J. M., and Salzmann, M. Expandnets: Linear over-parameterization to train compact convolutional networks. In NeurIPS, 2020

    2020

  13. [21]

    Learning efficient vision transformers via fine-grained manifold distillation

    Hao, Z., Guo, J., Jia, D., Han, K., Tang, Y., Zhang, C., Hu, H., and Wang, Y. Learning efficient vision transformers via fine-grained manifold distillation. In NeurIPS, 2022

    2022

  14. [22]

    Mask r-cnn

    He, K., Gkioxari, G., Doll \'a r, P., and Girshick, R. Mask r-cnn. In ICCV, 2017

    2017

  15. [23]

    and Zhou, J

    He, Y. and Zhou, J. T. Data-independent module-aware pruning for hierarchical vision transformers. In ICLR, 2024

    2024

  16. [24]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  17. [25]

    and Szegedy, C

    Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015

    2015

  18. [26]

    All tokens matter: Token labeling for training better vision transformers

    Jiang, Z.-H., Hou, Q., Yuan, L., Zhou, D., Shi, Y., Jin, X., Wang, A., and Feng, J. All tokens matter: Token labeling for training better vision transformers. In NeurIPS, 2021

    2021

  19. [27]

    Token fusion: Bridging the gap between token pruning and token merging

    Kim, M., Gao, S., Hsu, Y.-C., Shen, Y., and Jin, H. Token fusion: Bridging the gap between token pruning and token merging. In WACV, 2024

    2024

  20. [28]

    C., Lo, W.-Y., et al

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In ICCV, 2023

    2023

  21. [29]

    Spvit: Enabling faster vision transformers via latency-aware soft token pruning

    Kong, Z., Dong, P., Ma, X., Meng, X., Niu, W., Sun, M., Shen, X., Yuan, G., Ren, B., Tang, H., et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In ECCV, 2022 a

    2022

  22. [30]

    Peeling the onion: Hierarchical reduction of data redundancy for efficient vision transformer training

    Kong, Z., Ma, H., Yuan, G., Sun, M., Xie, Y., Dong, P., Meng, X., Shen, X., Tang, H., Qin, M., et al. Peeling the onion: Hierarchical reduction of data redundancy for efficient vision transformer training. In AAAI, 2022 b

    2022

  23. [31]

    R., and Hinton, G

    Lei Ba, J., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    2016 arXiv

  24. [32]

    Efficientformer: Vision transformers at mobilenet speed

    Li, Y., Yuan, G., Wen, Y., Hu, J., Evangelidis, G., Tulyakov, S., Wang, Y., and Ren, J. Efficientformer: Vision transformers at mobilenet speed. In NeurIPS, 2022

    2022

  25. [33]

    Evit: Expediting vision transformers via token reorganizations

    Liang, Y., Chongjian, G., Tong, Z., Song, Y., Wang, J., and Xie, P. Evit: Expediting vision transformers via token reorganizations. In ICLR, 2021

    2021

  26. [34]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV, 2014

    2014

  27. [35]

    Focal loss for dense object detection

    Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Doll \'a r, P. Focal loss for dense object detection. In ICCV, 2017

    2017

  28. [36]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021

    2021

  29. [37]

    Swin transformer v2: Scaling up capacity and resolution

    Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., et al. Swin transformer v2: Scaling up capacity and resolution. In CVPR, 2022

    2022

  30. [38]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017

    2017

  31. [39]

    Shufflenet v2: Practical guidelines for efficient cnn architecture design

    Ma, N., Zhang, X., Zheng, H.-T., and Sun, J. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018

    2018

  32. [40]

    W., Anwer, R

    Maaz, M., Shaker, A., Cholakkal, H., Khan, S., Zamir, S. W., Anwer, R. M., and Shahbaz Khan, F. Edgenext: efficiently amalgamated cnn-transformer architecture for mobile vision applications. In ECCV, 2022

    2022

  33. [41]

    R., Ranjan, A., Prabhu, A., Rastegari, M., and Tuzel, O

    Marin, D., Chang, J.-H. R., Ranjan, A., Prabhu, A., Rastegari, M., and Tuzel, O. Token pooling in vision transformers for image classification. In WACV, 2023

    2023

  34. [42]

    and Rastegari, M

    Mehta, S. and Rastegari, M. Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. In ICLR, 2022 a

    2022

  35. [43]

    and Rastegari, M

    Mehta, S. and Rastegari, M. Separable self-attention for mobile vision transformers. arXiv preprint arXiv:2206.02680, 2022 b

    2022 arXiv

  36. [44]

    Adavit: Adaptive vision transformers for efficient image recognition

    Meng, L., Li, H., Chen, B.-C., Lan, S., Wu, Z., Jiang, Y.-G., and Lim, S.-N. Adavit: Adaptive vision transformers for efficient image recognition. In CVPR, 2022

    2022

  37. [45]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 2019

    2019

  38. [46]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  39. [47]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., and Hsieh, C.-J. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In NeurIPS, 2021

    2021

  40. [48]

    Tokenlearner: Adaptive space-time tokenization for videos

    Ryoo, M., Piergiovanni, A., Arnab, A., Dehghani, M., and Angelova, A. Tokenlearner: Adaptive space-time tokenization for videos. In NeurIPS, 2021

    2021

  41. [49]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. In NeurIPS Data Centric AI Workshop, 2021

    2021

  42. [50]

    Shaker, A., Maaz, M., Rasheed, H., Khan, S., Yang, M.-H., and Khan, F. S. Swiftformer: Efficient additive attention for transformer-based real-time mobile vision applications. In ICCV, 2023

    2023

  43. [51]

    Boosting vanilla lightweight vision transformers via re-parameterization

    Tan, Z., Li, X., Wu, Y., Chu, Q., Lu, L., Yu, N., and Ye, J. Boosting vanilla lightweight vision transformers via re-parameterization. In ICLR, 2024

    2024

  44. [52]

    Patch slimming for efficient vision transformers

    Tang, Y., Han, K., Wang, Y., Xu, C., Guo, J., Xu, C., and Tao, D. Patch slimming for efficient vision transformers. In CVPR, 2022

    2022

  45. [53]

    O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al

    Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al. Mlp-mixer: An all-mlp architecture for vision. In NeurIPS, 2021

    2021

  46. [54]

    Training data-efficient image transformers & distillation through attention

    Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J \'e gou, H. Training data-efficient image transformers & distillation through attention. In ICML, 2021

    2021

  47. [55]

    Vasu, P. K. A., Gabriel, J., Zhu, J., Tuzel, O., and Ranjan, A. Fastvit: A fast hybrid vision transformer using structural reparameterization. In ICCV, 2023 a

    2023

  48. [56]

    Vasu, P. K. A., Gabriel, J., Zhu, J., Tuzel, O., and Ranjan, A. Mobileone: An improved one millisecond mobile backbone. In CVPR, 2023 b

    2023

  49. [57]

    Vaswani, A. et al. Attention is all you need. In NeurIPS, 2017

    2017

  50. [58]

    Repvit: Revisiting mobile cnn from vit perspective

    Wang, A., Chen, H., Lin, Z., Han, J., and Ding, G. Repvit: Revisiting mobile cnn from vit perspective. In CVPR, 2024

    2024

  51. [59]

    Tinyvit: Fast pretraining distillation for small vision transformers

    Wu, K., Zhang, J., Peng, H., Liu, M., Xiao, B., Fu, J., and Yuan, L. Tinyvit: Fast pretraining distillation for small vision transformers. In ECCV, 2022

    2022

  52. [60]

    Unified perceptual parsing for scene understanding

    Xiao, T., Liu, Y., Zhou, B., Jiang, Y., and Sun, J. Unified perceptual parsing for scene understanding. In ECCV, 2018

    2018

  53. [61]

    Lpvit: Low-power semi-structured pruning for vision transformers

    Xu, K., Wang, Z., Chen, C., Geng, X., Lin, J., Yang, X., Wu, M., Li, X., and Lin, W. Lpvit: Low-power semi-structured pruning for vision transformers. In ECCV, 2024 a

    2024

  54. [62]

    No token left behind: Efficient vision transformer via dynamic token idling

    Xu, X., Li, C., Chen, Y., Chang, X., Liu, J., and Wang, S. No token left behind: Efficient vision transformer via dynamic token idling. In AJCAI, 2023

    2023

  55. [63]

    Gtp-vit: Efficient vision transformers via graph-based token propagation

    Xu, X., Wang, S., Chen, Y., Zheng, Y., Wei, Z., and Liu, J. Gtp-vit: Efficient vision transformers via graph-based token propagation. In WACV, 2024 b

    2024

  56. [64]

    Evo-vit: Slow-fast token evolution for dynamic vision transformer

    Xu, Y., Zhang, Z., Zhang, M., Sheng, K., Li, K., Dong, W., Zhang, L., Xu, C., and Sun, X. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In AAAI, 2022

    2022

  57. [65]

    Leveraging batch normalization for vision transformers

    Yao, Z., Cao, Y., Lin, Y., Liu, Z., Zhang, Z., and Hu, H. Leveraging batch normalization for vision transformers. In ICCV, 2021

    2021

  58. [66]

    Large batch optimization for deep learning: Training bert in 76 minutes

    You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes. In ICLR, 2020

    2020

  59. [67]

    Width & depth pruning for vision transformers

    Yu, F., Huang, K., Wang, M., Cheng, Y., Chu, W., and Cui, L. Width & depth pruning for vision transformers. In AAAI, 2022 a

    2022

  60. [68]

    and Xiang, W

    Yu, L. and Xiang, W. X-pruner: explainable pruning for vision transformers. In CVPR, 2023

    2023

  61. [69]

    Unified visual transformer compression

    Yu, S., Chen, T., Shen, J., Yuan, H., Tan, J., Yang, S., Liu, J., and Wang, Z. Unified visual transformer compression. In ICLR, 2022 b

    2022

  62. [70]

    Metaformer is actually what you need for vision

    Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., and Yan, S. Metaformer is actually what you need for vision. In CVPR, 2022 c

    2022

  63. [71]

    Dense vision transformer compression with few samples

    Zhang, H., Zhou, Y., and Wang, G.-H. Dense vision transformer compression with few samples. In CVPR, 2024

    2024

  64. [72]

    Rethinking mobile block for efficient attention-based models

    Zhang, J., Li, X., Li, J., Liu, L., Xue, Z., Zhang, B., Jiang, Z., Huang, T., Wang, Y., and Wang, C. Rethinking mobile block for efficient attention-based models. In ICCV, 2023

    2023

  65. [73]

    Scene parsing through ade20k dataset

    Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. In CVPR, 2017

    2017

  66. [74]

    Structural reparameterization lightweight network for video action recognition

    Zhu, A., Wang, Y., Li, W., and Qian, P. Structural reparameterization lightweight network for video action recognition. In ICASSP, 2023

    2023

  67. [75]

    Self-slimmed vision transformer

    Zong, Z., Li, K., Song, G., Wang, Y., Qiao, Y., Leng, B., and Liu, Y. Self-slimmed vision transformer. In ECCV, 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.