Pith. sign in

REVIEW 5 major objections 7 minor 40 references

Lightweight Backbone Networks Only Require Adaptive Lightweight Self-Attention Mechanisms

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LOLViT claims CNN-level CPU speed with an adaptive attention mechanism, running 5x faster than MobileViT-X.

desk verdict A plausible attention variant buried in an incomplete submission with a baseline that doesn't hold up to scrutiny. read the letter →

arxiv 2508.01385 v1 pith:EP2V4JCT submitted 2025-08-02 cs.CV

Fengyun Li , Chao Zheng , Yangyang Fang , Jialiang Lan , Jianhua Liang , Luhao Zhang , Fa Si This is my paper
classification cs.CV
keywords TransformerCNNSoftmaxAttentionVisionFastWindowlightweightbackboneobjectdetectioninstancesegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a lightweight hybrid backbone does not have to choose between CNN-like inference speed and attention-level accuracy. It introduces Fast Window Attention (FWA), a SoftMax attention variant whose number of key and value sequences adapts to the input feature map through window aggregation, avoiding both the quadratic cost of global attention and the need to hand-pick a fixed reduction ratio. The authors combine FWA with a GhostNet backbone and a global-local fusion branch to build LOLViT, and report that it matches or beats same-level CNN backbones in speed and accuracy on classification, detection, and segmentation benchmarks. The reason to care is the practical claim that attention-equipped lightweight models can run at CNN speed on CPUs without giving up the global modeling that attention provides; most notably, LOLViT-X's CPU inference time is reported as about one fifth of MobileViT-X's on Mini ImageNet.

What carries the argument

The central mechanism is Fast Window Attention (FWA), whose Full-Scene Adaptive Window Aggregation (FAWA) generates a small number of key and value sequences by aggregating each window of the feature map, making attention cost adapt to input resolution without manual reduction-ratio selection. The machinery also includes DReLu, which uses ReLU in place of SoftMax for the attention weights, and QK caching, which stores key sequences to avoid repeated computation. A parallel global-local FeatureFusion branch added to the GhostNet bottleneck captures local detail within patches. FWA carries the speed claim: it turns the quadratic sequence length of SoftMax attention into a short, resolution-adaptive key set while keeping attention weights fine-grained and CPU-friendly.

What would settle it

Run MobileViT-X at 384x384, the same resolution used for LOLViT-X, on the same CPU and measure per-image inference time; if the resulting time is not roughly five times LOLViT-X's, the headline speed claim collapses.

Watch

Extended reading notes

Core claim

The central claim is that the computational bottleneck of SoftMax attention in lightweight hybrid backbones can be removed by making the number of key/value sequences adaptive rather than fixed. FWA's Full-Scene Adaptive Window Aggregation (FAWA) partitions the feature map into windows and aggregates each window into a small set of keys and values; because the window count depends on the input size, no separate reduction-ratio tuning is needed. The paper also argues that ReLU can stand in for SoftMax in lightweight global attention, introducing a variant called DReLu, and caches the resulting key sequences to avoid redundant computation. These components are assembled on a GhostNet backbone with a parallel FeatureFusion branch to form LOLViT. Across ImageNet-1K, Mini ImageNet, COCO, VOC, and BDD100K, the authors report that LOLViT reaches CNN-like inference speed with accuracy competitive with MobileViT, and that LOLViT-X's CPU inference time is about one fifth of MobileViT-X's on Mini ImageNet.

Load-bearing premise

The speed comparisons treat CPU inference timings at different input resolutions as directly comparable, and assume hyperparameters tuned on ImageNet-1K transfer to Mini ImageNet, COCO, VOC, and BDD100K without re-tuning.

Editorial extensions

If this is right

  • If the reported timings hold, lightweight hybrid vision backbones can run on standard CPUs at CNN-like speeds, removing a main deployment barrier for attention models.
  • Because FAWA adapts the key-sequence count to input size, the same model can switch between 224, 320, 384, and 640 resolution inputs without re-tuning attention hyperparameters.
  • The DReLu result implies that ReLU can replace SoftMax in lightweight global attention without sacrificing accuracy, offering a faster and simpler alternative for similar architectures.
  • In detection with an SSD head, LOLViT nearly doubles FPS over MobileViT-XS at comparable mAP, suggesting the speed advantage transfers beyond classification.
  • The concrete observation behind the '5x' headline is LOLViT-X's 12.1 ms CPU inference versus MobileViT-X's 63.0 ms on Mini ImageNet at 224x224.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test is video or multi-scale inference, where input resolution changes per frame and FAWA's adaptive key count could be especially valuable.
  • The paper attributes the speed and accuracy gains to FWA, but a controlled test that swaps only the attention module into MobileViT's own block would more cleanly separate FWA's contribution from the GhostNet base and fusion branch.
  • The ReLU-for-SoftMax argument is presented as a rationale rather than a formal derivation; if it generalizes, the same substitution could benefit other lightweight transformers whose attention inputs stay positive and bounded.
  • A fairer CPU benchmark would run both models at the same input resolution, since the headline speed ratio mixes resolution differences with architectural differences.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes Fast Window Attention (FWA), a lightweight SoftMax attention mechanism that adaptively generates key/value sequences via window aggregation, and a ReLU-based approximation (DReLu) of SoftMax to speed up hybrid backbones. The FWA mechanism is integrated with GhostNet to form the LOLViT backbone. The authors claim that LOLViT outperforms same-level CNN models in both accuracy and inference speed and that LOLViT-X is 5x faster than MobileViT-X, with experiments reported for ImageNet-1K, Mini ImageNet, COCO, PASCAL VOC, and BDD100K.

Significance. If the central claims are substantiated, the contribution would be practically valuable: a lightweight hybrid backbone that approaches CNN-level CPU speed while retaining attention-model accuracy would be a useful data point for efficient vision architectures. The manuscript's strengths are the breadth of the experimental coverage (classification, detection, segmentation, and ablations) and the explicit architecture tables (Tables 16-17). The paper also candidly acknowledges limitations, such as accuracy drops against YOLO models and information loss for large-scale images in detection. However, the current form is not yet publishable: the method sections are absent, the headline speed comparison uses an undefined baseline, and the tuning protocol is not described. These issues prevent an evaluation of the central novelty and the correctness of the reported speed advantages.

major comments (5)
  1. [Sections 2-4 (Method)] The manuscript jumps from Section 1 (Introduction) directly to Section 5 (Experiments), and no equations or algorithmic descriptions are provided for FWA, FAWA, DReLu, FeatureFusion, or the LOLViTBlock. Tables 16 and 17 list only layer sizes and parameter counts, not the attention computation. As a result, the central methodological claims—adaptive key-sequence generation, the ReLU-as-SoftMax approximation, and the caching mechanism—cannot be verified or reproduced from the submitted text. This omission is load-bearing for the paper's central claims and must be fixed.
  2. [Table 4 and Abstract] The headline claim that "the inference speed of LOLViT-X is 5x that of MobileViT-X" rests on a baseline that is not defined anywhere in the manuscript. No parameters, FLOPs, or source are given for "MobileViT-X," and Table 3 reports the same-resolution MobileViT-XS at 46.3 ms, not 63.0 ms. If "MobileViT-X" is intended to be MobileViT-XS, the speedup is 12.1/46.3 = 3.8x, not 5x; if it is a different or custom model, the comparison is unreproducible. This claim appears in the abstract and Section 1 and must be corrected or substantiated before the paper can be accepted.
  3. [Table 3] The ImageNet-1K speed comparison in Table 3 pairs LOLViT-X and LOLViT-S at 384x384 input with all baseline models at 224x224 input. The 224x224 rows for LOLViT are present later in the same table, so the fair comparison exists, but the table layout and the associated text should be reorganized so that speed comparisons are made at matching resolutions and the 384x384 rows are not presented as directly comparable to the 224x224 baseline timings.
  4. [Tables 9, 10, 12, and 24 (Ablation experiments)] The free parameters DP, L, Fold, Patch size, and the number of heads are selected by maximizing Top-1 accuracy on ImageNet-1K, and the final ImageNet-1K models in Table 3 are then evaluated on the same dataset. No held-out validation split or cross-validation is described. This creates a risk that the reported gains over baselines reflect model selection on the test benchmark rather than a method-independent advantage. The authors should specify the validation protocol used for all ablations and final training, or use a separate validation set.
  5. [Section 5.2 and section G (VOC and BDD100K)] The text claims results on PASCAL VOC and BDD100K, but the corresponding result tables are not included in the manuscript. Table 7 is referenced in Section 5.2 but is absent, and no table presents instance-segmentation results on BDD100K, despite the abstract claiming segmentation experiments. Without these data, the generalizability claims for detection and segmentation cannot be checked.
minor comments (7)
  1. [Abstract and throughout] There are several typos and inconsistently used terms: "efficiencys" in the abstract, "subection" in Section 5.1, "FW A" vs. "FWA", and "ReLu" vs. "ReLU". Please standardize the notation and proofread.
  2. [Table 24] The Top-1 values in Table 24 (around 57%) are far lower than the ImageNet-1K results in Table 3 (around 69-75%) for models with similar names. If these are short-epoch (e.g., 20-epoch) ablations, the table must state the number of epochs; otherwise, the inconsistency is confusing.
  3. [Table 3] The 224x224 rows for LOLViT-X and LOLViT-S list Params and FLOPs as "-"; these values should be reported for all rows to allow fair comparison with the baselines.
  4. [References] The reference list is not included in the visible manuscript, although citations such as [1] and [19] appear in the text. All references must be resolved and included.
  5. [Section H (Ablation experiments)] Figure 2 is referenced in the description of the LOLViTBlock but is not visible in the manuscript; all referenced figures should be included.
  6. [Tables 5 and 6] Statements such as "5% reduction in accuracy" and "2x speedup" in Section 5.2 are ambiguous about the reference baseline (MobileViT-XS vs. MobileViT vs. MobileViTv2). Please state the exact comparison model for each claim.
  7. [Section 5.2] The sentence "when trained with large-scale images, FWA will lose some key information, leading to a decrease in accuracy" is an important limitation, but it is not quantified or further discussed. The authors should provide quantitative evidence or a more detailed explanation of this trade-off.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper's attention mechanism and backbone are empirically measured against external benchmarks, and the central speed and accuracy claims are not defined into existence. The 5x speed figure relies on an unclear 'MobileViT-X' baseline, but that is a reproducibility issue, not circularity.

full rationale

This paper contains no formal derivation chain whose conclusions reduce to its premises. FWA is defined as a window-aggregation attention mechanism, and the reported classification, detection, segmentation, and inference-time results are measured outcomes on external benchmarks (ImageNet-1K, COCO, VOC, BDD100K) rather than quantities implied by the definition. Hyperparameters such as DP=0.5, the number of heads, and the L-Fold-Patch configuration are selected through ablations on ImageNet-1K (Tables 10, 12, 24) before the final ImageNet-1K result is reported; this is standard model selection, not a fitted parameter disguised as a prediction, because the final Top-1 accuracies are independent measurements rather than statistically forced values. The abstract's claim that 'the inference speed of LOLViT-X is 5x that of MobileViT-X' is simple arithmetic from Table 4 (12.1 ms vs 63.0 ms), and while the 'MobileViT-X' baseline is not a standard public model and its 63.0 ms timing is inconsistent with the MobileViT-XS timing of 46.3 ms in Table 3, that inconsistency undermines reproducibility and fair comparison, not the logical independence of the measurement. The paper also states a limitation that 'when trained with large-scale images, FWA will lose some key information, leading to a decrease in accuracy,' which is an honest caveat rather than a circular justification. No load-bearing self-citation chain, uniqueness theorem imported from the authors' prior work, or ansatz smuggled in via citation is present. Therefore, no step in the claimed derivation reduces to its own inputs by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central performance claims rest on a small number of architectural choices and experimental assumptions. No free parameters are derived from first principles; all are selected by ablation or tuning. The ReLU-as-SoftMax 'explanation' is an assumption in the visible text, not a proof.

free parameters (5)
  • DP (DReLu learnable scale) = 0.5
    Table 10 scans DP from 0.1 to 1.0 on ImageNet-1K; 0.5 gives Top1=58.0 and is adopted in the model. It is tuned on the same benchmark used to report final results.
  • L (Transformer repeat counts) = 2-4-3 (or 1-1-1 in ablations)
    Table 24 selects L/Fold/Patch by ImageNet-1K Top1; main model uses 2-4-3.
  • Fold factors = 1-2-4
    Further compresses key sequences; chosen by ablation in Table 24.
  • Patch sizes = 1-1-1
    Selected in Table 24.
  • Number of attention heads = 4 per block (4-4-4)
    Table 12 finds 4-4-4 best on ImageNet-1K.
assumptions (4)
  • domain assumption SoftMax attention with global key sequences is necessary for the accuracy of hybrid backbones
    The paper frames linear attention as weakening long-sequence global modeling (Section 1), motivating FWA; this is a design premise.
  • ad hoc to paper FAWA window aggregation preserves task-relevant global information
    No theoretical guarantee or analysis is provided; FWA's accuracy depends on this.
  • ad hoc to paper DReLu (ReLU plus learned DP) approximates SoftMax attention weights
    Abstract says the paper explains this, but the visible text only shows ablations and a fitted DP.
  • domain assumption CPU inference timings in Tables 3 and 4 are comparable across models and implementations
    No code or benchmarks are provided to verify fair timing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lightweight Backbone Networks Only Require Adaptive Lightweight Self-Attention Mechanisms." pith.science (2026). https://pith.science/paper/EP2V4JCT

@misc{pith2026250801385,
  author       = {Pith},
  title        = {Pith review of: Lightweight Backbone Networks Only Require Adaptive Lightweight Self-Attention Mechanisms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EP2V4JCT}},
  note         = {Machine review of arXiv:2508.01385}
}
read the original abstract

Currently, lightweight hybrid backbone networks have partially alleviated the issue of computational saturation, but the imbalance in computational efficiencys between convolutional neural networks (CNNs) and attention mechanisms is becoming increasingly apparent. Specifically, although linear attention mechanisms and their variants have made progress in lightweight design, they still fail to meet the demands of hybrid models for long-sequence modeling. On the other hand, existing lightweight SoftMax attention computations typically reduce the feature map to a fixed size to decrease the number of sequences, thereby compressing the computational scale. However, the process of determining the feature map reduction ratio is cumbersome, and computational saturation issues still persist. To address this issue, this paper proposes a lightweight SoftMax attention mechanism with adaptive feature map sizes, named Fast Window Attention (FWA), which generates a small number of key sequences (Key and Value) through window aggregation for attention computation. Additionally, it explains the rationality of using ReLU to simulate SoftMax operations in lightweight global attention mechanisms. Finally, the paper designs a global-local feature fusion mechanism and combines it with GhostNet to propose a lightweight hybrid backbone network, LOLViT. Through visual tasks such as classification (ImageNet 1K), detection (COCO 2017), and segmentation (BDD100K), along with extensive ablation studies, it is demonstrated that LOLViT outperforms CNN models of the same level in both inference speed and model accuracy. Notably, the inference speed of LOLViT-X is 5x that of MobileViT-X.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 24 canonical work pages

  1. [1]

    Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer

    Sachin Mehta and Mohammad Rastegari. Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. arXiv preprint arXiv:2110.02178 , 2021

    arXiv 2021

  2. [2]

    Rethinking spatial dimensions of vision transformers

    Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 11936--11945, 2021

    work page 2021

  3. [3]

    Mobile-former: Bridging mobilenet and transformer

    Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5270--5279, 2022

    work page 2022

  4. [4]

    Efficientformer: Vision transformers at mobilenet speed

    Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems , 35:12934--12949, 2022

    2022

  5. [5]

    Cvt: Introducing convolutions to vision transformers

    Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 22--31, 2021

    work page 2021

  6. [6]

    Tokenlearner: What can 8 learned tokens do for images and videos? arXiv preprint arXiv:2106.11297 , 2021

    Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: What can 8 learned tokens do for images and videos? arXiv preprint arXiv:2106.11297 , 2021

    arXiv 2021

  7. [7]

    Edgevits: Competing light-weight cnns on mobile devices with vision transformers

    Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tzimiropoulos, and Brais Martinez. Edgevits: Competing light-weight cnns on mobile devices with vision transformers. In European conference on computer vision , pages 294--311. Springer, 2022

    work page 2022

  8. [8]

    Hiri-vit: Scaling vision transformer with high resolution inputs

    Ting Yao, Yehao Li, Yingwei Pan, and Tao Mei. Hiri-vit: Scaling vision transformer with high resolution inputs. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024

    work page 2024

Show all 40 references
  1. [9]

    Levit: a vision transformer in convnet's clothing for faster inference

    Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herv \'e J \'e gou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF international conference on computer vision , pages 12...

    2021

  2. [10]

    You only need less attention at each stage in vision transformers

    Shuoxi Zhang, Hanpeng Liu, Stephen Lin, and Kun He. You only need less attention at each stage in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6057--6066, 2024

    2024

  3. [11]

    Separable self-attention for mobile vision transformers

    Sachin Mehta and Mohammad Rastegari. Separable self-attention for mobile vision transformers. arXiv preprint arXiv:2206.02680 , 2022

    2022 arXiv

  4. [12]

    Cas-vit: Convolutional additive self-attention vision transformers for efficient mobile applications

    Tianfang Zhang, Lei Li, Yang Zhou, Wentao Liu, Chen Qian, Jenq-Neng Hwang, and Xiangyang Ji. Cas-vit: Convolutional additive self-attention vision transformers for efficient mobile applications. arXiv preprint arXiv:2408.03703 , 2024

    2024 arXiv

  5. [13]

    Efficientvit: Multi-scale linear attention for high-resolution dense prediction

    Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han. Efficientvit: Multi-scale linear attention for high-resolution dense prediction. arXiv preprint arXiv:2205.14756 , 2022

    2022 arXiv

  6. [14]

    Lightweight vision transformer with cross feature attention

    Youpeng Zhao, Huadong Tang, Yingying Jiang, Qiang Wu, et al. Lightweight vision transformer with cross feature attention. arXiv preprint arXiv:2207.07268 , 2022

    2022 arXiv

  7. [15]

    Fastervit: Fast vision transformers with hierarchical attention

    Ali Hatamizadeh, Greg Heinrich, Hongxu Yin, Andrew Tao, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. Fastervit: Fast vision transformers with hierarchical attention. arXiv preprint arXiv:2306.06189 , 2023

    2023 arXiv

  8. [16]

    P 2fevit: Plug-and-play cnn feature embedded hybrid vision transformer for remote sensing image classification

    Guanqun Wang, He Chen, Liang Chen, Yin Zhuang, Shanghang Zhang, Tong Zhang, Hao Dong, and Peng Gao. P 2fevit: Plug-and-play cnn feature embedded hybrid vision transformer for remote sensing image classification. Remote Sensing , 15(7):1773, 2023

    2023

  9. [17]

    Restoring images in adverse weather conditions via histogram transformer

    Shangquan Sun, Wenqi Ren, Xinwei Gao, Rui Wang, and Xiaochun Cao. Restoring images in adverse weather conditions via histogram transformer. In European Conference on Computer Vision , pages 111--129. Springer, 2024

    2024

  10. [18]

    Slab: Efficient transformers with simplified linear attention and progressive re-parameterized batch normalization

    Jialong Guo, Xinghao Chen, Yehui Tang, and Yunhe Wang. Slab: Efficient transformers with simplified linear attention and progressive re-parameterized batch normalization. arXiv preprint arXiv:2405.11582 , 2024

    2024 arXiv

  11. [19]

    Agent attention: On the integration of softmax and linear attention

    Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Siyuan Pan, Pengfei Wan, Shiji Song, and Gao Huang. Agent attention: On the integration of softmax and linear attention. In European Conference on Computer Vision , pages 124--140. Springer, 2024

    2024

  12. [20]

    Mobilenetv4: universal models for the mobile ecosystem

    Danfeng Qin, Chas Leichner, Manolis Delakis, Marco Fornoni, Shixin Luo, Fan Yang, Weijun Wang, Colby Banbury, Chengxi Ye, Berkin Akin, et al. Mobilenetv4: universal models for the mobile ecosystem. In European Conference on Computer Vision , pages 78--96. Springer, 2024

    2024

  13. [21]

    Ghostnetv3: Exploring the training strategies for compact models

    Zhenhua Liu, Zhiwei Hao, Kai Han, Yehui Tang, and Yunhe Wang. Ghostnetv3: Exploring the training strategies for compact models. arXiv preprint arXiv:2404.11202 , 2024

    2024 arXiv

  14. [22]

    Repvit: Revisiting mobile cnn from vit perspective

    Ao Wang, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. Repvit: Revisiting mobile cnn from vit perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15909--15920, 2024

    2024

  15. [23]

    Searching for mobilenetv3

    Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1314--1324, 2019

    2019

  16. [24]

    Patches are all you need? arXiv preprint arXiv:2201.09792 , 2022

    Asher Trockman and J Zico Kolter. Patches are all you need? arXiv preprint arXiv:2201.09792 , 2022

    2022 arXiv

  17. [25]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017

    2017

  18. [26]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  19. [27]

    Mlp-mixer: An all-mlp architecture for vision

    Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems , 34:24...

    2021

  20. [28]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770--778, 2016

    2016

  21. [29]

    Mobilevitv3: Mobile-friendly vision transformer with simple and effective fusion of local, global and input features

    Shakti N Wadekar and Abhishek Chaurasia. Mobilevitv3: Mobile-friendly vision transformer with simple and effective fusion of local, global and input features. arXiv preprint arXiv:2209.15159 , 2022

    2022 arXiv

  22. [30]

    Tinyvit: Fast pretraining distillation for small vision transformers

    Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European conference on computer vision , pages 68--85. Springer, 2022

    2022

  23. [31]

    Ghostnet: More features from cheap operations

    Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1580--1589, 2020

    2020

  24. [32]

    Replacing softmax with relu in vision transformers

    Mitchell Wortsman, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Replacing softmax with relu in vision transformers. arXiv preprint arXiv:2309.08586 , 2023

    2023 arXiv

  25. [33]

    YOLO series , 2024

    Jocher Glenn. YOLO series , 2024

    2024

  26. [34]

    Yolov9: Learning what you want to learn using programmable gradient information

    Chien-Yao Wang, I-Hau Yeh, and Hong-Yuan Mark Liao. Yolov9: Learning what you want to learn using programmable gradient information. In European conference on computer vision , pages 1--21. Springer, 2024

    2024

  27. [35]

    Yolov10: Real-time end-to-end object detection

    Ao Wang, Hui Chen, Lihao Liu, Kai Chen, Zijia Lin, Jungong Han, et al. Yolov10: Real-time end-to-end object detection. Advances in Neural Information Processing Systems , 37:107984--108011, 2024

    2024

  28. [36]

    Yolov12: Attention-centric real-time object detectors

    Yunjie Tian, Qixiang Ye, and David Doermann. Yolov12: Attention-centric real-time object detectors. arXiv preprint arXiv:2502.12524 , 2025

    2025 arXiv

  29. [37]

    Ghostnetv2: Enhance cheap operation with long-range attention

    Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu, Chao Xu, and Yunhe Wang. Ghostnetv2: Enhance cheap operation with long-range attention. Advances in Neural Information Processing Systems , 35:9969--9982, 2022

    2022

  30. [38]

    Run, don't walk: chasing higher flops for faster neural networks

    Jierun Chen, Shiu-hong Kao, Hao He, Weipeng Zhuo, Song Wen, Chul-Ho Lee, and S-H Gary Chan. Run, don't walk: chasing higher flops for faster neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12021--12031, 2023

    2023

  31. [39]

    Yolop: You only look once for panoptic driving perception

    Dong Wu, Man-Wen Liao, Wei-Tian Zhang, Xing-Gang Wang, Xiang Bai, Wen-Qing Cheng, and Wen-Yu Liu. Yolop: You only look once for panoptic driving perception. Machine Intelligence Research , 19(6):550--562, 2022

    2022

  32. [40]

    You only look at once for real-time and generic multi-task

    Jiayuan Wang, QM Jonathan Wu, and Ning Zhang. You only look at once for real-time and generic multi-task. IEEE Transactions on Vehicular Technology , 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.