Pith. sign in

REVIEW 4 major objections 6 minor 44 references

UniForm: A Reuse Attention Mechanism Optimized for Efficient Vision Transformers on Edge Devices

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A shared attention matrix, reused across heads, is enough to run vision transformers on edge devices at up to 5x the speed of comparable baselines, without giving up accuracy.

desk verdict The shared-attention idea is not new and the headline speedup is not isolated by the experiments, but the broad edge-device benchmark and the honest body text make it a borderline case that needs a controlled ablation and an abstract rewrite. read the letter →

arxiv 2412.02344 v1 pith:XIRRIBQU submitted 2024-12-03 cs.CV

Seul-Ki Yeom , Tae-Ho Kim This is my paper · ORCID
classification cs.CV
keywords reuseattentionvisiontransformeredgeinferencememoryefficiencyImageNetclassificationmulti-scalevalueprocessingon-devicespeedupefficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the main obstacle to running vision transformers on edge devices is the memory traffic caused by multi-head attention, and that this obstacle can be removed by computing a single shared attention matrix per layer rather than one matrix per head. It introduces Reuse Attention, a mechanism that reuses that one matrix across all heads and compensates for the lost per-head diversity with multi-scale depthwise convolutions on the value projections. The authors build UniForm, a family of hierarchical vision transformers around this mechanism, and report ImageNet-1K top-1 accuracy of up to 76.7% with inference times as low as 2.4ms on Jetson AGX Orin and 21.8ms on Raspberry Pi 5, claiming up to 5x speedups over baselines such as EfficientViT-M0 while also beating them in accuracy. If correct, the mechanism makes real-time transformer-based vision practical on memory-constrained hardware without sacrificing accuracy.

What carries the argument

The load-bearing object is the shared attention matrix $A = \mathrm{softmax}(QK^\top/\sqrt{D})$, computed once per layer from a single query/key projection and reused by every head; the paper's argument is that this cuts memory traffic for the attention scores from $N^2 \times h$ (per-head) to a single $N^2$, which matters because edge devices are memory-bandwidth-bound. The complementary mechanism is multi-scale value processing: each head applies a depthwise convolution of a different kernel size to its value projection before multiplying by $A$, which the authors claim preserves the representational diversity that multi-head attention would otherwise provide.

What would settle it

Take the exact UniForm-l configuration and replace only the shared attention matrix with standard per-head attention, leaving every other component (patch embedding, DWConv+FFN, value convolutions) unchanged; measure inference time on the same edge devices and batch sizes. If the time on Jetson AGX Orin rises to the same level as EfficientViT-M4 or above, the causal claim that reuse produces the speedup is falsified.

Watch

Extended reading notes

Core claim

The central claim is that per-head attention matrices are redundant computation, so a single shared attention matrix per layer preserves model performance while drastically cutting memory load and store. Given input $X$, UniForm computes one query/key projection, forms $A = \mathrm{softmax}(QK^\top/\sqrt{D})$, and applies this same $A$ to every head's value output $V_h$, where $V_h$ is produced by a depthwise convolution with a distinct kernel size. The authors state this replaces $h$ separate $N \times N$ attention matrices with one, reducing memory movement (e.g., from 141.73 GB to 8.59 GB for a Llama-scale model in their extrapolation) and enabling faster inference on GPUs, CPUs, and edge devices. They support this with ImageNet-1K classification, downstream classification, and COCO instance segmentation results showing accuracy on par with or better than comparably sized efficient models.

Load-bearing premise

The claimed speedups assume the comparisons are fair and implementation-matched, because UniForm also changes patch embedding, DWConv+FFN placement, and value convolutions; if these other changes carry most of the speedup, the reuse mechanism itself may contribute far less than the 5x figure suggests.

Editorial extensions

If this is right

  • UniForm-l reaches 76.7% ImageNet-1K top-1 accuracy with 2.4ms inference on Jetson AGX Orin, showing transformer-based classification can run in real time on edge hardware.
  • Reuse Attention reduces attention memory movement from $N^2 \times h$ to $N^2$ per layer, so the benefit grows with token count and head count, making it attractive for high-resolution inputs and larger models.
  • Smaller UniForm variants (UniForm-t, 66.0% top-1) beat EfficientViT-M0 (63.2%) in accuracy and run up to ~5x faster on Jetson-Nano (11.9ms vs 56.8ms), implying the efficiency gain is not bought by accuracy loss.
  • The same backbone transfers to COCO instance segmentation with Mask R-CNN, where UniForm-l achieves 33.2 box AP and 31.5 mask AP, outperforming comparable efficient backbones listed in the paper.
  • On CPU and GPU throughput, UniForm-s reaches 50,582 images/s on A100 and 231 images/s on CPU, exceeding MobileNetV3-small and EfficientViT-M1, so the mechanism is not edge-specific.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the shared-matrix mechanism is what drives the speedup, the same design could be applied to language-model inference, where repeated attention-matrix reads dominate memory traffic; the paper's Table 2 extrapolates this to LLMs and VLMs, but reports no language-model measurements, so that extension needs a direct test.
  • The paper changes several architectural components at once (patch embedding, DWConv+FFN placement, multi-scale value convolutions), so the 5x speedup may not be solely attributable to the shared attention matrix; a controlled ablation that swaps only the attention module would isolate the mechanism's true contribution.
  • The memory-traffic reductions for models like Llama 2 and GPT-3 are computed from the reuse formula rather than measured, so the 93-94% reduction figures should be read as upper bounds until benchmarked on those architectures.
  • Since the shared matrix is computed from a single pair of query/key projections, the mechanism is also a parameter reduction of the attention module; this may reduce overfitting in small-data fine-tuning regimes, which the paper's Flowers-102 results hint at but do not analyze.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Reuse Attention, a mechanism that computes a single shared attention matrix per transformer layer and reuses it across all heads, with multi-scale depthwise convolutions on value projections to preserve representational diversity. The authors build a hierarchical vision backbone, UniForm, in four sizes, and report ImageNet-1K accuracy, FLOPs, latency on GPUs and edge devices, plus downstream classification and instance segmentation results. The central empirical claim is that Reuse Attention provides up to a 5x speedup over efficient baselines such as EfficientViT on edge devices, while the abstract also claims reduced computational complexity and memory overhead.

Significance. If the central claims were well supported, sharing an attention matrix across heads would be a useful memory-traffic optimization for edge inference, and the proposed multi-scale value processing is a plausible way to retain accuracy. The paper does include a memory-movement accounting (Tables 1 and 2) that makes the intended I/O-reduction argument explicit, and it evaluates across a broad set of edge platforms. However, the key empirical attribution and several headline claims are contradicted by the paper's own content, so the contribution as presented is not yet established. No code or measurement harness is provided, which further weakens the reproducibility of the latency results.

major comments (4)
  1. [Abstract vs. Table 6] The abstract states that UniForm-l achieves 21.8ms inference time on "edge devices like the Jetson AGX Orin," but Table 6 reports 2.4ms on the Jetson AGX Orin and 21.8ms on the Raspberry Pi 5. This is an internal inconsistency in the paper's most visible claim. The abstract either conflates two different devices or misreports the hardware, and the "up to 5x speedup" claim must be tied to a specific comparison on a specific device.
  2. [Abstract and §3.4] The abstract and introduction claim that Reuse Attention "significantly reduc[es] ... computational complexity," but Section 3.4 explicitly concedes that "the matrix multiplication complexity remains O(N^2D)." The proposed mechanism reduces attention-matrix memory traffic, not the asymptotic FLOP count. The wording of the central contribution is therefore misleading and should be corrected to claim reduced memory I/O or memory overhead rather than reduced computational complexity.
  3. [§4.4, Ablation study] The ablation section is the only place where Reuse Attention is compared against a standard-attention variant of UniForm, but it reports only Grad-CAM visualizations. The text asserts that Reuse Attention "significantly improv[es] inference time" without providing any latency or throughput numbers for the controlled comparison. Since UniForm differs from baselines such as EfficientViT in patch embedding, DWConv+FFN placement, value convolutions, and stage widths/depths, the 5x speedup attributed to Reuse Attention in Table 6 is not isolated by any experiment in the paper.
  4. [§4.2.2 and Table 6] The edge-device latency methodology is under-specified. The paper does not report the inference precision (FP32/FP16/INT8), number of warmup iterations, number of repeated trials, or whether the same inference framework and kernel implementations were used across all compared models. Without this information, the cross-model latency numbers in Table 6 cannot be treated as controlled measurements, and the speedup ratios derived from them are not reliable.
minor comments (6)
  1. [§4.4] The sentence "In this section, we ablate" is incomplete; it should introduce the ablation experiments and their settings.
  2. [Figure 7 caption] The caption contains the typo "prodyced" and claims that UniForm with Reuse Attention "can more precisely locate the objects of interest than other methods," which is not supported by the qualitative Grad-CAM figures alone.
  3. [§3.5] The phrase "Similar to previous Like previous hierarchical backbones" is a duplicated-word error and should be rewritten.
  4. [§4.1] There are typos such as "buit" for "built" and "plateforms" for "platforms," and the section would benefit from a careful proofread.
  5. [Table 3] The star-rating comparison of attention mechanisms is subjective and lacks a defined rubric; consider replacing it with quantitative metrics or removing it.
  6. [§4.3.1] The downstream classification paragraph reports qualitative trends but does not describe the fine-tuning protocol for CIFAR, Flowers-102, and Oxford-IIIT Pet, making the results hard to reproduce.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circular derivation found; the memory reduction follows from the mechanism's definition and the headline numbers are external measurements, so the main issues are ablation and evidence quality, not circularity.

full rationale

The paper contains no load-bearing circular derivation. The proposed Reuse Attention mechanism is defined by Equations (1)-(5): a single shared attention matrix A = softmax(QK^T/sqrt(D)) is computed once per layer, and each head applies this same A to its own multi-scale value projection, V_h = DWConv_{k_h}(X_h W_{V_h}). The claimed reduction in memory traffic is an arithmetic consequence of this definition (one N x N attention matrix instead of h), which is the mechanism itself rather than a fitted parameter disguised as a prediction. The ImageNet accuracy and edge-device latencies in Tables 5-7 are external measurements compared against published baselines, not quantities derived from the attention equations. No parameter is fitted to a subset of data and then 'predicted' on a closely related quantity, and there are no self-citations by Yeom/Kim carrying an argument. The main weaknesses are correctness and evidence issues, not circularity: the ablation in Section 4.4 does not isolate Reuse Attention from the overlapping patch embedding, DWConv+FFN placement, multi-scale value convolutions, and stage design; the abstract's headline '21.8ms inference time on edge devices like the Jetson AGX Orin' conflicts with Table 6, where 21.8ms is the Raspberry Pi 5 value and the AGX Orin value is 2.4ms; and the abstract's claim of reduced 'computational complexity' is contradicted by Section 3.4, which concedes the matrix multiplication complexity remains O(N^2D). These concerns affect how the speedup is attributed and how the contribution is stated, but they do not make the derivation circular.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the assumption that sharing one attention map preserves accuracy and that memory traffic, not compute, dominates edge inference. No new theoretical entities are introduced. The multi-scale value kernels and stage hyperparameters are hand-chosen without ablation.

free parameters (2)
  • Value kernel sizes per head = not reported
    Equation 3 uses different depthwise convolution kernel sizes per head, but the values are never listed or ablated.
  • Stage channel/depth/head settings = Table 4 variants
    Hand-selected architecture hyperparameters for Tiny, Small, Medium, and Large variants, with no sensitivity analysis.
assumptions (3)
  • domain assumption Attention maps across heads are redundant enough that a single shared map preserves accuracy.
    Invoked in Section 3.1, citing prior work on attention-map redundancy and head importance.
  • ad hoc to paper Multi-scale depthwise convolutions on value projections add enough representational diversity to compensate for the loss of per-head attention maps.
    Proposed in Section 3.3, motivated by MixConv and Inception, but no quantitative isolation is provided.
  • domain assumption Memory I/O, not arithmetic operations, is the dominant bottleneck on edge devices.
    Section 2.3 argues this from memory bandwidth numbers; it is load-bearing for the speed claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniForm: A Reuse Attention Mechanism Optimized for Efficient Vision Transformers on Edge Devices." pith.science (2026). https://pith.science/paper/XIRRIBQU

@misc{pith2026241202344,
  author       = {Pith},
  title        = {Pith review of: UniForm: A Reuse Attention Mechanism Optimized for Efficient Vision Transformers on Edge Devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XIRRIBQU}},
  note         = {Machine review of arXiv:2412.02344}
}
read the original abstract

Transformer-based architectures have demonstrated remarkable success across various domains, but their deployment on edge devices remains challenging due to high memory and computational demands. In this paper, we introduce a novel Reuse Attention mechanism, tailored for efficient memory access and computational optimization, enabling seamless operation on resource-constrained platforms without compromising performance. Unlike traditional multi-head attention (MHA), which redundantly computes separate attention matrices for each head, Reuse Attention consolidates these computations into a shared attention matrix, significantly reducing memory overhead and computational complexity. Comprehensive experiments on ImageNet-1K and downstream tasks show that the proposed UniForm models leveraging Reuse Attention achieve state-of-the-art imagenet classification accuracy while outperforming existing attention mechanisms, such as Linear Attention and Flash Attention, in inference speed and memory scalability. Notably, UniForm-l achieves a 76.7% Top-1 accuracy on ImageNet-1K with 21.8ms inference time on edge devices like the Jetson AGX Orin, representing up to a 5x speedup over competing benchmark methods. These results demonstrate the versatility of Reuse Attention across high-performance GPUs and edge platforms, paving the way for broader real-time applications

Figures

Figures reproduced from arXiv: 2412.02344 by the authors.

Figure 1
Figure 1. Comparison of speed and accuracy between UniForm [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of Memory Bandwidth between High [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Impact of the number of tokens on Memory Access and [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Overview of the Proposed Method compared to the previous attention mechanisms [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: (a) The architecture of UniForm; (b) UniForm Block including Reuse Attention. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Chart comparison of UniForm models with comparable state-of-the-art models across different sizes on a variety of metrics [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Visualization of feature maps prodyced by three different models (Swin-T, UniForm without reuse attention, and UniForm with [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 37 canonical work pages

  1. [1]

    GQA: training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr ´on, and Sumit Sanghai. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2023, Singapore, December 6-10, 2023 , pages 4895–4901. Associat...

    work page 2023

  2. [2]

    Reducing transformer key-value cache size with cross-layer attention

    William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan-Kelley. Reducing transformer key-value cache size with cross-layer attention. CoRR, abs/2405.12981, 2024. 5

    arXiv 2024

  3. [3]

    MMDetection: Open mmlab detection toolbox and benchmark

    Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tian- heng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and...

    arXiv 1906

  4. [4]

    Mobile- former: Bridging mobilenet and transformer

    Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile- former: Bridging mobilenet and transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 5260–5269. IEEE, 2022. 7 Table 8. Performance comparison of instance segmentation on C...

    work page 2022

  5. [5]

    DHA: learning decoupled-head attention from transformer checkpoints via adaptive heads fusion

    Yilong Chen, Linhao Zhang, Junyuan Shang, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, and Yu Sun. DHA: learning decoupled-head attention from transformer checkpoints via adaptive heads fusion. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Infor- mation Processing Systems 2024, NeurIPS 2024, Vancouver, Canada, December ...

    work page 2024

  6. [6]

    Openmmlab’s pre-training tool- box and benchmark

    MMPreTrain Contributors. Openmmlab’s pre-training tool- box and benchmark. https://github.com/open- mmlab/mmpretrain, 2023. 7

    work page 2023

  7. [7]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R ´e

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher R ´e. Flashattention: Fast and memory-efficient (A) Small (B) Middle (C) Large Figure 6. Chart comparison of UniForm models with comparable state-of-the-art models across different sizes on a variety of metrics (Top-1 Accuracy and GPU/CPU/Edge-device throughput). Input imagesSwin-TUniFormw...

    work page 2022

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20- 25 June 2009, Miami, Florida, USA , pages 248–255. IEEE Computer Society, 2009. 7

    work page 2009

Show all 44 references
  1. [9]

    The case for 4-bit pre- cision: k-bit inference scaling laws

    Tim Dettmers and Luke Zettlemoyer. The case for 4-bit pre- cision: k-bit inference scaling laws. In International Con- ference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pages 7750–7774. PMLR, 2023. 2

    2023

  2. [10]

    Speeddetr: Speed-aware transformers for end-to-end object detection

    Peiyan Dong, Zhenglun Kong, Xin Meng, Peng Zhang, Hao Tang, Yanzhi Wang, and Chih-Hsien Chou. Speeddetr: Speed-aware transformers for end-to-end object detection. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pages 8227–824...

    2023

  3. [11]

    Is flash attention stable? CoRR, abs/2405.02803, 2024

    Alicia Golden, Samuel Hsia, Fei Sun, Bilge Acun, Basil Hos- mer, Yejin Lee, Zachary DeVito, Jeff Johnson, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. Is flash attention stable? CoRR, abs/2405.02803, 2024. 4

    2024 arXiv

  4. [12]

    Levit: a vision transformer in convnet’s clothing for faster inference

    Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herv ´e J ´egou, and Matthijs Douze. Levit: a vision transformer in convnet’s clothing for faster inference. In 2021 IEEE/CVF International Con- ference on Computer Vision, ICCV 2021, Montreal, QC,...

    2021

  5. [13]

    Flatten transformer: Vision transformer using fo- cused linear attention

    Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using fo- cused linear attention. In IEEE/CVF International Confer- ence on Computer Vision, ICCV 2023, Paris, France, Octo- ber 1-6, 2023, pages 5938–5948. IEEE, 2023. 4

    2023

  6. [14]

    Le, Mark Sandler, Bo Chen, Weijun Wang, Liang-Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu

    Andrew Howard, Ruoming Pang, Hartwig Adam, Quoc V . Le, Mark Sandler, Bo Chen, Weijun Wang, Liang-Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Searching for mobilenetv3. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Kor...

    2019

  7. [15]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InProceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , pages...

    2020

  8. [16]

    Spvit: Enabling faster vision transformers via latency-aware soft token pruning

    Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, Minghai Qin, and Yanzhi Wang. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In Computer Vision - ECCV 2022 - 17th European Confe...

    2022

  9. [17]

    Couplformer: Rethinking vision transformer with cou- pling attention

    Hai Lan, Xihao Wang, Hao Shen, Peidong Liang, and Xian Wei. Couplformer: Rethinking vision transformer with cou- pling attention. In IEEE/CVF Winter Conference on Applica- tions of Computer Vision, WACV 2023, Waikoloa, HI, USA, January 2-7, 2023, pages 6464–6473. IEEE, 2023. 2

    2023

  10. [18]

    Re- thinking vision transformers for mobilenet size and speed

    Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Re- thinking vision transformers for mobilenet size and speed. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, page...

    2023

  11. [19]

    Re- thinking vision transformers for mobilenet size and speed

    Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Re- thinking vision transformers for mobilenet size and speed. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, page...

    2023

  12. [20]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Computer Vision - ECCV 2014 - 13th Eu- ropean Conference, Zurich, Switzerland, September 6-12, 2014, ...

    2014

  13. [21]

    Efficientvit: Memory effi- cient vision transformer with cascaded group attention

    Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang, Han Hu, and Yixuan Yuan. Efficientvit: Memory effi- cient vision transformer with cascaded group attention. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17- 24, 202...

    2023

  14. [22]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vi- sion, ICCV 2021, Montreal, QC, Canada, October 10-17, ...

    2021

  15. [23]

    SGDR: stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenRe- view.net, 2017. 7

    2017

  16. [24]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. 7

    2019

  17. [25]

    SOFT: softmax-free transformer with linear complexity

    Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. SOFT: softmax-free transformer with linear complexity. In Advances in Neural Information Processing Systems 34: An- nual Conference on Neural Information Processing Sy...

    2021

  18. [26]

    Delight: Deep and light-weight transformer

    Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Delight: Deep and light-weight transformer. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Aus- tria, May 3-7, 2021. OpenReview.net, 2021. 4

    2021

  19. [27]

    Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? InAdvances in Neural Informa- tion Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, De- cember 8-14, 2019, Vancouver, BC, Canada , pages 14...

    2019

  20. [28]

    Fast vi- sion transformers with hilo attention

    Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Fast vi- sion transformers with hilo attention. In Advances in Neu- ral Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9,...

    2022

  21. [29]

    Yang, Zachary DeVito, Mar- tin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zem- ing Lin, Natalia Gimelshein, Luca Antiga, Alban Desmai- son, Andreas K ¨opf, Edward Z. Yang, Zachary DeVito, Mar- tin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit S...

    2019

  22. [30]

    Efficient neural net- works: From algorithm design to practical mobile deploy- ment

    Jian Ren, Sergey Tulyakov, and Ju Hu. Efficient neural net- works: From algorithm design to practical mobile deploy- ment. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2023. 2

    2023

  23. [31]

    Sparq attention: Bandwidth-efficient LLM inference

    Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Char- lie Blake, Carlo Luschi, and Douglas Orr. Sparq attention: Bandwidth-efficient LLM inference. In Forty-first Interna- tional Conference on Machine Learning, ICML 2024, Vi- enna, Austria, July 21-27, 2024. OpenReview.net, 2024. 6

    2024

  24. [32]

    Linear transformers are secretly fast weight programmers

    Imanol Schlag, Kazuki Irie, and J ¨urgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Pro- ceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pages 9355–9366. PMLR, 2021. 4

    2021

  25. [33]

    Efficient attention: Attention with linear complexities

    Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, Jan- uary 3-8, 2021, pages 3530–3538. IEEE, 2021. 4

    2021

  26. [34]

    Ex- ploring attention map reuse for efficient transformer neural networks

    Kyuhong Shim, Jungwook Choi, and Wonyong Sung. Ex- ploring attention map reuse for efficient transformer neural networks. CoRR, abs/2301.12444, 2023. 6

    2023 arXiv

  27. [35]

    Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June...

    2015

  28. [36]

    Mingxing Tan and Quoc V . Le. Mixconv: Mixed depthwise convolutional kernels. In 30th British Machine Vision Con- ference 2019, BMVC 2019, Cardiff, UK, September 9-12, 2019, page 74. BMV A Press, 2019. 5, 6

    2019

  29. [37]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neu- ral Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017...

    2017

  30. [38]

    Pyra- mid vision transformer: A versatile backbone for dense pre- diction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyra- mid vision transformer: A versatile backbone for dense pre- diction without convolutions. In 2021 IEEE/CVF Interna- tional Conference on Computer Vision, ICCV 2021,...

    2021

  31. [39]

    Global vision trans- former pruning with hessian-aware saliency

    Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision trans- former pruning with hessian-aware saliency. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023 , pages 18547–1...

    2023

  32. [40]

    Cutmix: Regulariza- tion strategy to train strong classifiers with localizable fea- tures

    Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regulariza- tion strategy to train strong classifiers with localizable fea- tures. In 2019 IEEE/CVF International Conference on Com- puter Vision, ICCV 2019, Seoul, Korea (South), ...

    2019

  33. [41]

    Dauphin, and David Lopez-Paz

    Hongyi Zhang, Moustapha Ciss ´e, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion. In 6th International Conference on Learning Represen- tations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net,

    2018

  34. [42]

    Minivit: Compressing vi- sion transformers with weight multiplexing

    Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Minivit: Compressing vi- sion transformers with weight multiplexing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pag...

    2022

  35. [43]

    Lightweight vision transformer with spatial and channel enhanced self-attention

    Jiahao Zheng, Longqi Yang, Yiying Li, Ke Yang, Zhiyuan Wang, and Jun Zhou. Lightweight vision transformer with spatial and channel enhanced self-attention. In IEEE/CVF International Conference on Computer Vision, ICCV 2023 - Workshops, Paris, France, October 2-6, 2023, pages 1484–

    2023

  36. [44]

    Random erasing data augmentation

    Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artifi- cial Intelligence Conference, IAAI 2020, The Tent...

    2020

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.