Pith. sign in

REVIEW 12 cited by

Separable Self-attention for Mobile Vision Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2206.02680 v1 pith:RPLVPQTU submitted 2022-06-06 cs.CV cs.AIcs.LG

Sachin Mehta , Mohammad Rastegari This is my paper
classification cs.CVcs.AIcs.LG
keywords mobileself-attentionvisionmobilevittransformersclassificationcomplexitycomputing
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Mobile vision transformers (MobileViT) can achieve state-of-the-art performance across several mobile vision tasks, including classification and detection. Though these models have fewer parameters, they have high latency as compared to convolutional neural network-based models. The main efficiency bottleneck in MobileViT is the multi-headed self-attention (MHA) in transformers, which requires $O(k^2)$ time complexity with respect to the number of tokens (or patches) $k$. Moreover, MHA requires costly operations (e.g., batch-wise matrix multiplication) for computing self-attention, impacting latency on resource-constrained devices. This paper introduces a separable self-attention method with linear complexity, i.e. $O(k)$. A simple yet effective characteristic of the proposed method is that it uses element-wise operations for computing self-attention, making it a good choice for resource-constrained devices. The improved model, MobileViTv2, is state-of-the-art on several mobile vision tasks, including ImageNet object classification and MS-COCO object detection. With about three million parameters, MobileViTv2 achieves a top-1 accuracy of 75.6% on the ImageNet dataset, outperforming MobileViT by about 1% while running $3.2\times$ faster on a mobile device. Our source code is available at: \url{https://github.com/apple/ml-cvnets}

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers

    cs.CV 2025-05 conditional novelty 7.0 of 10

    Keeping 75% of FFN channels idle (linear) lets RePaViT merge two large projections into one, reducing inference FLOPs by ~41% and latency by ~40% on large ViTs.

  2. Toward Deployable Bangla Sign Language Recognition with Expert-Validated Data and a Lightweight Attention-Based Model

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A 0.30M-parameter from-scratch CNN matches ImageNet-pretrained backbones on a new expert-validated 38-class Bangla sign language benchmark and runs on commodity phones.

  3. UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A heterogeneous pyramidal vision encoder plus two-stage dense-then-generative pre-training yields faster, stronger on-device LVLM vision encoding than FastVLM.

  4. MAC-Gaze: Motion-Aware Continual Calibration for Mobile Gaze Tracking

    cs.HC 2025-05 conditional novelty 6.0 of 10

    MAC-Gaze uses IMU-based motion detection and replay-based continual learning to automatically recalibrate mobile gaze trackers, reducing gaze estimation error by 19.9% on RGBDGaze and 31.7% on MotionGaze.

  5. CoAtNeXt:An Attention-Enhanced ConvNeXtV2-Transformer Hybrid Model for Gastric Tissue Classification

    cs.CV 2025-09 conditional novelty 5.0 of 10

    CoAtNeXt, a CoAtNet variant using ConvNeXtV2 blocks and CBAM attention, reports state-of-the-art accuracy on two gastric histopathology datasets.

  6. Lightweight Backbone Networks Only Require Adaptive Lightweight Self-Attention Mechanisms

    cs.CV 2025-08 conditional novelty 5.0 of 10

    LOLViT, a GhostNet-based lightweight backbone using adaptive window attention, reports CNN-like CPU speed with MobileViT-level accuracy.

  7. Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Vision networks whose bottlenecks avoid feature-map interference scale better than MobileNet-style designs at very low parameter counts, and the new NoDepth block demonstrates this on ImageNet.

  8. VCMamba: Bridging Convolutions with Multi-Directional Mamba for Efficient Visual Representation

    cs.CV 2025-09 conditional novelty 4.0 of 10

    VCMamba reports that using convolutional feed-forward blocks for the first three stages followed by multi-directional Mamba blocks in the final stage yields 82.6% ImageNet-1K and 47.1 ADE20K mIoU at 31.5M parameters, ...

  9. Foundation Models and Transformers for Anomaly Detection: A Survey

    cs.LG 2025-07 reject novelty 4.0 of 10

    A taxonomy and literature review of Transformer-based visual anomaly detection, compromised by fabricated citations with dummy arXiv IDs.

  10. LAID: Lightweight AI-Generated Image Detection in Spatial and Spectral Domains

    cs.CV 2025-07 conditional novelty 4.0 of 10

    A benchmark showing lightweight off-the-shelf models can exceed 99% accuracy on GenImage for AI-image detection, with spectral inputs and decision fusion, at a fraction of larger models' compute.

  11. DeepTraverse: A Depth-First Search Inspired Network for Algorithmic Visual Understanding

    cs.CV 2025-06 reject novelty 4.0 of 10

    DeepTraverse is a weight-tied residual network plus squeeze-and-excitation attention, framed as depth-first search, with claimed efficiency gains that rest on a questionable ImageNet subset comparison.

  12. EfficientQuant: An Efficient Post-Training Quantization for CNN-Transformer Hybrid Models on Edge Devices

    cs.CV 2025-06 reject novelty 4.0 of 10

    EfficientQuant applies uniform weight quantization to CNN blocks and logarithmic activation quantization to transformer blocks in hybrid models, reporting latency reductions of 2.5x to 8.7x with modest accuracy loss.

Pith tools