REVIEW 8 cited by
Separable Self-attention for Mobile Vision Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Mobile vision transformers (MobileViT) can achieve state-of-the-art performance across several mobile vision tasks, including classification and detection. Though these models have fewer parameters, they have high latency as compared to convolutional neural network-based models. The main efficiency bottleneck in MobileViT is the multi-headed self-attention (MHA) in transformers, which requires $O(k^2)$ time complexity with respect to the number of tokens (or patches) $k$. Moreover, MHA requires costly operations (e.g., batch-wise matrix multiplication) for computing self-attention, impacting latency on resource-constrained devices. This paper introduces a separable self-attention method with linear complexity, i.e. $O(k)$. A simple yet effective characteristic of the proposed method is that it uses element-wise operations for computing self-attention, making it a good choice for resource-constrained devices. The improved model, MobileViTv2, is state-of-the-art on several mobile vision tasks, including ImageNet object classification and MS-COCO object detection. With about three million parameters, MobileViTv2 achieves a top-1 accuracy of 75.6% on the ImageNet dataset, outperforming MobileViT by about 1% while running $3.2\times$ faster on a mobile device. Our source code is available at: \url{https://github.com/apple/ml-cvnets}
Forward citations
Cited by 8 Pith papers
-
UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models
A heterogeneous pyramidal vision encoder plus two-stage dense-then-generative pre-training yields faster, stronger on-device LVLM vision encoding than FastVLM.
-
CoAtNeXt:An Attention-Enhanced ConvNeXtV2-Transformer Hybrid Model for Gastric Tissue Classification
CoAtNeXt, a CoAtNet variant using ConvNeXtV2 blocks and CBAM attention, reports state-of-the-art accuracy on two gastric histopathology datasets.
-
Lightweight Backbone Networks Only Require Adaptive Lightweight Self-Attention Mechanisms
LOLViT, a GhostNet-based lightweight backbone using adaptive window attention, reports CNN-like CPU speed with MobileViT-level accuracy.
-
Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models
Vision networks whose bottlenecks avoid feature-map interference scale better than MobileNet-style designs at very low parameter counts, and the new NoDepth block demonstrates this on ImageNet.
-
VCMamba: Bridging Convolutions with Multi-Directional Mamba for Efficient Visual Representation
VCMamba reports that using convolutional feed-forward blocks for the first three stages followed by multi-directional Mamba blocks in the final stage yields 82.6% ImageNet-1K and 47.1 ADE20K mIoU at 31.5M parameters, ...
-
Foundation Models and Transformers for Anomaly Detection: A Survey
A taxonomy and literature review of Transformer-based visual anomaly detection, compromised by fabricated citations with dummy arXiv IDs.
-
LAID: Lightweight AI-Generated Image Detection in Spatial and Spectral Domains
A benchmark showing lightweight off-the-shelf models can exceed 99% accuracy on GenImage for AI-image detection, with spectral inputs and decision fusion, at a fraction of larger models' compute.
-
DeepTraverse: A Depth-First Search Inspired Network for Algorithmic Visual Understanding
DeepTraverse is a weight-tied residual network plus squeeze-and-excitation attention, framed as depth-first search, with claimed efficiency gains that rest on a questionable ImageNet subset comparison.
Discussion (0). Sign in to comment.