Pith. sign in

REVIEW 15 cited by

Focal Self-attention for Local-Global Interactions in Vision Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2107.00641 v1 pith:ZEM5FB3P submitted 2021-07-01 cs.CV cs.AIcs.LG

Jianwei Yang , Chunyuan Li , Pengchuan Zhang , Xiyang Dai , Bin Xiao , Lu Yuan , Jianfeng Gao This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords focalvisiontransformerself-attentiontransformersdetectionobjecttasks
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Recently, Vision Transformer and its variants have shown great promise on various computer vision tasks. The ability of capturing short- and long-range visual dependencies through self-attention is arguably the main source for the success. But it also brings challenges due to quadratic computational overhead, especially for the high-resolution vision tasks (e.g., object detection). In this paper, we present focal self-attention, a new mechanism that incorporates both fine-grained local and coarse-grained global interactions. Using this new mechanism, each token attends the closest surrounding tokens at fine granularity but the tokens far away at coarse granularity, and thus can capture both short- and long-range visual dependencies efficiently and effectively. With focal self-attention, we propose a new variant of Vision Transformer models, called Focal Transformer, which achieves superior performance over the state-of-the-art vision Transformers on a range of public image classification and object detection benchmarks. In particular, our Focal Transformer models with a moderate size of 51.1M and a larger size of 89.8M achieve 83.5 and 83.8 Top-1 accuracy, respectively, on ImageNet classification at 224x224 resolution. Using Focal Transformers as the backbones, we obtain consistent and substantial improvements over the current state-of-the-art Swin Transformers for 6 different object detection methods trained with standard 1x and 3x schedules. Our largest Focal Transformer yields 58.7/58.9 box mAPs and 50.9/51.3 mask mAPs on COCO mini-val/test-dev, and 55.4 mIoU on ADE20K for semantic segmentation, creating new SoTA on three of the most challenging computer vision tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Partial Ring Scan: Revisiting Scan Order in Vision State Space Models

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Ring-based scanning with selective channel routing improves accuracy, speed, and rotation robustness of vision state-space models.

  2. Vision encoders should be image size agnostic and task driven

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A position paper proposing task-driven, image-size-agnostic vision encoders, demonstrated by an iterative transformer with top-down multi-zoom patches and a GRPO-trained gaze policy on ImageNet.

  3. Resolving Token-Space Gradient Conflicts: Token Space Manipulation for Transformer-Based Multi-Task Learning

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A token-space SVD-based method that separately resolves gradient conflicts in the range and null spaces of transformer tokens improves multi-task learning performance with minimal extra parameters.

  4. ViT-Split: Unleashing the Power of Vision Foundation Models via Efficient Splitting Heads

    cs.CV 2025-06 conditional novelty 6.0 of 10

    ViT-Split freezes a vision foundation model and adds a copied task head plus a multi-scale prior head, matching or beating prior adapters with fewer parameters and up to 4x faster training.

  5. Describe Anything: Detailed Localized Image and Video Captioning

    cs.CV 2025-04 conditional novelty 6.0 of 10

    DAM achieves state-of-the-art detailed localized captioning on images and videos via focal prompting and a localized vision backbone trained on VLM-expanded segmentation data and self-labeled web images.

  6. Can We Get Rid of Handcrafted Feature Extractors? SparseViT: Nonsemantics-Centered, Parameter-Efficient Image Manipulation Localization through Spare-Coding Transformer

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A sparse-attention vision transformer, trained without handcrafted feature extractors, reports state-of-the-art image manipulation localization and lower compute on four public benchmarks.

  7. MMA-Former: Multi-Window Mixture-of-Head Attention Transformer for Adaptive PNI Prediction in 3D MRI

    cs.CV 2026-07 conditional novelty 5.5 of 10

    MMA-Former with window-level Mixture-of-Head attention reaches AUC 0.752 for PNI prediction on 168 cropped 3D MRI scans, beating 3D CNN and Swin baselines.

  8. Quantum-Enhanced Optimization by Warm Starts

    quant-ph 2025-08 unverdicted novelty 5.0 of 10

    Quantum-generated QAOA samples, used as warm starts for classical heuristics, yield faster Max-Cut and MIS solving, including on quantum hardware.

  9. RelayFormer: A Unified Local-Global Attention Framework for Scalable Image and Video Manipulation Localization

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    RelayFormer splits inputs into fixed-size pieces and uses relay tokens to share global context, aiming for unified image and video tamper localization at any resolution.

  10. A Retrospective Systematic Study on Hierarchical Sparse Query Transformer-assisted Ultrasound Screening for Early Hepatocellular Carcinoma

    cs.CV 2025-02 conditional novelty 5.0 of 10

    A hybrid CNN-transformer with mixture-of-experts (HSQformer) yields 95.38% AUC for HCC screening on a multi-center ultrasound test set, beating several baselines and matching senior radiologists.

  11. STARFormer: A Novel Spatio-Temporal Aggregation Reorganization Transformer of FMRI for Brain Disorder Diagnosis

    eess.IV 2024-12 reject novelty 5.0 of 10

    STARFormer uses eigenvector centrality to reorder fMRI brain regions and a variable-window transformer to reach reported state-of-the-art accuracy on ASD and ADHD, but its cross-validation protocol may be contaminated...

  12. Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training

    cs.CV 2024-11 conditional novelty 5.0 of 10

    A spike-driven Transformer trained with integer activations reaches 86.2% top-1 on ImageNet, the highest reported accuracy for a directly trained spiking network at this scale.

  13. AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformer

    cs.CV 2025-05 conditional novelty 4.0 of 10

    A vision transformer reduces self-attention complexity from O(n^2) to O(mn) by using m learnable anchor tokens and a two-step Markov transition between anchors and tokens.

  14. Local-Global Attention: An Adaptive Mechanism for Multi-Scale Feature Integration

    cs.CV 2024-11 reject novelty 4.0 of 10

    Local-Global Attention, a module fusing multi-scale local and global attention branches with learned weights, reports marginal accuracy gains (0.1 to 0.9 mAP points) over standard attention mechanisms on several detec...

  15. MVFormer: Diversifying Feature Normalization and Token Mixing for Efficient Vision Transformers

    cs.CV 2024-11 conditional novelty 3.0 of 10

    MVFormer combines a weighted blend of three normalizations with a three-branch multiscale convolutional token mixer, achieving modest top-1 accuracy gains on ImageNet-1K and downstream vision tasks.

Pith tools