The reviewed record of science sign in
Pith

arxiv: 2106.04803 · v2 · pith:Y4AH3AU2 · submitted 2021-06-09 · cs.CV · cs.LG

CoAtNet: Marrying Convolution and Attention for All Data Sizes

Zihang Dai , Hanxiao Liu , Quoc V. Le , Mingxing Tan This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:Y4AH3AU2record.jsonopen to challenge →

classification cs.CV cs.LG
keywords coatnetaccuracyachievesattentionconvolutiondatastate-of-the-arttop-1
0
0 comments X
read the original abstract

Transformers have attracted increasing interests in computer vision, but they still fall behind state-of-the-art convolutional networks. In this work, we show that while Transformers tend to have larger model capacity, their generalization can be worse than convolutional networks due to the lack of the right inductive bias. To effectively combine the strengths from both architectures, we present CoAtNets(pronounced "coat" nets), a family of hybrid models built from two key insights: (1) depthwise Convolution and self-Attention can be naturally unified via simple relative attention; (2) vertically stacking convolution layers and attention layers in a principled way is surprisingly effective in improving generalization, capacity and efficiency. Experiments show that our CoAtNets achieve state-of-the-art performance under different resource constraints across various datasets: Without extra data, CoAtNet achieves 86.0% ImageNet top-1 accuracy; When pre-trained with 13M images from ImageNet-21K, our CoAtNet achieves 88.56% top-1 accuracy, matching ViT-huge pre-trained with 300M images from JFT-300M while using 23x less data; Notably, when we further scale up CoAtNet with JFT-3B, it achieves 90.88% top-1 accuracy on ImageNet, establishing a new state-of-the-art result.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ClawEnvKit: Automatic Environment Generation for Claw-Like Agents

    cs.AI 2026-04 unverdicted novelty 6.0

    ClawEnvKit automates generation of diverse verified environments for claw-like agents from natural language, producing the Auto-ClawEval benchmark of 1,040 environments that matches human-curated quality at 13,800x lo...

  2. Florence: A New Foundation Model for Computer Vision

    cs.CV 2021-11 unverdicted novelty 6.0

    Florence is a new vision foundation model that learns universal visual-language representations from web-scale data and reports state-of-the-art results on 44 benchmarks including 83.74% zero-shot ImageNet top-1 accuracy.

  3. MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer

    cs.CV 2021-10 unverdicted novelty 6.0

    MobileViT is a lightweight vision transformer that reports 78.4% top-1 accuracy on ImageNet-1k with ~6M parameters, outperforming MobileNetv3 by 3.2% and DeIT by 6.2% at similar size, plus gains on MS-COCO detection.

  4. Advancing Vision Transformer with Enhanced Spatial Priors

    cs.CV 2026-04 unverdicted novelty 4.0

    EVT improves Vision Transformers by using Euclidean distance decay for spatial priors and simpler grouping, achieving 86.6% top-1 accuracy on ImageNet-1k.

  5. ClawEnvKit: Automatic Environment Generation for Claw-Like Agents

    cs.AI 2026-04 conditional novelty 4.0

    EVT improves the RMT backbone by using Euclidean-distance attention decay and 1D token grouping, achieving 86.6% top-1 on ImageNet-1K at 384×384 resolution.