Pith. sign in

REVIEW 3 major objections 5 minor 85 references

Revisiting the Integration of Convolution and Attention for Vision Backbone

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that convolutions can handle fine-grained grid features while 64 semantic slots with multi-head self-attention are enough to match state-of-the-art vision backbones, and builds the GLNet family to show it.

desk verdict A credible hybrid backbone paper with a genuinely new parallel local-conv/global-slot design; the main claim about 64 slots is well-tested for ImageNet but under-tested for high-resolution dense tasks. read the letter →

arxiv 2411.14429 v1 pith:SCE7WBSZ submitted 2024-11-21 cs.CV cs.AI

Lei Zhu , Xinjiang Wang , Wayne Zhang , Rynson W. H. Lau This is my paper · ORCID
classification cs.CVcs.AI
keywords visionbackbonemulti-headself-attentionconvolutionsemanticslotssoftclusteringglobal-localfusionefficientattentionImageNet-1k
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes GLMix, a block that processes a fine feature grid with lightweight convolutions and a coarse set of 64 semantic slots with multi-head self-attention, bridged by learnable soft clustering and dispatching modules. The authors argue that this division of labor—convolutions extract per-pixel detail, attention models global relations—removes the quadratic-resolution cost of full attention without losing accuracy. The GLNet family built from these blocks reaches 82.5% to 85.0% ImageNet-1k top-1 accuracy and improves the accuracy-throughput frontier over recent backbones. The same advantages transfer to COCO object detection, instance segmentation, and ADE20K semantic segmentation, suggesting a general design principle for vision backbones.

What carries the argument

The central object is the GLMix block, which couples a convolutional local branch on the fine feature grid with a global branch that runs multi-head self-attention over M semantic slots (M = 64). Slots are initialized by average pooling, refined by one soft-clustering step: correspondence logits $A = \mathrm{CosineSimilarity}(S_{\mathrm{init}}, \bar{X})/\sigma$, refined slots $S = \mathrm{Softmax}(A)\bar{X}$, then dispatched back to the grid with $\mathrm{Softmax}(A^\top)S'$. This non-iterative, differentiable bridge keeps attention cost independent of input resolution and fuses local and global features via additive fusion.

What would settle it

Run GLNet on a high-resolution dataset where global context is spread across many small, spatially dispersed objects (e.g., satellite or whole-slide medical imagery) and compare accuracy at 64 slots versus 256 slots; if the larger slot count gives a substantial consistent gain beyond normal training noise, the claim that 64 slots suffice at all resolutions is falsified.

Watch

Extended reading notes

Core claim

The central claim is that multi-head self-attention does not need to operate on the full-resolution pixel or patch grid once convolutions have extracted local features; a fixed set of 64 semantic slots is sufficient to capture global context and match state-of-the-art backbones. The paper demonstrates this with the GLMix block, where image features are represented twice—as a regular grid and as a set of semantic slots—and the two representations are fused by a fully differentiable, one-step soft clustering and dispatching pair. Empirically, GLNet models achieve competitive or better accuracy than recent hybrid and pure-transformer backbones at lower FLOPs and higher throughput, and the emergent semantic grouping in the clustering module appears without dense supervision.

Load-bearing premise

The paper assumes that 64 semantic slots, initialized by average pooling and refined by one soft-clustering step, can carry all the global information a vision backbone needs at every resolution; if natural images are less redundant than that assumption, the slot bottleneck loses information the convolutional branch cannot replace.

Editorial extensions

If this is right

  • Attention cost no longer scales with input resolution; a backbone can include global attention at any resolution for the price of a fixed number of slots.
  • The accuracy-FLOPs frontier on ImageNet-1k moves upward, so similarly sized models can be either more accurate or faster than recent state-of-the-art backbones.
  • Dense prediction tasks inherit the same benefit, as shown by consistent gains on COCO detection and ADE20K segmentation with the same 64-slot configuration.
  • The soft clustering module produces semantically meaningful grouping maps with only image-level supervision, which could seed weakly-supervised segmentation approaches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editor's extension: a fixed slot count implies a strong redundancy prior; at higher resolutions or on dense, spatially dispersed scenes the sufficient number of slots may grow, and a dynamic or resolution-adaptive slot mechanism would be a natural next step.
  • The paper shows the clustering maps are meaningful but does not convert them into a segmentation method; a testable extension is to use the assignment maps as pseudo-labels for weakly-supervised semantic segmentation.
  • A domain-shift test would isolate the architectural claim from the redundancy assumption: on satellite or medical imagery, increasing slots well beyond 64 should not help if the 64-slot sufficiency claim is general.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes GLMix, a vision backbone building block that processes a fine-grained feature grid with lightweight convolutions and, in parallel, processes a coarse set of M semantic slots with multi-head self-attention. A pair of differentiable soft clustering and dispatching modules (Eqs. 1-4) bridges the grid and slot representations. From this block the authors build GLNet-STL and a scaled family GLNet-4G/9G/16G, reporting ImageNet-1k top-1 accuracies of 82.5/83.7/84.5/85.0 under standard supervised training, improved throughput relative to several recent hybrid backbones, and consistent gains on COCO object detection/instance segmentation and ADE20K semantic segmentation. Ablations on GLNet-STL isolate the contribution of the parallel local-global design, slot count, convolution kernel size, and clustering strategy.

Significance. If the results hold, the paper makes a useful conceptual contribution: global MHSA can be confined to a small fixed set of semantic slots while local convolutions handle fine detail, avoiding the quadratic cost of pixel-level attention. The experimental coverage is broad (classification, detection, instance segmentation, semantic segmentation, multiple model scales) and the ablation study in Sec. 4.5 is informative. The slot visualizations in Sec. 4.4 are a nice byproduct. The main caveats are that the "64 slots suffice" claim is not directly tested in high-resolution dense tasks, and several headline margins are small and reported without variance information. Both issues are addressable within the manuscript's scope.

major comments (3)
  1. [Sec. 4 (p. 7) and Table 6] The paper states that for dense prediction "we keep the number of semantic slots to 64… increasing the number does not help," but the only slot-count sweep in Table 6 is performed with GLNet-STL on ImageNet-1k at 224x224. In COCO (800x1333) and ADE20K (512x2048), the early-stage grids are much larger, so the 64-slot bottleneck is far more aggressive than in the classification setting. The sufficiency of 64 slots at high resolution is therefore a load-bearing assumption for the paper's central claim rather than a demonstrated result. Please add a slot-count comparison for at least one detection and one segmentation setting (e.g., GLNet-4G with Mask R-CNN 1x and Semantic FPN), or restrict the claim to the resolution actually tested.
  2. [Table 3 and Figure 2] The accuracy margins over the strongest baselines are small: 84.5 vs 84.3 for GLNet-9G vs SMT-B, and 85.0 vs 84.9 for GLNet-16G vs MaxViT-B. The paper reports no error bars or multiple runs, and the checklist acknowledges this. Because the abstract and Sec. 4.1 claim a "clear margin" over prior state of the art, please report run-to-run variability or explicitly qualify the margins as being within typical single-run variation for this training recipe.
  3. [Figure 2 and Table 1] Throughput is central to the paper's efficiency claim, but only MaxViT-STL in Table 1 is explicitly marked as measured by the authors with timm modules. If the other throughput points in Figure 2 and Table 1 are taken from different papers, hardware, or frameworks, the Pareto-frontier comparison may not be apples-to-apples. Please specify the measurement source and protocol for every throughput value, or re-measure the baselines under the same conditions.
minor comments (5)
  1. [Abstract and Secs. 1-3] The acronym "MSHAs" appears repeatedly; it should be "MHSAs" (multi-head self-attentions).
  2. [Figure 2 caption] The caption says "Top-1 Acc vs. FLOPs," but the x-axis of the figure is throughput (im/s); please correct the caption to match the plot.
  3. [Table 5] The CSWin-T row cites reference [43] (SG-Former); it should cite [14] (CSWin Transformer).
  4. [Sec. 4.5, Table 6 discussion] The text says accuracy decreases to 82.3% with fewer slots "e.g., 9, 25, 36 or 49," but Table 6 reports 81.9% for 9 slots and 82.1% for 25 slots; the sentence should be rephrased to match the table.
  5. [Sec. 3.2, Eqs. (3)-(4)] The softmax axes in Eqs. (3) and (4) are only indicated in Figure 4; stating the axes explicitly in the text would remove ambiguity about row-wise vs column-wise normalization.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity: all central comparisons are against external benchmarks, and the only author-overlapping citations are non-load-bearing.

full rationale

The paper does not derive its performance from its own inputs. The GLMix mechanism is defined by Eqs. (1)-(4) — pooling-based slot initialization, cosine-similarity correspondence, one-step soft clustering, and dispatching — and the number of semantic slots is chosen by an explicit ablation in Table 6, not fitted to the target accuracies. All headline accuracy/throughput comparisons in Tables 3-5 and Figure 2 are against externally published models, so the central claim is self-contained. The only author-overlapping citations are BiFormer [70], used as related work, a comparison baseline, and a source of the Swin-Tiny-Layout protocol, and Dense Distinct Query [68], cited only for the minor hypothesis that too many similar slots hurt optimization; neither supports the load-bearing claim. The paper itself asserts that 64 slots suffice for dense tasks ('we keep the number of semantic slots to 64 ... increasing the number does not help') without reporting a dense-task slot-count ablation; this is an empirical validation gap and a possible overclaim, but not a circular derivation. No step can be exhibited where a prediction reduces by construction to a fitted input or to a self-citation chain.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central claim rests on domain assumptions about image redundancy and the sufficiency of a single clustering step. There are no physically invented entities. The main hand-chosen hyperparameter is the slot count, along with a learnable temperature and conv kernel size, all of which are ablated to some degree.

free parameters (3)
  • number of semantic slots (M) = 64
    Default global-branch width. Ablation (Table 6) shows 9 to 81 slots give 81.9% to 82.4% top-1, with 64 at 82.5%, so the central efficiency claim depends on this hand-tuned value.
  • cosine similarity temperature sigma = learned
    Learnable scale factor in Eq. 2 that smooths the correspondence logits; learned from data during training.
  • local branch conv kernel size = 5x5
    Default depth-wise conv kernel. Ablation shows 3x3 and 7x7 both give 82.4% (Table 6), so the claim is robust to this choice.
assumptions (3)
  • domain assumption Natural images contain heavy spatial redundancy, so a fixed small number of visual tokens can summarize an image regardless of resolution.
    Invoked in Sec. 3.1 to justify limiting MHSAs to 64 semantic slots. Supported by citations to Flamingo and mPLUG-Owl, not by an analysis in this paper.
  • domain assumption One step of soft clustering suffices to produce useful semantic slots.
    The 1-step update in Eq. 3 replaces iterative k-means. Justified empirically by ablations (Table 6) and visualizations (Fig. 5), with no formal guarantee.
  • domain assumption Convs act as high-pass filters and MHSAs act as low-pass filters, making them complementary.
    Cited from Park and Kim (ICLR 2022) in Sec. 1 and the related work discussion; motivates the parallel local-global design.
invented entities (1)
  • semantic slots independent evidence
    purpose: A coarse set representation (default 64 slots) that carries global context through self-attention, bridged to the pixel grid by soft clustering and dispatching.
    Visualized in Sec. 4.4, showing object-like groupings, and ablated in Table 6 (global branch removal drops accuracy to 78.0%). This is an architectural construct with empirical support, not an observed natural kind.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting the Integration of Convolution and Attention for Vision Backbone." pith.science (2026). https://pith.science/paper/SCE7WBSZ

@misc{pith2026241114429,
  author       = {Pith},
  title        = {Pith review of: Revisiting the Integration of Convolution and Attention for Vision Backbone},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SCE7WBSZ}},
  note         = {Machine review of arXiv:2411.14429}
}
read the original abstract

Convolutions (Convs) and multi-head self-attentions (MHSAs) are typically considered alternatives to each other for building vision backbones. Although some works try to integrate both, they apply the two operators simultaneously at the finest pixel granularity. With Convs responsible for per-pixel feature extraction already, the question is whether we still need to include the heavy MHSAs at such a fine-grained level. In fact, this is the root cause of the scalability issue w.r.t. the input resolution for vision transformers. To address this important problem, we propose in this work to use MSHAs and Convs in parallel \textbf{at different granularity levels} instead. Specifically, in each layer, we use two different ways to represent an image: a fine-grained regular grid and a coarse-grained set of semantic slots. We apply different operations to these two representations: Convs to the grid for local features, and MHSAs to the slots for global features. A pair of fully differentiable soft clustering and dispatching modules is introduced to bridge the grid and set representations, thus enabling local-global fusion. Through extensive experiments on various vision tasks, we empirically verify the potential of the proposed integration scheme, named \textit{GLMix}: by offloading the burden of fine-grained features to light-weight Convs, it is sufficient to use MHSAs in a few (e.g., 64) semantic slots to match the performance of recent state-of-the-art backbones, while being more efficient. Our visualization results also demonstrate that the soft clustering module produces a meaningful semantic grouping effect with only IN1k classification supervision, which may induce better interpretability and inspire new weakly-supervised semantic segmentation approaches. Code will be available at \url{https://github.com/rayleizhu/GLMix}.

Figures

Figures reproduced from arXiv: 2411.14429 by the authors.

Figure 1
Figure 1. Existing integration schemes, e.g., ACMix [40], apply MHSAs and Convs at the same granularity (top). In contrast, we affirm that by offloading the burden of extracting fine-grained features to lightweight Convs, MHSAs can be ag￾gressively applied to coarse semantic slots to make spatial mixing more efficient (bottom). 100 200 300 400 500 600 700 800 Throughputs@224x224 (im/s) 81.5 82.0 82.5 83.0 83.5 84.0 84.5 85.0 … view at source ↗
Figure 4
Figure 4. Structure of our GLMix block. At the core is a pair of conjugated soft clustering and [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 3
Figure 3. The semantic slots correspond to “soft” irregular semantic regions (left). Compared to us￾ing hard-divided regular patches (as adopted by plain ViTs [15, 46]) on the right, our formulation is closer to tokenization in NLP. We apply Convs to the grid representation to extract local features as they are lightweight and thus efficient in processing the fine-grained fea￾ture grid. To model global context, we apply MHSAs… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Visualization of semantic slots. For each sample, we show the input image (left), assignment [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Visualization of semantic slots of blocks at different depths. The setting is the same as [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Slot evolution over training epochs. The setting is the same as in Figure 5, except that [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Visualization of slots using learned parameters as clustering initialization. The setting is [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

85 extracted references · 51 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022

    2022

  2. [2]

    High-performance large-scale image recognition without normalization

    Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In International Conference on Machine Learning, pages 1059–1071. 10 PMLR, 2021

    2021

  3. [3]

    Regionvit: Regional-to-local attention for vision transformers

    Chun-Fu Chen, Rameswar Panda, and Quanfu Fan. Regionvit: Regional-to-local attention for vision transformers. In International Conference on Learning Representations. OpenReview.net, 2022. URL https://openreview.net/forum?id=T__V3uLix7V

    2022

  4. [4]

    Mmdetection: Open mmlab detection toolbox and benchmark

    Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv:1906.07155, 2019

    arXiv 1906

  5. [5]

    Mixformer: Mixing features across windows and dimensions

    Qiang Chen, Qiman Wu, Jian Wang, Qinghao Hu, Tao Hu, Errui Ding, Jian Cheng, and Jingdong Wang. Mixformer: Mixing features across windows and dimensions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5249–5259, 2022

    2022

  6. [6]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv:1904.10509, 2019

    arXiv 1904

  7. [7]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020

    arXiv 2009

  8. [8]

    Twins: Revisiting the design of spatial attention in vision transformers

    Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34:9355–9366, 2021

    2021

Show all 85 references
  1. [9]

    Conditional positional encodings for vision transformers

    Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. In International Conference on Learning Representations, 2023

    2023

  2. [10]

    Openmmlab semantic segmentation toolbox and benchmark

    MMSegmentation Contributors. Openmmlab semantic segmentation toolbox and benchmark. https: //github.com/open-mmlab/mmsegmentation, 2020

    2020

  3. [11]

    Randaugment: Practical automated data augmentation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition workshops, pages 702–703, 2020

    2020

  4. [12]

    Coatnet: Marrying convolution and attention for all data sizes

    Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in neural information processing systems, 34:3965–3977, 2021

    2021

  5. [13]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

    2009

  6. [14]

    Cswin transformer: A general vision transformer backbone with cross-shaped windows

    Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pa...

    2022

  7. [15]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  8. [16]

    Study on density peaks clustering based on k-nearest neighbors and principal component analysis

    Mingjing Du, Shifei Ding, and Hongjie Jia. Study on density peaks clustering based on k-nearest neighbors and principal component analysis. Knowledge-Based Systems, 99:135–145, 2016

    2016

  9. [17]

    Paca-vit: learning patch-to-cluster attention in vision transformers

    Ryan Grainger, Thomas Paniagua, Xi Song, Naresh Cuntoor, Mun Wai Lee, and Tianfu Wu. Paca-vit: learning patch-to-cluster attention in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18568–18578, 2023

    2023

  10. [18]

    Cmt: Convolutional neural networks meet vision transformers

    Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12175–12185, 2022

    2022

  11. [19]

    Flatten transformer: Vision transformer using focused linear attention

    Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5961–5971, 2023. 11

    2023

  12. [20]

    Neighborhood attention transformer

    Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6185– 6194, 2023

    2023

  13. [21]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pages 2961–2969, 2017

    2017

  14. [22]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022

    2022

  15. [23]

    Squeeze-and-excitation networks

    Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018

    2018

  16. [24]

    Deep networks with stochastic depth

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. In Proceedings of the European conference on computer vision, pages 646–661, 2016

    2016

  17. [25]

    How much position information do convolutional neural networks encode? arXiv preprint arXiv:2001.08248, 2020

    Md Amirul Islam, Sen Jia, and Neil DB Bruce. How much position information do convolutional neural networks encode? arXiv preprint arXiv:2001.08248, 2020

    2001 arXiv

  18. [26]

    All tokens matter: Token labeling for training better vision transformers

    Zi-Hang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. Advances in Neural Information Processing Systems, 34:18590–18602, 2021

    2021

  19. [27]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR, 2020

    2020

  20. [28]

    Panoptic feature pyramid networks

    Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6399–6408, 2019

    2019

  21. [29]

    Uniformer: Unifying convolution and self-attention for visual recognition

    Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

    2023

  22. [30]

    Clusterfomer: clustering as a universal visual learner

    James Liang, Yiming Cui, Qifan Wang, Tong Geng, Wenguan Wang, and Dongfang Liu. Clusterfomer: clustering as a universal visual learner. Advances in neural information processing systems, 36, 2024

    2024

  23. [31]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European conference on computer vision, pages 740–755. Springer, 2014

    2014

  24. [32]

    Feature pyramid networks for object detection

    Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017

    2017

  25. [33]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2980–2988, 2017

    2017

  26. [34]

    Scale-aware modulation meet transformer

    Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, and Lianwen Jin. Scale-aware modulation meet transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6015–6026, 2023

    2023

  27. [35]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021

    2021

  28. [36]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022

    2022

  29. [37]

    Stochastic gradient descent with warm restarts

    I Loshchilov and F Hutter. Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, pages 1–16, 2017

    2017

  30. [38]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. 12

    2019

  31. [39]

    Understanding the effective receptive field in deep convolutional neural networks

    Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Advances in neural information processing systems, 29, 2016

    2016

  32. [40]

    On the integration of self-attention and convolution

    Xuran Pan, Chunjiang Ge, Rui Lu, Shiji Song, Guanfu Chen, Zeyi Huang, and Gao Huang. On the integration of self-attention and convolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 815–825, 2022

    2022

  33. [41]

    How do vision transformers work? In International Conference on Learning Representations

    Namuk Park and Songkuk Kim. How do vision transformers work? In International Conference on Learning Representations. OpenReview.net, 2022. URL https://openreview.net/forum?id= D78Go4hVcxO

    2022

  34. [42]

    From sparse to soft mixtures of experts

    Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.00951, 2023

    2023 arXiv

  35. [43]

    Sg-former: Self-guided transformer with evolving token reallocation

    Sucheng Ren, Xingyi Yang, Songhua Liu, and Xinchao Wang. Sg-former: Self-guided transformer with evolving token reallocation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6003–6014, 2023

    2023

  36. [44]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  37. [45]

    Rethinking the inception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2818–2826, 2016

    2016

  38. [46]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021

    2021

  39. [47]

    Maxvit: Multi-axis vision transformer

    Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In Proceedings of the European conference on computer vision, 2022

    2022

  40. [48]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  41. [49]

    Fast transformers with clustered attention

    Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. Advances in Neural Information Processing Systems, 33:21665–21674, 2020

    2020

  42. [50]

    Linformer: Self-attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv:2006.04768, 2020

    2006 arXiv

  43. [51]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pa...

    2021

  44. [52]

    Internimage: Exploring large-scale vision foundation models with deformable convolutions

    Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision a...

    2023

  45. [53]

    Crossformer: A versatile vision transformer hinging on cross-scale attention

    Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. Crossformer: A versatile vision transformer hinging on cross-scale attention. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=_PHymLIxuI

    2022

  46. [54]

    Pytorch image models

    Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models , 2019

    2019

  47. [55]

    Cvt: Introducing convolutions to vision transformers

    Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 22–31, 2021

    2021

  48. [56]

    Vision transformer with deformable attention

    Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4794–4803, 2022

    2022

  49. [57]

    Unified perceptual parsing for scene understanding

    Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision, pages 418–434, 2018. 13

    2018

  50. [58]

    Clustr: Exploring efficient self-attention via clustering for vision transformers

    Yutong Xie, Jianpeng Zhang, Yong Xia, Anton van den Hengel, and Qi Wu. Clustr: Exploring efficient self-attention via clustering for vision transformers. arXiv preprint arXiv:2208.13138, 2022

    2022 arXiv

  51. [59]

    Moat: Alternating mobile convolution and attention brings strong vision models

    Chenglin Yang, Siyuan Qiao, Qihang Yu, Xiaoding Yuan, Yukun Zhu, Alan Yuille, Hartwig Adam, and Liang-Chieh Chen. Moat: Alternating mobile convolution and attention brings strong vision models. In International Conference on Learning Representations, 2023

    2023

  52. [60]

    Scalablevit: Rethinking the context-oriented generalization of vision transformer

    Rui Yang, Hailong Ma, Jie Wu, Yansong Tang, Xuefeng Xiao, Min Zheng, and Xiu Li. Scalablevit: Rethinking the context-oriented generalization of vision transformer. InEuropean Conference on Computer Vision, pages 480–496. Springer, 2022

    2022

  53. [61]

    Wave-vit: Unifying wavelet and transformers for visual representation learning

    Ting Yao, Yingwei Pan, Yehao Li, Chong-Wah Ngo, and Tao Mei. Wave-vit: Unifying wavelet and transformers for visual representation learning. In Proceedings of the European conference on computer vision, pages 328–345. Springer, 2022

    2022

  54. [62]

    Dual vision transformer

    Ting Yao, Yehao Li, Yingwei Pan, Yu Wang, Xiao-Ping Zhang, and Tao Mei. Dual vision transformer. IEEE transactions on pattern analysis and machine intelligence, 2023

    2023

  55. [63]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023

    2023 arXiv

  56. [64]

    Metaformer is actually what you need for vision

    Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10819–10829, 2022

    2022

  57. [65]

    Cutmix: Regularization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023–6032, 2019

    2019

  58. [66]

    Not all tokens are equal: Human-centric visual analysis via token clustering transformer

    Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang, and Xiaogang Wang. Not all tokens are equal: Human-centric visual analysis via token clustering transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11101–1...

    2022

  59. [67]

    Dauphin, and David Lopez-Paz

    Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations , 2018. URL https:// openreview.net/forum?id=r1Ddp1-Rb

    2018

  60. [68]

    Dense distinct query for end-to-end object detection

    Shilong Zhang, Xinjiang Wang, Jiaqi Wang, Jiangmiao Pang, Chengqi Lyu, Wenwei Zhang, Ping Luo, and Kai Chen. Dense distinct query for end-to-end object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7329–7338, 2023

    2023

  61. [69]

    Semantic understanding of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302–321, 2019

    2019

  62. [70]

    Biformer: Vision transformer with bi-level routing attention

    Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, and Rynson WH Lau. Biformer: Vision transformer with bi-level routing attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10323–10333, 2023. 14 A Effect of Advanced Architectur...

    2023

  63. [71]

    The contributions are summarized at the end of the Introduction (Sec

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The scope is stated at the beginning of the abstract. The contributions are summarized at the end of the Introduction ...

  64. [72]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We mentioned the limitations at the end of the conclusion (Sec. 5). Guidelines: • The answer NA means that the paper has no limitation while the answ...

  65. [73]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: The paper does not include theoretical results. Guidelines: • The answer NA means that the pa...

  66. [74]

    Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  67. [75]

    We will release the code upon the acceptance of this paper

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: All datasets used in this pape...

  68. [76]

    Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Experimental settings are provi...

  69. [77]

    The experimental results are not sensitive to random initialization

    Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Datasets such as ImageNet-1K, COCO, and ADE20K are ...

  70. [78]

    Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments? Answer: [Yes] Justification: The information on the ...

  71. [79]

    We only use existing and publicly available datasets for evaluations

    Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: This paper does not involve human subjects or participants. We only use existin...

  72. [80]

    It is too broad to discuss the societal impacts of such a general topic

    Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: This paper focuses on designing general vision backbones. It is too broad to discuss the societal impacts ...

  73. [81]

    It poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper focu...

  74. [82]

    These works are properly cited

    Licenses for existing assets 22 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We only use e...

  75. [83]

    Guidelines: • The answer NA means that the paper does not release new assets

    New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Our code and documents will be released at https://github.com/rayleizhu/ GLMix. Guidelines: • The answer NA means th...

  76. [84]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  77. [85]

    Guidelines: 23 • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approva...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.