Pith. sign in

REVIEW 3 major objections 6 minor 80 references

ContextFormer: Redefining Efficiency in Semantic Segmentation

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ContextFormer is a hybrid CNN-Transformer segmentation network built on the claim that the bottleneck—not the encoder—is where efficiency and accuracy are lost, and that a branched depthwise-convolution plus slim self-attention block…

desk verdict A modest low-FLOP segmentation architecture with a clean ablation, buried under a state-of-the-art claim that its own Table 2 disproves. read the letter →

arxiv 2501.19255 v2 pith:TO632ARY submitted 2025-01-31 cs.CV

Mian Muhammad Naeem Abid , Nancy Mehta , Zongwei Wu , Radu Timofte This is my paper · ORCID
classification cs.CV
keywords semanticsegmentationreal-timehybridCNN-Transformerbottleneckdesigntokenpyramidlightweightattentiondepthwiseconvolutionsedgemaps
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ContextFormer is a semantic segmentation architecture built around a simple claim: the bottleneck—the stage between the encoder and the segmentation head—is where compute is most wasted, and a hybrid block there can keep global context without paying quadratic attention costs. The paper proposes three coordinated modules—a Token Pyramid Extraction Module that creates multi-scale tokens, a Trans-BDC block that runs branched depthwise convolutions and a lightweight self-attention branch in parallel, and a Feature Merging Module that gates global features into local ones. The authors report 35.0% mIoU on ADE20K with 0.6 GFLOPs, 1.7M parameters, and 15 ms latency, and similar efficiency-dominated trade-offs on Pascal Context, CityScapes, and COCO-Stuff, plus a transfer to object detection. If the measurements hold, the work would mean that near-real-time mobile segmentation no longer requires choosing between local inductive bias and global context.

What carries the argument

The load-bearing mechanism is the Trans-BDC block, a parallel two-branch bottleneck that operates on heavily pooled tokens: a Branched Depthwise Convolution branch (three depthwise paths—3x3, 1x1, and 3x3 depthwise separable—with channel attention) and a lightweight self-attention branch with 1x1-convolution MLPs and low-dimensional Q/K/V projections. Token pooling before the block is what makes the global branch cheap, and the Feature Merging Module (a 1x1-convolution gating layer with a sigmoid, plus a residual sum) is what recombines the local pyramid features with the global semantics. The Token Pyramid Extraction Module supplies the multi-scale inputs by concatenating image intensity, gradient magnitude, and edge maps through MobileNetV2-style blocks. Together the three modules realize the paper's argument that global context can be obtained from small tokens while local detail is handled by inexpensive convolutions.

What would settle it

Run ContextFormer, SeaFormer-T, and TopFormer-T on the same GPU with identical batch size, input resolution, and inference settings, and compare mIoU on ADE20K validation; the central claim fails if ContextFormer's latency is not 15 ms or if SeaFormer-T is faster at equal or better mIoU.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that a carefully arranged hybrid bottleneck can outperform or match much larger mobile segmentation models at a fraction of the cost. The TPEM concatenates RGB, gradient-magnitude, and edge channels, runs them through stacked MobileNetV2-style inverted residual blocks, and pools the multi-scale tokens down to 1/64 of the input resolution before any attention is applied. The Trans-BDC block then fuses local features from three parallel depthwise convolutions with global features from a slim self-attention branch (1x1-convolution MLPs, 16/16/32-dim Q/K/V, batch norm, ReLU6), and the FMM gates and sums the two streams before upsampling. Across ADE20K, Pascal Context, CityScapes, and COCO-Stuff, the paper reports mIoU scores that are competitive with or above models in the same sub-1-GFLOP class, such as TopFormer and SeaFormer, and large compute savings relative to heavier CNN and transformer baselines. The paper also reports that the same backbone, used as a RetinaNet feature extractor on COCO, reaches 31.6 mAP, matching SeaFormer-T at identical FLOP and parameter counts.

Load-bearing premise

The entire efficiency advantage rests on the assumption that the latency and FLOP numbers for ContextFormer and its baselines were measured under the same conditions; the paper does not specify the hardware, batch size, or measurement method behind the 15 ms figure.

Editorial extensions

If this is right

  • On ADE20K, ContextFormer reports 35.0% mIoU at 0.6 GFLOPs and 15 ms, within 2.4 points of SegFormer's 37.4% at 8.4 GFLOPs, suggesting that global context can be retained at a small fraction of the usual attention budget.
  • Adding Sobel gradient magnitude and edge maps as extra input channels raises mIoU by 0.2% at no FLOP cost, and the paper states the edge input can be swapped to another detector such as Canny in a zero-shot way.
  • On CityScapes, the model reports 68.2% mIoU at 1.2 GFLOPs, 1.7 points above TopFormer at the same FLOP budget and within 4.5 points of L-ASPP at 12.6 GFLOPs.
  • On COCO-Stuff and Pascal Context, the reported mIoU slightly exceeds SeaFormer at equal or lower GFLOPs, indicating the design transfers across scene classes and dataset scales.
  • When reused as a RetinaNet backbone for COCO object detection, ContextFormer reaches 31.6 mAP at 160 GFLOPs and 10.9M parameters, matching SeaFormer-T, which supports the claim that the backbone generalizes beyond segmentation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The comparison tables imply that the headline claim is best understood as a claim about the sub-1-GFLOP mobile regime, since the paper's own Table 2 lists recent efficient models reaching 39.6-43.6 mIoU at 4-16 GFLOPs.
  • A transferable recipe can be extracted from the design: push global attention to the smallest remaining token resolution, spend the saved compute on parallel local convolutions, and feed cheap geometric cues such as gradients and edges as extra input channels.
  • The stated dependence on ImageNet pretraining suggests the reported accuracy is partly a property of the initialization; a from-scratch training study would reveal how much of the mIoU comes from the architecture alone.
  • A standardized latency measurement on a single device across all baselines would settle whether the reported 15 ms is an architectural advantage or a measurement artifact, since SeaFormer also reports 15 ms.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ContextFormer, a hybrid CNN-Transformer architecture for real-time semantic segmentation, with three modules: a Token Pyramid Extraction Module (TPEM), a Transformer and Branched Depthwise Conv (Trans-BDC) block, and a Feature Merging Module (FMM). The authors report experiments on ADE20K, PASCAL Context, CityScapes, and COCO-Stuff, with the best ADE20K model reaching 35.0 mIoU at 0.6 GFLOPs and 1.7M parameters, plus an object detection generalization study on COCO. The abstract and introduction claim that ContextFormer significantly outperforms existing models and achieves state-of-the-art mIoU scores.

Significance. If the stated accuracy-efficiency trade-off were correct, ContextFormer would be a meaningful contribution to lightweight semantic segmentation. The paper's modular design is reasonable engineering, and the authors provide ImageNet pretraining details and a supplementary architecture table, which are useful. However, the central claim of state-of-the-art accuracy is contradicted by the paper's own comparison tables: on ADE20K, ContextFormer achieves 35.0 mIoU while CGRSeg, MegaSeg, U-MixFormer, SegNeXt, and PEM reach 43.6, 42.4, 41.2, 41.1, and 39.6 mIoU respectively. The manuscript provides no theoretical analysis, no statistical significance testing, and no code at review time, so its contribution rests entirely on the empirical tables, which do not support the headline claim.

major comments (3)
  1. [Abstract, §1, and Table 2] The central claim that ContextFormer "significantly outperforms existing models, achieving state-of-the-art mIoU scores" is contradicted by the paper's own Table 2. On ADE20K, ContextFormer(GM E) reaches 35.0 mIoU at 0.6 GFLOPs, while the listed recent efficient models CGRSeg (43.6 mIoU), MegaSeg (42.4), U-MixFormer (41.2), SegNeXt (41.1), and PEM (39.6) all substantially exceed it, including at higher but still efficient FLOP budgets. Even the same-FLOP comparison against SeaFormer is a 0.3 mIoU difference (35.0 vs. 34.7), which does not constitute a "substantial margin." The paper needs to remove the state-of-the-art claim or provide a clearly defined comparison class in which the claim is actually true, but as written this is an internal inconsistency between the headline and the reported data.
  2. [§4.1 and Table 1] The efficiency claims rest on latency numbers that are not reproducible or comparable across methods. Section 4.1 reports latency in milliseconds but gives no hardware, batch size, inference framework, or measurement protocol, and Table 1 mixes input resolutions by marking models trained at 448×448 with an asterisk without clarifying how latency was measured for those entries. Figure 1 uses these latency values to make the central efficiency argument. Without a documented protocol, the claim that ContextFormer delivers 15 ms latency against the listed baselines cannot be verified, and the "redefining efficiency" framing is not supported. The authors should specify the exact measurement conditions and, ideally, report latency on a common benchmark with a fixed protocol.
  3. [§4.3, Table 2, §4.4–4.6] The paper claims "significant" and "substantial" outperformance over existing models, but the observed margins over the closest same-cost baselines are small: on ADE20K, 0.3 mIoU over SeaFormer; on PASCAL Context, 0.36 mIoU (mIoU59) and 0.51 mIoU (mIoU60) over SeaFormer; on COCO-Stuff, 0.02 mIoU over SeaFormer. No error bars, multiple seeds, or statistical significance tests are reported, so these differences are within typical run-to-run variance for semantic segmentation training. The claim of significant improvement is therefore not established by the evidence presented.
minor comments (6)
  1. [§4.3.2, Table 3] The text says the model demonstrates "a substantial 32.7% mIoU improvement when utilizing ViT alone," but Table 3 reports 32.7 mIoU, not a 32.7% improvement. The actual absolute gain over the CNN-only baseline is 2.3 mIoU points (from 32.7 to 35.0 with the full model), and the sentence should be corrected to avoid misreporting the ablation results.
  2. [Tables 1, 2, 4] The abbreviation "GME" (or "GM E") is used in tables and figure captions but is not defined in the main text. The reader must infer that it refers to gradient magnitude and edge maps; the acronym should be introduced explicitly.
  3. [Reference [30] and text] Reference [30] is titled "MetaSeg: Metaformer-based global contexts-aware network for efficient semantic segmentation," but the text and Table 2 call the method "MegaSeg." Please harmonize the name across the manuscript.
  4. [Throughout] The dataset name "CityScapes" is misspelled; the correct name is "Cityscapes." The same typo appears in multiple section headings and table captions.
  5. [§3.1] The phrase "to 1/64×64 of the original input size" is unclear. It should state the target resolution explicitly, e.g., "to 1/64 of the original input resolution (8×8 for a 512×512 input)".
  6. [Table 1 footnote] The footnote states that a batch size of 16 is used for ContextFormer, SegFormer, TopFormer, and SeaFormer, while a batch size of 32 is used for CNN-based models. Batch size affects batch normalization statistics and training dynamics, so this asymmetry should be justified as fair for the comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark results are direct empirical measurements, and the internal inconsistency between the SOTA claim and Table 2 is a correctness issue, not a circularity issue.

full rationale

The paper contains no derivation chain whose output is defined by its inputs. ContextFormer's mIoU, GFLOPs, parameter, and latency numbers are direct measurements on external benchmarks (ADE20K, Pascal Context, CityScapes, COCO-Stuff, and COCO detection); no fitted parameter is renamed as a prediction, and no equation equates the claimed result with a training input. The only dependence on the authors' own work is ImageNet-1K pretraining, which is a standard upstream initialization and is acknowledged in the supplementary limitations section as a dependency rather than smuggled in as evidence. The architectural equations (TPEM concatenation, Trans-BDC residual fusion, FMM gated summation) define a network structure, not a predicted quantity that reduces to its own inputs. The abstract's 'state-of-the-art mIoU' claim is contradicted by the paper's own Table 2, where ContextFormer reaches 35.0 mIoU while CGRSeg reaches 43.6, MegaSeg 42.4, U-MixFormer 41.2, SegNeXt 41.1, and PEM 39.6; however, that is an internal factual overclaim, which the instructions treat as a correctness risk rather than circularity. Accordingly, the honest circularity finding is 0 with no circular steps identified.

Assumptions & free parameters 5 free parameters · 4 assumptions · 3 invented entities

The central empirical claim rests on standard benchmark datasets, ImageNet pretraining, and a set of hand-chosen architectural hyperparameters such as block counts, channel widths, attention dimensions, and pooling resolutions. No theory or fitted physical constant is involved. The three modules are proposed components validated only inside the paper, and FMM is not separately ablated. No invented physical entities are proposed. The largest ledger item is the set of architecture choices that determine the reported 0.6 GFLOPs and 35.0 mIoU.

free parameters (5)
  • Number of Trans-BDC blocks and attention heads = N=4, H=4
    Chosen by hand in Supplementary Table B to balance accuracy and FLOPs; no sensitivity sweep is reported.
  • Q/K/V projection dimensions = 16, 16, 32
    Set in Section 3.2 to reduce memory footprint; the paper does not analyze how this choice affects accuracy.
  • Token pooling target resolution = 1/64 x 1/64 of input (8x8 at 512x512)
    Defined in Sections 3.1 and 3.2 and Supplementary Table B; controls both FLOPs and receptive field, with no resolution sweep.
  • FFN expansion factor = 2
    Supplementary Section 3 states an expansion factor of two to minimize complexity; selected by hand.
  • TPEM channel widths and block counts = 16, 32, 64, 96 and varied block counts
    Given in Supplementary Table B; an architecture choice without reported search or ablation.
assumptions (4)
  • domain assumption ImageNet-1K pretraining is required for the model to reach the reported performance
    Supplementary Section 5 states that without ImageNet pretraining, performance declines significantly. The benchmark numbers in the paper assume this pretraining step.
  • domain assumption mIoU on the four chosen benchmarks is a valid measure of segmentation quality and comparison protocol is fair
    Section 4.1 defines mIoU as the metric. Comparability across papers is assumed even though training settings, hardware, and augmentation pipelines are not fully controlled.
  • standard math The MobileNetV2 blocks, depthwise separable convolutions, and self-attention components behave as documented in the cited literature
    The method section builds on standard blocks without proving or re-validating them; this is accepted background in the field.
  • domain assumption Batch size differences between ContextFormer (16) and CNN baselines (32) do not materially affect the comparison
    Section 4.2 states that batch size 16 is used for ContextFormer and batch size 32 for CNN-based models. No experiment quantifies the effect of this difference on mIoU.
invented entities (3)
  • Token Pyramid Extraction Module (TPEM)
    purpose: Produces multi-scale token representations from RGB, gradient magnitude, and edge features using MobileNetV2 blocks
    Evaluated only within this paper's benchmarks and ablations; no external verification exists outside the paper.
  • Trans-BDC block
    purpose: Combines branched depthwise convolutions with lightweight self-attention to capture local and global context
    Supported only by the paper's internal ablations in Table 3; there is no independent replication.
  • Feature Merging Module (FMM)
    purpose: Fuses local TPEM features with global Trans-BDC features using a gated shifted mechanism
    FMM is listed as a key contribution, but no ablation removes it, so its individual contribution is not quantified even within the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ContextFormer: Redefining Efficiency in Semantic Segmentation." pith.science (2026). https://pith.science/paper/TO632ARY

@misc{pith2026250119255,
  author       = {Pith},
  title        = {Pith review of: ContextFormer: Redefining Efficiency in Semantic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TO632ARY}},
  note         = {Machine review of arXiv:2501.19255}
}
read the original abstract

Semantic segmentation assigns labels to pixels in images, a critical yet challenging task in computer vision. Convolutional methods, although capturing local dependencies well, struggle with long-range relationships. Vision Transformers (ViTs) excel in global context capture but are hindered by high computational demands, especially for high-resolution inputs. Most research optimizes the encoder architecture, leaving the bottleneck underexplored - a key area for enhancing performance and efficiency. We propose ContextFormer, a hybrid framework leveraging the strengths of CNNs and ViTs in the bottleneck to balance efficiency, accuracy, and robustness for real-time semantic segmentation. The framework's efficiency is driven by three synergistic modules: the Token Pyramid Extraction Module (TPEM) for hierarchical multi-scale representation, the Transformer and Branched DepthwiseConv (Trans-BDC) block for dynamic scale-aware feature modeling, and the Feature Merging Module (FMM) for robust integration with enhanced spatial and contextual consistency. Extensive experiments on ADE20K, Pascal Context, CityScapes, and COCO-Stuff datasets show ContextFormer significantly outperforms existing models, achieving state-of-the-art mIoU scores, setting a new benchmark for efficiency and performance. The codes will be made publicly available upon acceptance.

Figures

Figures reproduced from arXiv: 2501.19255 by the authors.

Figure 1
Figure 1. Comparison of mIoU, parameters, GFLOPs, and latency [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Holistic architecture of the proposed ContextFormer. On (left), the whole architecture is shown, whereas, on (right) details of each of the blocks are depicted. Where, Attention = softmax  QKT √ dk  V . The outputs from the BDC and ViT branches are fused to form X ′ f , forming a refined feature representation that is then pro￾cessed by a Feed-Forward Network (FFN) enhanced with depthwise convolutions and the over… view at source ↗
Figure 3
Figure 3. Visual results on the ADE20K validation set. The results [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 59 canonical work pages

  1. [1]

    Segnet: A deep convolutional encoder-decoder architecture for image segmentation

    Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern anal- ysis and machine intelligence, 39(12):2481–2495, 2017. 1

    work page 2017

  2. [2]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 1209–1218, 2018. 2, 4, 5, 8

    work page 2018

  3. [3]

    Pem: Prototype-based efficient maskformer for image segmentation

    Niccol `o Cavagnero, Gabriele Rosi, Claudia Cuttano, Francesca Pistilli, Marco Ciccone, Giuseppe Averta, and Fabio Cermelli. Pem: Prototype-based efficient maskformer for image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15804–15813, 2024. 5, 6

    work page 2024

  4. [4]

    Encoder-decoder with atrous separable convolution for semantic image segmentation

    Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018. 6, 8

    work page 2018

  5. [5]

    Deeplabv3+: Encoderde- coder with atrous separable convolution for semantic image segmentation [m]

    Liang-Chieh CHEN, Yukun ZHU, George PAPANDREOU, F Schroff, Aug CV , and H Adam. Deeplabv3+: Encoderde- coder with atrous separable convolution for semantic image segmentation [m]. ferrari v, hebert m, sminchisescu c, et al. eccv (7), 2018. 1

    work page 2018

  6. [6]

    Mobile- former: Bridging mobilenet and transformer

    Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile- former: Bridging mobilenet and transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5270–5279, 2022. 2

    work page 2022

  7. [7]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290–1299, 2022. 1

    2022

  8. [8]

    MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark

    MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https : / / github . com / open - mmlab/mmsegmentation, 2020. 5

    2020

Show all 80 references
  1. [9]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceed- ings of the IEEE conference on computer vision and pattern re...

    2016

  2. [10]

    Coatnet: Marrying convolution and attention for all data sizes

    Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in neural information processing systems , 34:3965–3977, 2021. 2

    2021

  3. [11]

    Scaling vision transformers to 22 billion pa- rameters

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdul- mohsin, et al. Scaling vision transformers to 22 billion pa- rameters. In International Conference on Machine...

    2023

  4. [12]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 5

    2009

  5. [13]

    Hr-nas: Searching ef- ficient high-resolution neural architectures with lightweight transformers

    Mingyu Ding, Xiaochen Lian, Linjie Yang, Peng Wang, Xi- aojie Jin, Zhiwu Lu, and Ping Luo. Hr-nas: Searching ef- ficient high-resolution neural architectures with lightweight transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , p...

    2021

  6. [14]

    Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks

    Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1911–1920, 2019. 1

    1911

  7. [15]

    Repvgg: Making vgg-style convnets great again

    Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 13733–13742, 2021. 1

    2021

  8. [16]

    Tinynet: A lightweight, modular, and unified network architecture for the internet of things

    Wei Dong, Jiamei Lv, Gonglong Chen, Yihui Wang, Huikang Li, Yi Gao, and Dinesh Bharadia. Tinynet: A lightweight, modular, and unified network architecture for the internet of things. In Proceedings of the 20th Annual International Con- ference on Mobile Systems, Applications a...

    2022

  9. [17]

    Cswin transformer: A general vision transformer backbone with cross-shaped windows

    Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pa...

    2022

  10. [18]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  11. [19]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 2

    2021

  12. [20]

    Levit: a vision transformer in convnet’s clothing for faster inference

    Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herv ´e J ´egou, and Matthijs Douze. Levit: a vision transformer in convnet’s clothing for faster inference. In Proceedings of the IEEE/CVF interna- tional conference on computer vision , pages 122...

  13. [21]

    Cmt: Convolutional neural networks meet vision transformers

    Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12175–12185, 2022. 2

    2022

  14. [22]

    Segnext: Rethink- ing convolutional attention design for semantic segmenta- tion

    Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethink- ing convolutional attention design for semantic segmenta- tion. Advances in Neural Information Processing Systems , 35:1140–1156, 2022. 6

    2022

  15. [23]

    Ghostnet: More features from cheap 9 operations

    Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap 9 operations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1580–1589,

  16. [24]

    mask r-cnn,

    Kaiming He and Georgia Gkioxari. P. doll ar, and r. gir- shick,“mask r-cnn,”. In Proc. IEEE Int. Conf. Comput. Vis , pages 2980–2988, 2017. 1

    2017

  17. [25]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 1, 5, 6

    2016

  18. [26]

    Axial attention in multidimensional transformers

    Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.12180, 2019. 1

    1912 arXiv

  19. [27]

    Searching for mo- bilenetv3

    Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mo- bilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019. 1, 2, 5, 6, 8

    2019

  20. [28]

    Mobilenets: Efficient convolu- tional neural networks for mobile vision applications

    Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco An- dreetto, and Hartwig Adam. Mobilenets: Efficient convolu- tional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. 1, 2, 3

    2017 arXiv

  21. [29]

    Trseg: Trans- former for semantic segmentation

    Youngsaeng Jin, David Han, and Hanseok Ko. Trseg: Trans- former for semantic segmentation. Pattern Recognition Let- ters, 148:29–35, 2021. 2

    2021

  22. [30]

    Metaseg: Metaformer-based global contexts-aware network for efficient semantic segmentation

    Beoungwoo Kang, Seunghun Moon, Yubin Cho, Hyunwoo Yu, and Suk-Ju Kang. Metaseg: Metaformer-based global contexts-aware network for efficient semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 434–443, 2024. 5, 6

    2024

  23. [31]

    Panoptic feature pyramid networks

    Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Doll´ar. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6399–6408, 2019. 5, 6

    2019

  24. [32]

    Dfanet: Deep feature aggregation for real-time semantic seg- mentation

    Hanchao Li, Pengfei Xiong, Haoqiang Fan, and Jian Sun. Dfanet: Deep feature aggregation for real-time semantic seg- mentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9522–9531,

  25. [33]

    Convmlp: Hierarchical convolutional mlps for vision

    Jiachen Li, Ali Hassani, Steven Walton, and Humphrey Shi. Convmlp: Hierarchical convolutional mlps for vision. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 6307–6316, 2023. 6

    2023

  26. [34]

    Partial order pruning: for best speed/accuracy trade-off in neural architecture search

    Xin Li, Yiming Zhou, Zheng Pan, and Jiashi Feng. Partial order pruning: for best speed/accuracy trade-off in neural architecture search. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 9145–9153, 2019. 2

    2019

  27. [35]

    Efficientformer: Vision transformers at mobilenet speed

    Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evan- gelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems , 35: 12934–12949, 2022. 1, 2

    2022

  28. [36]

    Re- thinking vision transformers for mobilenet size and speed

    Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Re- thinking vision transformers for mobilenet size and speed. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16889–16900, 2023. 1

    2023

  29. [37]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceeding...

    2014

  30. [38]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. In Pro- ceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 8

    2017

  31. [39]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 2

    2021

  32. [40]

    Swin transformer v2: Scaling up capacity and resolution

    Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 12009–12019, 2022. 1

    2022

  33. [41]

    Fully convolutional networks for semantic segmentation

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Pro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 3431–3440, 2015. 6, 8

    2015

  34. [42]

    Shufflenet v2: Practical guidelines for efficient cnn architec- ture design

    Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architec- ture design. In Proceedings of the European conference on computer vision (ECCV), pages 116–131, 2018. 1, 2, 5, 6, 8

    2018

  35. [43]

    Mobilevit: light- weight, general-purpose, and mobile-friendly vision trans- former

    Sachin Mehta and Mohammad Rastegari. Mobilevit: light- weight, general-purpose, and mobile-friendly vision trans- former. arXiv preprint arXiv:2110.02178, 2021. 1, 2

    2021 arXiv

  36. [44]

    Separable self- attention for mobile vision transformers

    Sachin Mehta and Mohammad Rastegari. Separable self- attention for mobile vision transformers. arXiv preprint arXiv:2206.02680, 2022. 1

    2022 arXiv

  37. [45]

    Espnetv2: A light-weight, power ef- ficient, and general purpose convolutional neural network

    Sachin Mehta, Mohammad Rastegari, Linda Shapiro, and Hannaneh Hajishirzi. Espnetv2: A light-weight, power ef- ficient, and general purpose convolutional neural network. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 9190–9200, 2019. 2

    2019

  38. [46]

    Review the state-of-the-art technologies of semantic segmentation based on deep learning

    Yujian Mo, Yan Wu, Xinneng Yang, Feilin Liu, and Yujun Liao. Review the state-of-the-art technologies of semantic segmentation based on deep learning. Neurocomputing, 493: 626–646, 2022. 1

    2022

  39. [47]

    The role of context for object detection and semantic segmentation in the wild

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recogni...

    2014

  40. [48]

    Context-guided spatial feature recon- struction for efficient semantic segmentation

    Zhenliang Ni, Xinghao Chen, Yingjie Zhai, Yehui Tang, and Yunhe Wang. Context-guided spatial feature recon- struction for efficient semantic segmentation. arXiv preprint arXiv:2405.06228, 2024. 6

    2024 arXiv

  41. [49]

    Edgevits: Competing light-weight cnns on mobile devices with vision transformers

    Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tzimiropoulos, and Brais 10 Martinez. Edgevits: Competing light-weight cnns on mobile devices with vision transformers. In European Conference on Computer Vision, pages 294–311. Springer,...

    2022

  42. [50]

    Enet: A deep neural network architec- ture for real-time semantic segmentation

    Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eu- genio Culurciello. Enet: A deep neural network architec- ture for real-time semantic segmentation. arXiv preprint arXiv:1606.02147, 2016. 2, 8

    2016 arXiv

  43. [51]

    Erfnet: Efficient residual factorized convnet for real-time semantic segmentation

    Eduardo Romera, Jos ´e M Alvarez, Luis M Bergasa, and Roberto Arroyo. Erfnet: Efficient residual factorized convnet for real-time semantic segmentation. IEEE Transactions on Intelligent Transportation Systems, 19(1):263–272, 2017. 2

    2017

  44. [52]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  45. [53]

    Mobilenetv2: Inverted residuals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4510–4520, 2018. 1, 2, 3, 5, 6, 8

    2018

  46. [54]

    Ssformer: A lightweight transformer for semantic segmentation

    Wen Shi, Jing Xu, and Pan Gao. Ssformer: A lightweight transformer for semantic segmentation. 2022 IEEE 24th International Workshop on Multimedia Signal Processing (MMSP), pages 1–5, 2022. 1

    2022

  47. [55]

    Feedformer: Revisiting transformer decoder for efficient semantic segmentation

    Jae-hun Shim, Hyunwoo Yu, Kyeongbo Kong, and Suk- Ju Kang. Feedformer: Revisiting transformer decoder for efficient semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 2263– 2271, 2023. 6

    2023

  48. [56]

    Deep high-resolution representation learning for human pose es- timation

    Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose es- timation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5693–5703,

  49. [57]

    Efficientnet: Rethinking model scaling for convolutional neural networks

    Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105–6114. PMLR,

  50. [58]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. In International conference on machine learning , pages 10347–10357. PMLR, 2021. 2

    2021

  51. [59]

    Seaformer: Squeeze-enhanced axial transformer for mobile semantic segmentation

    Qiang Wan, Zilong Huang, Jiachen Lu, YU Gang, and Li Zhang. Seaformer: Squeeze-enhanced axial transformer for mobile semantic segmentation. In The eleventh international conference on learning representations, 2023. 5, 6, 8

    2023

  52. [60]

    Deep high-resolution repre- sentation learning for visual recognition

    Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution repre- sentation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence , 43(10):3...

    2020

  53. [61]

    Rtformer: Effi- cient design for real-time semantic segmentation with trans- former

    Jian Wang, Chenhui Gou, Qiman Wu, Haocheng Feng, Junyu Han, Errui Ding, and Jingdong Wang. Rtformer: Effi- cient design for real-time semantic segmentation with trans- former. Advances in Neural Information Processing Systems, 35:7423–7436, 2022. 2

    2022

  54. [62]

    A novel transformer based se- mantic segmentation scheme for fine-resolution remote sens- ing images

    Libo Wang, Rui Li, Chenxi Duan, Ce Zhang, Xiaoliang Meng, and Shenghui Fang. A novel transformer based se- mantic segmentation scheme for fine-resolution remote sens- ing images. IEEE Geoscience and Remote Sensing Letters , 19:1–5, 2022. 2

    2022

  55. [63]

    Segformer: Simple and efficient design for semantic segmentation with transform- ers

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transform- ers. Advances in neural information processing systems, 34: 12077–12090, 2021. 1, 5, 6, 8

    2021

  56. [64]

    U-mixformer: Unet- like transformer with mix-attention for efficient semantic segmentation

    Seul-Ki Yeom and Julian von Klitzing. U-mixformer: Unet- like transformer with mix-attention for efficient semantic segmentation. arXiv preprint arXiv:2312.06272, 2023. 6

    2023 arXiv

  57. [65]

    Bisenet: Bilateral segmentation network for real-time semantic segmentation

    Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceed- ings of the European conference on computer vision (ECCV), pages 325–341, 2018. 2

    2018

  58. [66]

    Tokens-to-token vit: Training vision transformers from scratch on imagenet

    Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF in- ternational conference on computer vision , page...

  59. [67]

    Object- contextual representations for semantic segmentation

    Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object- contextual representations for semantic segmentation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16, pages 173–190. Springer, 2020. 6

    2020

  60. [68]

    In- terleaved group convolutions

    Ting Zhang, Guo-Jun Qi, Bin Xiao, and Jingdong Wang. In- terleaved group convolutions. In Proceedings of the IEEE international conference on computer vision , pages 4373– 4382, 2017. 2

    2017

  61. [69]

    Topformer: Token pyramid transformer for mobile semantic segmentation

    Wenqiang Zhang, Zilong Huang, Guozhong Luo, Tao Chen, Xinggang Wang, Wenyu Liu, Gang Yu, and Chunhua Shen. Topformer: Token pyramid transformer for mobile semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12083– ...

    2022

  62. [70]

    Shufflenet: An extremely efficient convolutional neural net- work for mobile devices

    Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural net- work for mobile devices. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 6848–6856, 2018. 1, 2

    2018

  63. [71]

    Pyramid scene parsing network

    Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017. 6, 8

    2017

  64. [72]

    Icnet for real-time semantic segmenta- tion on high-resolution images

    Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Icnet for real-time semantic segmenta- tion on high-resolution images. In Proceedings of the Eu- ropean conference on computer vision (ECCV) , pages 405– 420, 2018. 2

    2018

  65. [73]

    Scene parsing through 11 ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through 11 ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 633–641,

  66. [74]

    Rethinking bottleneck structure for efficient mobile network design

    Daquan Zhou, Qibin Hou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. Rethinking bottleneck structure for efficient mobile network design. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16 , pages 680–697. Springer,

    2020

  67. [75]

    Biformer: Vision transformer with bi-level routing attention

    Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, and Rynson WH Lau. Biformer: Vision transformer with bi-level routing attention. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 10323–10333, 2023. 2 12 ContextFormer: Redefining E...

    2023

  68. [76]

    As demonstrated in Fig

    ImageNet Pre-training For a fair comparison, we initialize the ContextFormer model using pre-trained parameters from ImageNet. As demonstrated in Fig. A, the classification framework of ContextFormer integrates an average pooling layer followed by a linear layer, leveraging gl...

  69. [77]

    Detailed Network Structure The detailed network structure of the proposed Con- textFormer model for the efficient real-time semantic seg- mentation task is provided in Tab. B. Where,dw and dw.sep denote depth-wise and depth-wise separable convolutions. Moreover, N, H and T den...

  70. [78]

    Details of Feed-Forward Network In the proposed ContextFormer model, the unified output semantics from the BDC block and ViT block are passed through the Feed-Forward Network. For the Feed-Forward Network, we have integrated depth-wise convolution layer between 1×1 convolution...

  71. [79]

    C shows additional visual results of the proposed Con- textFormer model with original images, ground-truth, Top- Former, and ContextFormer (GM E)

    Visual Results Fig. C shows additional visual results of the proposed Con- textFormer model with original images, ground-truth, Top- Former, and ContextFormer (GM E). The proposed model demonstrates superior segmentation performance on the validation set of the ADE20K benchmar...

  72. [80]

    However, cer- tain limitations warrant further investigation

    Limitations and Future Work The proposed model exhibits strong performance and ef- ficiency in semantic segmentation tasks. However, cer- tain limitations warrant further investigation. A notable constraint for lightweight efficient models, including ours, is their dependency ...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.