Pith. sign in

REVIEW 11 cited by

More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2207.03620 v3 pith:OFR23JYB submitted 2022-07-07 cs.CV

Shiwei Liu , Tianlong Chen , Xiaohan Chen , Xuxi Chen , Qiao Xiao , Boqian Wu , Tommi Kärkkäinen , Mykola Pechenizkiy
show 2 more authors
Decebal Mocanu Zhangyang Wang
This is my paper · ORCID
classification cs.CV
keywords kernelsperformancekernellargetransformersadvancedbetterconvolutional
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Transformers have quickly shined in the computer vision world since the emergence of Vision Transformers (ViTs). The dominant role of convolutional neural networks (CNNs) seems to be challenged by increasingly effective transformer-based models. Very recently, a couple of advanced convolutional models strike back with large kernels motivated by the local-window attention mechanism, showing appealing performance and efficiency. While one of them, i.e. RepLKNet, impressively manages to scale the kernel size to 31x31 with improved performance, the performance starts to saturate as the kernel size continues growing, compared to the scaling trend of advanced ViTs such as Swin Transformer. In this paper, we explore the possibility of training extreme convolutions larger than 31x31 and test whether the performance gap can be eliminated by strategically enlarging convolutions. This study ends up with a recipe for applying extremely large kernels from the perspective of sparsity, which can smoothly scale up kernels to 61x61 with better performance. Built on this recipe, we propose Sparse Large Kernel Network (SLaK), a pure CNN architecture equipped with sparse factorized 51x51 kernels that can perform on par with or better than state-of-the-art hierarchical Transformers and modern ConvNet architectures like ConvNeXt and RepLKNet, on ImageNet classification as well as a wide range of downstream tasks including semantic segmentation on ADE20K, object detection on PASCAL VOC 2007, and object detection/segmentation on MS COCO.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SWIFT: Spatio-temporal Wavelet Integrated Forecasting Framework for Workload Traces

    cs.DC 2026-05 conditional novelty 6.0 of 10

    SWIFT forecasts cloud workloads at SOTA accuracy with linear complexity by combining learnable cascaded wavelet convolutions and sequential inter/intra-variable mixing.

  2. On the Reliability of Cue Conflict and Beyond

    cs.CV 2026-03 conditional novelty 6.0 of 10

    Stylized cue-conflict bias scores are confounded by impure cues, imbalance, ratio metrics and restricted labels; REFINED-BIAS supplies pure balanced stimuli and full-label MRR sensitivity for reliable diagnosis.

  3. Partial Ring Scan: Revisiting Scan Order in Vision State Space Models

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Ring-based scanning with selective channel routing improves accuracy, speed, and rotation robustness of vision state-space models.

  4. MambaFusion: Height-Fidelity Dense Global Fusion for Multi-modal 3D Object Detection

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A camera-LiDAR 3D detector built around a hybrid local-global Mamba block with height-fidelity LiDAR encoding reports 75.0 NDS on nuScenes validation, outperforming prior transformer-based fusion methods.

  5. Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Mix-LN, which uses Post-LN in early layers and Pre-LN in later layers, gives more uniform layer gradients and better LLM pretraining and fine-tuning results than Pre-LN or Post-LN alone.

  6. STLight: a Fully Convolutional Approach for Efficient Predictive Learning by Spatio-Temporal joint Processing

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A fully convolutional spatio-temporal patch mixer, STLight, matches or beats RNN-based baselines on predictive learning benchmarks at a fraction of the parameter and FLOP budget.

  7. BEVANet: Bilateral Efficient Visual Attention Network for Real-Time Semantic Segmentation

    cs.CV 2025-08 conditional novelty 5.0 of 10

    BEVANet reports 81.0% mIoU at 33 FPS on Cityscapes, 0.9% above PIDNet-M but with more parameters and FLOPs.

  8. A Retrospective Systematic Study on Hierarchical Sparse Query Transformer-assisted Ultrasound Screening for Early Hepatocellular Carcinoma

    cs.CV 2025-02 conditional novelty 5.0 of 10

    A hybrid CNN-transformer with mixture-of-experts (HSQformer) yields 95.38% AUC for HCC screening on a multi-center ultrasound test set, beating several baselines and matching senior radiologists.

  9. Moir\'e Zero: An Efficient and High-Performance Neural Architecture for Moir\'e Removal

    cs.CV 2025-07 conditional novelty 4.0 of 10

    MZNet reports state-of-the-art moiré removal on high-resolution benchmarks with a U-Net combining multi-scale dual attention, multi-shape large-kernel convolutions, and fused skip connections.

  10. DCFormer: Efficient 3D Vision-Language Modeling with Decomposed Convolutions

    cs.CV 2025-02 conditional novelty 4.0 of 10

    A decomposed 3D convolution encoder for chest CT reaches competitive pathology detection and image-text retrieval with far fewer parameters and FLOPs than transformer or full 3D convolution baselines.

  11. RecConv: Efficient Recursive Convolutions for Multi-Frequency Representations

    cs.CV 2024-12 conditional novelty 4.0 of 10

    RecConv recursively decomposes feature maps into multiple scales with shared small-kernel depthwise convolutions to grow the effective receptive field to k times 2^ell at roughly constant FLOPs, yielding the RecNeXt b...

Pith tools