Pith. sign in

REVIEW 11 cited by

Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2301.03580 v2 pith:BRWZY25Y submitted 2023-01-09 cs.CV cs.AIcs.LG

Keyu Tian , Yi Jiang , Qishuai Diao , Chen Lin , Liwei Wang , Zehuan Yuan This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords maskedmodelingsparseconvolutionconvolutionalhierarchicalmodelspre-training
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We identify and overcome two key obstacles in extending the success of BERT-style pre-training, or the masked image modeling, to convolutional networks (convnets): (i) convolution operation cannot handle irregular, random-masked input images; (ii) the single-scale nature of BERT pre-training is inconsistent with convnet's hierarchical structure. For (i), we treat unmasked pixels as sparse voxels of 3D point clouds and use sparse convolution to encode. This is the first use of sparse convolution for 2D masked modeling. For (ii), we develop a hierarchical decoder to reconstruct images from multi-scale encoded features. Our method called Sparse masKed modeling (SparK) is general: it can be used directly on any convolutional model without backbone modifications. We validate it on both classical (ResNet) and modern (ConvNeXt) models: on three downstream tasks, it surpasses both state-of-the-art contrastive learning and transformer-based masked modeling by similarly large margins (around +1.0%). Improvements on object detection and instance segmentation are more substantial (up to +3.5%), verifying the strong transferability of features learned. We also find its favorable scaling behavior by observing more gains on larger models. All this evidence reveals a promising future of generative pre-training on convnets. Codes and models are released at https://github.com/keyu-tian/SparK.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Advancing Metallic Surface Defect Detection via Anomaly-Guided Pretraining on a Large Industrial Dataset

    cs.CV 2025-09 conditional novelty 7.0 of 10

    AGSSP pretrains object detectors by distilling anomaly maps from a CLIP-based teacher into the backbone and using derived pseudo-boxes to pretrain the detector, improving defect detection over ImageNet pretraining.

  2. Curia-MAE: Multi-Modal Multi-Anatomy MAE Pre-Training for 3D Medical Image Segmentation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Curia-MAE is a multi-modal, multi-anatomy masked autoencoder whose frozen encoder modestly improves 3D segmentation over its MAE baseline, with the largest gains on lesion tasks.

  3. Hi-End-MAE: Hierarchical encoder-driven masked autoencoders are stronger vision learners for medical image segmentation

    cs.CV 2025-02 conditional novelty 6.0 of 10

    Hi-End-MAE pre-trains ViT encoders for medical image segmentation by having the decoder query encoder features from multiple layers, improving one-shot segmentation DSC by about 6 points over prior medical SSL methods.

  4. SELMA3D challenge: Self-supervised learning for 3D light-sheet microscopy image segmentation

    eess.IV 2025-01 conditional novelty 6.0 of 10

    A new benchmark for self-supervised segmentation of 3D light-sheet microscopy shows that SSL improved generalization to an unseen structure in one controlled comparison, while slightly reducing performance on a known ...

  5. An OpenMind for 3D medical vision self-supervised learning

    cs.CV 2024-12 conditional novelty 6.0 of 10

    OpenMind is a 114k-volume open brain MRI pretraining dataset and a standardized benchmark showing MAE-style pretraining improves 3D segmentation, while contrastive methods help classification.

  6. GaussianPretrain: A Simple Unified 3D Gaussian Representation for Visual Pre-training in Autonomous Driving

    cs.CV 2024-11 conditional novelty 6.0 of 10

    GaussianPretrain uses fixed 3D Gaussian anchors, guided by LiDAR depth masks, to pre-train camera-based BEV models by jointly reconstructing color, depth, and occupancy, improving downstream 3D perception on nuScenes.

  7. BiVM: Accurate Binarized Neural Network for Efficient Video Matting

    cs.CV 2025-07 conditional novelty 5.0 of 10

    BiVM is a 1-bit binarized video matting network that beats prior binarized methods on accuracy and efficiency, with 11.82 MAD on VideoMatte240K versus 28.49 for ReActNet-binarized RVM.

  8. Self-supervised Spatial-Temporal Learner for Precipitation Nowcasting

    cs.LG 2024-12 conditional novelty 5.0 of 10

    SpaT-SparK, a SparK-based masked-image-modeling pretrainer with a translation network, reduces pMSE for short-term precipitation nowcasting but sacrifices recall and skill scores versus the smaller SmaAt-UNet.

  9. A BERT-Style Self-Supervised Learning CNN for Disease Identification from Retinal Images

    cs.CV 2025-04 conditional novelty 4.0 of 10

    Applying a SparK-style masked autoencoder to a lightweight CNN improves retinal disease classification, but the AD/PD gains are weakened by participant overlap between pre-training and evaluation sets.

  10. Data-Efficient Challenges in Visual Inductive Priors: A Retrospective

    cs.CV 2025-06 conditional novelty 3.0 of 10

    A retrospective of four data-limited computer vision challenges finds that ensembles and heavy augmentation, not novel inductive priors, drove winning performance.

  11. Thoughts on Objectives of Sparse and Hierarchical Masked Image Model

    eess.IV 2025-05 reject novelty 3.0 of 10

    A checkerboard-like 'mesh mask' for SparK pre-training ties the random mask at F1 87.7 on brain CT tumor classification, with no improvement.

Pith tools