Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representation

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Random masking alone can generate the views contrastive learning needs, without augmentations, reconstruction, or auxiliary modules.

desk verdict A clean, efficient SSL recipe that deserves referee time once the epoch inconsistency and missing comparisons are fixed. read the letter →

arxiv 2411.09858 v2 pith:KHTUREC4 submitted 2024-11-15 cs.CV

Xiaoyu Yang , Lijian Xu , Hongsheng Li , Shaoting Zhang This is my paper · ORCID
classification cs.CV
keywords occludedimagecontrastivelearningself-supervisedmaskedmodelingVisionTransformersemanticrepresentationefficientpre-trainingNet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the two dominant strands of visual self-supervision, masked image modeling and contrastive learning, can be merged into one simple recipe. The recipe, called occluded image contrastive learning (OCL), randomly masks each image, splits the visible patches into two non-overlapping groups, and treats the two resulting views as a positive pair to be contrasted against views from other images in the batch. It discards pixel reconstruction, hand-crafted data augmentations, momentum encoders, and projection heads. If correct, OCL would deliver competitive ImageNet-level representations with far less compute, with the abstract reporting 133 hours on four A100 GPUs for a ViT-L/16 that reaches 85.8% top-1 accuracy after fine-tuning.

What carries the argument

The load-bearing mechanism is the random non-overlapping partition of visible patches into two groups, which supplies the two views for contrastive learning. Each group carries its own [CLS] token through the Vision Transformer; the [CLS] tokens are L2-normalized and compared with a T-distributed spherical similarity that concentrates the softmax so only close pairs count as positives. The same encoder processes both groups with no auxiliary head, and the loss is a symmetric cross-entropy contrastive loss computed over the whole mini-batch.

What would settle it

Take images with two clearly distinct objects, mask each image so one contrastive branch sees mostly object A and the other mostly object B, and check whether the pre-trained model still treats those views as a positive pair; a large drop in fine-tuning accuracy on multi-object images, or a direct similarity probe showing such pairs are far apart, would falsify the semantic-equivalence premise.

Watch

Extended reading notes

Core claim

The paper claims that occlusion alone can create the semantically meaningful views that contrastive learning needs. Concretely, OCL masks a batch of images with a moderate global ratio, randomly divides the surviving patches of each image into two non-overlapping groups, and feeds both groups through one Vision Transformer, using each group's [CLS] token as the view representation. Positive pairs are the two [CLS] tokens from the same image; negative pairs are [CLS] tokens from different images in the batch. Similarity is scored with a T-distributed spherical metric, and a symmetric cross-entropy loss pulls the positives together. The argument is that masking removes conceptual redundancy and creates fine-grained semantic differences at the concept level rather than the instance level, so the contrastive task can learn high-level categories without reconstructing pixels or engineering augmentations.

Load-bearing premise

The load-bearing premise is that two random, non-overlapping halves of an image's visible patches always show the same high-level thing, so that pulling them together teaches categories rather than coincidental patches.

Editorial extensions

If this is right

  • Pre-training can drop hand-crafted augmentation pipelines entirely, since random masking is the only view generator.
  • Removing pixel reconstruction and auxiliary modules shortens pre-training: the paper reports 800 epochs where classic masked autoencoding uses 1,600, with less wall-clock time.
  • The method scales with model size, with ViT-L/16 improving linear probing by roughly four points over ViT-B/16 while keeping pre-training time manageable.
  • The learned representations transfer to object detection, semantic segmentation, and robustness benchmarks at a competitive level despite the simpler objective.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The central premise, that two random non-overlapping views of an image share the same category-level meaning, is testable by measuring whether positive pairs stay closer than negatives on images containing multiple distinct objects; if they do not, object-aware masking could strengthen OCL.
  • The abstract's 133-hour figure for ViT-L/16 appears inconsistent with the 533-hour pre-training time shown in Table 3 for the same configuration; the efficiency claim is the abstract's, while the table is the numerical record.
  • The fixed concentration parameter κ of the T-SP metric, reported as best at 64, could plausibly be annealed from loose to tight during training, a schedule the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Occluded Image Contrastive Learning (OCL), a self-supervised pre-training method for Vision Transformers. OCL randomly masks image patches, splits the visible patches into two non-overlapping groups, and applies a contrastive loss to the [CLS] tokens of the two groups, using the T-distributed spherical similarity metric. The authors claim that this creates semantically meaningful views without hand-crafted augmentations or auxiliary modules, and that it is more efficient than masked image modeling and prior contrastive methods. They report ImageNet-1K linear probing and fine-tuning results for ViT-B/16 and ViT-L/16, additional results on COCO, ADE20K, and robustness benchmarks, and ablations over masking ratio, batch size, MLP head, and the concentration parameter κ.

Significance. If the central claims hold, OCL would be a useful simplification of contrastive pre-training: it removes hand-crafted view augmentations and momentum encoders, avoids pixel reconstruction, and relies only on random masking and a contrastive loss. The paper's strengths include a simple and reproducible recipe, extensive experiments across model scales and downstream tasks, and ablations that probe several design choices. However, the significance of the efficiency claim is currently undermined by direct numerical inconsistencies in the reported pre-training time and epoch counts, and the 'semantic-aware views' mechanism that motivates the method is not directly verified. These issues are fixable but are load-bearing for the paper's main message.

major comments (5)
  1. [Abstract, §3.2.2 (Table 3), §B.2] The abstract states that ViT-L/16 can complete pre-training in 133 hours using 4 A100 GPUs, but Table 3 reports 533 pre-training hours for the ViT-L/16 configuration that achieves 85.8% fine-tuning accuracy, and §B.2 describes the setup as 2×2 A100 GPUs. The headline efficiency number differs from the body by a factor of four. Please correct the abstract and ensure that all reported pre-training times use the same hardware and epoch count.
  2. [Table 6, §3.3, Table 9] Table 6 lists OCL with 1,600 pre-training epochs and reports ViT-B/16 LIN 74.2 and FT 83.4, while §3.3 states that 'the training process is finished in 800 epochs' and Table 9 lists Training Epochs 800 for both ViT-B/16 and ViT-L/16. This discrepancy directly affects the comparison with MAE and CAE, which are listed at 1,600 epochs, and must be resolved: if OCL uses 800 epochs, Table 6 should say 800; if it uses 1,600, the text and Table 9 are wrong.
  3. [§3.2.1 (Table 2)] The masking-ratio ablation in Table 2 varies the overall masked ratio together with the effective batch size and the per-branch visible ratio. For example, the ViT-L/16 rows change Eff. Bsz. from 1,024 to 2,048 to 7,200 while the masked ratio changes from 0.2 to 0.4 to 0.8. The text attributes all observed changes to the masked ratio ('we attribute this improvement to the masked ratio'), but the batch-size effects are known to be large from Table 3. A factorial or controlled ablation (e.g., holding effective batch size constant) is needed to support the stated conclusion.
  4. [§2.2, Eq. (3), Figure 2] The core premise that two random non-overlapping subsets of visible patches 'share the same high-level semantic concept' is assumed but never directly tested. In multi-object or cluttered images, a random subset of patches can easily depict a different object or mostly background from the other subset, in which case Eq. (3) would treat patches from different categories as positives and degrade category-level discrimination. Since the paper's novelty claim is specifically about 'semantic-aware views,' please provide a direct verification of semantic equivalence, for example by measuring label agreement between the two views on a labeled dataset, or by comparing against a variant that uses semantically informed patch grouping. The current ImageNet accuracy alone does not distinguish this mechanism from a generic masking augmentation.
  5. [Eq. (3), Algorithm 1] The formal loss in Eq. (3) is inconsistent with the implementation in Algorithm 1. The denominator sums over k=1 to 2B with an indicator [k≠i] (and the text later writes [k≠j]), but Algorithm 1 computes a B×B similarity matrix m @ n.T and applies cross-entropy with labels torch.arange(B), which includes the positive pair in the denominator. As written, Eq. (3) would exclude the positive from the denominator, which is not the standard InfoNCE form and not what the pseudo-code does. Please correct the equation and the indicator notation so the method is formally unambiguous.
minor comments (5)
  1. [Algorithm 1] The pseudo-code hardcodes masking ratio = 0.3, but §3 specifies 0.3 for ViT-B/16 and 0.4 for ViT-L/16. Please make the ratio an explicit parameter or align the example with the default setting.
  2. [Table 9] Table 9 lists the effective batch size for ViT-L/16 as 2,400, while §3 and Tables 2–4 use 2,048. This should be reconciled.
  3. [Table 7] The ViT-L/16 row for MoCo-v3 is empty; either fill in the results or remove the row to avoid suggesting missing data.
  4. [Tables 4 and 5] The fine-tuning differences in the MLP-head and κ ablations are small (e.g., 85.6 vs. 85.8 in Table 5), and no multiple-seed variance is reported. Please add error bars or a statement about run-to-run variability.
  5. [Throughout] There are several typos and notation errors, including 'non-overleaping' (§2.1), 'temputare' (§2.3), '⊮[k̸=j]' in Eq. (3), 'Comparsion' in Table 6 and Table 7 captions, and 'ADK20K' in Table 7. These should be corrected in a revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: OCL is an empirical method with explicit losses and external benchmarks; the self-cited T-SP formula is fully stated and independently ablated.

full rationale

OCL is an empirical pre-training recipe rather than a first-principles derivation, so there is no chain in which an output is forced by an input by construction. The self-citation with methodological weight is the T-SP similarity (Ref. [14]) used in Eq. (2); the similarity function is written out explicitly in the paper, and the kappa parameter is ablated in Section 3.2.4 rather than fitted to the reported result, so the citation merely names the origin of a formula and is not load-bearing. The 'semantic-aware views' premise in Section 2.2, namely that random non-overlapping patch subsets of one image share a high-level semantic concept, is an untested design assumption and a genuine validity risk for multi-object or cluttered images, but it is not a step in which an equation reduces to its own input. All headline results are compared against external methods such as MAE, MoCo v3, SiameseIM, and iBOT, and they are reported after standard hyperparameter selection rather than after fitting a parameter to the claimed prediction. No uniqueness theorem is imported from the authors' prior work, no hidden ansatz is smuggled in via citation, and the manuscript explicitly acknowledges related masked-contrastive work (ExtreMA, MSN, CAN) in Appendix A.4. Accordingly, no circular step is identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or architectural entities are introduced. The T-SP metric is borrowed from prior work. The free parameters are standard hyperparameters chosen via ablation on the target benchmark.

free parameters (4)
  • overall_masking_ratio = 0.3 (ViT-B/16), 0.4 (ViT-L/16)
    Selected via ablation on ImageNet accuracy (Table 2); it controls the visible ratio and effective batch size.
  • kappa_TSP = 64
    Chosen from ablation on ImageNet fine-tuning (Table 5) to maximize top-1 accuracy.
  • effective_batch_size = 9,600 (ViT-B/16), 2,048 or 2,400 (ViT-L/16)
    Set large for contrastive learning; ablation (Table 3) shows sensitivity, with 2,048 used for the headline 85.8% result.
  • pre-training_epochs = 800
    Chosen to balance compute and accuracy; the paper's efficiency claim depends on this choice versus MAE's 1,600.
assumptions (4)
  • domain assumption Uniform random sampling of visible patches yields two non-overlapping groups whose [CLS] representations are semantically related and can serve as positive pairs.
    Invoked in Section 2.2 and Figure 2; if the two views capture unrelated content, the contrastive target is noisy.
  • domain assumption The T-SP similarity (Eq. 2) with kappa and temperature is an appropriate similarity for contrastive learning.
    Taken from prior work [14]; its suitability for this masked-view setting is not derived or measured.
  • domain assumption A ViT [CLS] token aggregates sufficient semantic information from a subset of visible patches (30-35% per branch) to support downstream transfer.
    The architecture uses only [CLS] tokens for the contrastive loss; if partial-view [CLS] tokens are not informative, the pretext task is not solvable as intended.
  • domain assumption ImageNet-1K self-supervised pre-training transfers to COCO, ADE20K, and robustness benchmarks.
    Standard SSL assumption; the paper evaluates on these benchmarks without a theoretical transfer guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representation." pith.science (2026). https://pith.science/paper/KHTUREC4

@misc{pith2026241109858,
  author       = {Pith},
  title        = {Pith review of: One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KHTUREC4}},
  note         = {Machine review of arXiv:2411.09858}
}
read the original abstract

This paper proposes a scalable and straightforward pre-training paradigm for efficient visual conceptual representation called occluded image contrastive learning (OCL). Our OCL approach is simple: we randomly mask patches to generate different views within an image and contrast them among a mini-batch of images. The core idea behind OCL consists of two designs. First, masked tokens have the potential to significantly diminish the conceptual redundancy inherent in images, and create distinct views with substantial fine-grained differences on the semantic concept level instead of the instance level. Second, contrastive learning is adept at extracting high-level semantic conceptual features during the pre-training, circumventing the high-frequency interference and additional costs associated with image reconstruction. Importantly, OCL learns highly semantic conceptual representations efficiently without relying on hand-crafted data augmentations or additional auxiliary modules. Empirically, OCL demonstrates high scalability with Vision Transformers, as the ViT-L/16 can complete pre-training in 133 hours using only 4 A100 GPUs, achieving 85.8\% accuracy in downstream fine-tuning tasks. Code is available at https://anonymous.4open.science/r/OLRS/.

Figures

Figures reproduced from arXiv: 2411.09858 by the authors.

Figure 1
Figure 1. Comparison between different pre-training paradigms. The Model in blue is the pre-training model, and the orange modules indicate auxiliary modules. (a) Contrastive Learning (CL) endeavours to maximize the agreement between different views of an image. (b) Masked Image Modeling (MIM) aims to restore masked image patches. (c) Our occluded image contrastive learning: Through non-overlapping occluding, distinct tokens … view at source ↗
Figure 2
Figure 2. A toy example of masked images for conceptual contrastive learning. The low global masking ratio aids the model in capturing comprehensive information from the image and understanding the interconnectedness of various concepts within a mini-batch. Beyond that, each contrastive branch has a higher masking ratio, generating diverse views with different semantic concepts for contrastive learning and diminishing concept… view at source ↗
Figure 3
Figure 3. Efficiency and Scaling. MAE [2], I-JEPA [18] and MoCo v3 [12] are opted for comparison. All methods are evaluated by linear probing with Top-1 accuracy (Acc) as the metric, and the pre-training GPU time with A100 hour as the indicator. The pre-training epochs (denoted as ep) and model architecture are also exhibited. the need for auxiliary modules like momentum encoders. This simplicity makes OCL’s framework more ef… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Walking the Tightrope: Disentangling Beneficial and Detrimental Drifts in Non-Stationary Custom-Tuning

    cs.LG 2025-05 conditional novelty 5.0 of 10

    The paper proposes Counterfactual Preference Optimization, a DPO-style method that uses concept-graph-generated counterfactual reasoning trajectories to improve multimodal LLM reinforcement fine-tuning on chest X-ray tasks.

Reference graph

Works this paper leans on

63 extracted references · 56 canonical work pages · cited by 1 Pith paper

  1. [1]

    A Simple Framework for Contrastive Learning of Visual Representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning, pages 1597–1607. PMLR, 2020

    work page 2020

  2. [2]

    Masked Autoencoders Are Scalable Vision Learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022

    work page 2022

  3. [3]

    BEiT: BERT Pre-Training of Image Transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT Pre-Training of Image Transformers. In International Conference on Learning Representations, 2021. 10 A PREPRINT - F EBRUARY 17, 2025

    work page 2021

  4. [4]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 8748–8763. PMLR, 2021

    work page 2021

  5. [5]

    DropPos: Pre- Training Vision Transformers by Reconstructing Dropped Positions

    Haochen Wang, Junsong Fan, Yuxi Wang, Kaiyou Song, Tong Wang, and Zhao-Xiang Zhang. DropPos: Pre- Training Vision Transformers by Reconstructing Dropped Positions. 36:46134–46151, 2023

    work page 2023

  6. [6]

    Enhancing visual grounding and generalization: A multi-task cycle training approach for vision-language models, 2024

    Xiaoyu Yang, Lijian Xu, Hao Sun, Hongsheng Li, and Shaoting Zhang. Enhancing visual grounding and generalization: A multi-task cycle training approach for vision-language models, 2024

    work page 2024

  7. [7]

    Ma, Guangyi Chen, Eric P

    Lingjing Kong, Martin Q. Ma, Guangyi Chen, Eric P. Xing, Yuejie Chi, Louis-Philippe Morency, and Kun Zhang. Understanding Masked Autoencoders via Hierarchical Latent Variable Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7918–7928, 2023

    work page 2023

  8. [8]

    How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders

    Qi Zhang, Yifei Wang, and Yisen Wang. How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders. 35:27127–27139, 2022

    work page 2022

Show all 63 references
  1. [9]

    Siamese Masked Autoencoders, 2023

    Agrim Gupta, Jiajun Wu, Jia Deng, and Li Fei-Fei. Siamese Masked Autoencoders, 2023

    2023

  2. [10]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at...

    2021

  3. [11]

    Big Self-Supervised Models are Strong Semi-Supervised Learners, October 2020

    Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big Self-Supervised Models are Strong Semi-Supervised Learners, October 2020

    2020

  4. [12]

    An Empirical Study of Training Self-Supervised Vision Transformers

    Xinlei Chen, Saining Xie, and Kaiming He. An Empirical Study of Training Self-Supervised Vision Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640–9649

  5. [13]

    Emerging Properties in Self-Supervised Vision Transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660, 2021

    2021

  6. [14]

    T-distributed Spherical Feature Representa- tion for Imbalanced Classification

    Xiaoyu Yang, Yufei Chen, Xiaodong Yue, Shaoxun Xu, and Chao Ma. T-distributed Spherical Feature Representa- tion for Imbalanced Classification. 37(9):10825–10833

  7. [15]

    Adapting multi-modal large language model to concept drift in the long-tailed open world, 2024

    Xiaoyu Yang, Jie Lu, and En Yu. Adapting multi-modal large language model to concept drift in the long-tailed open world, 2024

    2024

  8. [16]

    T-vMF Similarity For Regularizing Intra-Class Feature Distribution

    Takumi Kobayashi. T-vMF Similarity For Regularizing Intra-Class Feature Distribution. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6612–6621. IEEE

    2021

  9. [17]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, and Michael Bernstein. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015

    2015

  10. [18]

    Self-Supervised Learning From Images With a Joint-Embedding Predictive Architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-Supervised Learning From Images With a Joint-Embedding Predictive Architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2023

  11. [19]

    Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, April 2018

    Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, April 2018

    2018

  12. [20]

    Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

    Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. In International Conference on Learning Representations, S...

    2019

  13. [21]

    Taud and J.F

    H. Taud and J.F. Mas. Multilayer Perceptron (MLP). In María Teresa Camacho Olmedo, Martin Paegelow, Jean-François Mas, and Francisco Escobar, editors, Geomatic Approaches for Modeling Land Change Scenarios, pages 451–455. Springer International Publishing, Cham, 2018

    2018

  14. [22]

    Gaussian Error Linear Units (GELUs), June 2023

    Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs), June 2023

    2023

  15. [23]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015

    2015 arXiv

  16. [24]

    Context Autoencoder for Self-supervised Representation Learning

    Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context Autoencoder for Self-supervised Representation Learning. International Journal of Computer Vision, 132(1):208–223, January 2024. 11 A PREPRI...

    2024

  17. [25]

    Siamese Image Modeling for Self-Supervised Vision Representation Learning

    Chenxin Tao, Xizhou Zhu, Weijie Su, Gao Huang, Bin Li, Jie Zhou, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese Image Modeling for Self-Supervised Vision Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2132–...

    2023

  18. [26]

    Masked Image Modeling with Denoising Contrast

    Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked Image Modeling with Denoising Contrast. In The Eleventh International Conference on Learning Representations, September 2023

    2023

  19. [27]

    Image BERT Pre-training with Online Tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT Pre-training with Online Tokenizer. In International Conference on Learning Representations, October 2022

    2022

  20. [28]

    Dense Contrastive Learning for Self- Supervised Visual Pre-Training

    Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense Contrastive Learning for Self- Supervised Visual Pre-Training. pages 3024–3033, 2021

    2021

  21. [29]

    An Empirical Study of Training Self-Supervised Vision Transformers

    Xinlei Chen, Saining Xie, and Kaiming He. An Empirical Study of Training Self-Supervised Vision Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640–9649, 2021

    2021

  22. [30]

    Green Hierarchical Vision Transformer for Masked Image Modeling

    Lang Huang, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, and Toshihiko Yamasaki. Green Hierarchical Vision Transformer for Masked Image Modeling. 35:19997–20010, 2022

    2022

  23. [31]

    Efficient Self-supervised Vision Transformers for Representation Learning

    Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient Self-supervised Vision Transformers for Representation Learning. In International Conference on Learning Representations, 2021

    2021

  24. [32]

    MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformers

    Jihao Liu, Xin Huang, Jinliang Zheng, Yu Liu, and Hongsheng Li. MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6252–6261, 2023

    2023

  25. [33]

    SimMIM: A Simple Framework for Masked Image Modeling

    Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. SimMIM: A Simple Framework for Masked Image Modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9653–9663, 2022

    2022

  26. [34]

    Mask R-CNN

    Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, pages 2961–2969

  27. [35]

    Unified Perceptual Parsing for Scene Understanding

    Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified Perceptual Parsing for Scene Understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 418–434

  28. [36]

    Masked Siamese Networks for Label-Efficient Learning

    Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked Siamese Networks for Label-Efficient Learning. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and ...

    2022

  29. [37]

    Natural Adversarial Examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural Adversarial Examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262–15271, 2021

    2021

  30. [38]

    The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization. In Pr...

    2021

  31. [39]

    Learning Robust Global Representations by Penalizing Local Predictive Power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning Robust Global Representations by Penalizing Local Predictive Power. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    2019

  32. [40]

    Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimicking

    Peng Gao, Ziyi Lin, Renrui Zhang, Rongyao Fang, Hongyang Li, Hongsheng Li, and Yu Qiao. Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimicking. 132(5):1546–1556, 2024

    2024

  33. [41]

    Towards latent masked image modeling for self-supervised visual representation learning

    Yibing Wei, Abhinav Gupta, and Pedro Morgado. Towards latent masked image modeling for self-supervised visual representation learning. In European Conference on Computer Vision, pages 1–17. Springer, 2025

    2025

  34. [42]

    An empirical study of training self-supervised vision transformers

    Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9640–9649, 2021

    2021

  35. [43]

    Data2vec: A general framework for self-supervised learning in speech, vision and language

    Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, pages 1298–1312. PMLR, 2022

    2022

  36. [44]

    Efficient self-supervised learning with contextual- ized target representations for vision, speech and language

    Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextual- ized target representations for vision, speech and language. In International Conference on Machine Learning, pages 1416–1429. PMLR, 2023. 12 A PREPRINT - F EBRUARY 17, 2025

    2023

  37. [45]

    ConCL: Concept Contrastive Learning for Dense Prediction Pre-training in Pathology Images

    Jiawei Yang, Hanbo Chen, Yuan Liang, Junzhou Huang, Lei He, and Jianhua Yao. ConCL: Concept Contrastive Learning for Dense Prediction Pre-training in Pathology Images. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors,Computer...

    2022

  38. [46]

    Self-Supervised Visual Representations Learning by Contrastive Mask Prediction

    Yucheng Zhao, Guangting Wang, Chong Luo, Wenjun Zeng, and Zheng-Jun Zha. Self-Supervised Visual Representations Learning by Contrastive Mask Prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10160–10169

  39. [47]

    MST: Masked Self-Supervised Transformer for Visual Representation

    Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, and Jinqiao Wang. MST: Masked Self-Supervised Transformer for Visual Representation. InAdvances in Neural Information Processing Systems, volume 34, pages 13165–131...

    2021

  40. [48]

    Masked siamese convnets: Towards an effective masking strategy for general-purpose siamese networks

    Li Jing, Jiachen Zhu, and Yann LeCun. Masked siamese convnets: Towards an effective masking strategy for general-purpose siamese networks

  41. [49]

    Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining

    Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In International Conference on Machine Learning, pages 28223–28243. PMLR, 2023

    2023

  42. [50]

    Contrastive masked autoencoders are stronger vision learners

    Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders are stronger vision learners. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

    2023

  43. [51]

    Contextual Image Masking Modeling via Synergized Contrasting without View Augmentation for Faster and Better Visual Pretraining

    Shaofeng Zhang, Feng Zhu, Rui Zhao, and Junchi Yan. Contextual Image Masking Modeling via Synergized Contrasting without View Augmentation for Faster and Better Visual Pretraining. In The Eleventh International Conference on Learning Representations, September 2023

    2023

  44. [52]

    CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasets, February 2023

    Jiange Yang, Sheng Guo, Gangshan Wu, and Limin Wang. CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasets, February 2023

    2023

  45. [53]

    Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label- Efficient Representations

    Ziyu Jiang, Yinpeng Chen, Mengchen Liu, Dongdong Chen, Xiyang Dai, Lu Yuan, Zicheng Liu, and Zhangyang Wang. Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label- Efficient Representations. In The Eleventh International Conference on Le...

    2023

  46. [54]

    Scaling language-image pre- training via masking

    Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre- training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23390–23400, 2023

    2023

  47. [55]

    Attentive mask clip

    Yifan Yang, Weiquan Huang, Yixuan Wei, Houwen Peng, Xinyang Jiang, Huiqiang Jiang, Fangyun Wei, Yin Wang, Han Hu, Lili Qiu, et al. Attentive mask clip. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2771–2781, 2023

    2023

  48. [56]

    A simple, efficient and scalable contrastive masked autoencoder for learning visual representations

    Shlok Mishra, Joshua Robinson, Huiwen Chang, David Jacobs, Aaron Sarna, Aaron Maschinot, and Dilip Krishnan. A simple, efficient and scalable contrastive masked autoencoder for learning visual representations. arXiv preprint arXiv:2210.16870, 2022

    arXiv

  49. [57]

    Extreme masking for learning instance and distributed visual representations

    Zhirong Wu, Zihang Lai, Xiao Sun, and Stephen Lin. Extreme masking for learning instance and distributed visual representations. arXiv preprint arXiv:2206.04667, 2022

    2022 arXiv

  50. [58]

    Masked siamese networks for label-efficient learning

    Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In European Conference on Computer Vision, pages 456–473. Springer, 2022

    2022

  51. [59]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, Lect...

    2014

  52. [60]

    Scene Parsing Through ADE20K Dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene Parsing Through ADE20K Dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 633–641. 13 A PREPRINT - F EBRUARY 17, 2025 A Related Works A.1...

    2025

  53. [62]

    The online branch encodes the initial view and predicts the representation of the second view based on their relative positions

    employs a Siamese network featuring two branches. The online branch encodes the initial view and predicts the representation of the second view based on their relative positions. Meanwhile, the target branch generates the target by encoding the second view. MSCN [48] generates...

    2025

  54. [63]

    MSCOCO [59] dataset is a large-scale dataset widely used for object detection and instance segmentation tasks created by Microsoft Research Asia

    and ImageNet-Sketch [39] are leveraged for validation of the generalization capability and robustness of the vision model, with the training source of ImageNet. MSCOCO [59] dataset is a large-scale dataset widely used for object detection and instance segmentation tasks create...

    2025

  55. [2022]

    Springer Nature Switzerland

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.