Pith. sign in

REVIEW 18 cited by

An Empirical Study of Training Self-Supervised Vision Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2104.02057 v4 pith:F7URNVIW submitted 2021-04-05 cs.CV cs.LG

Xinlei Chen , Saining Xie , Kaiming He This is my paper · ORCID
classification cs.CVcs.LG
keywords self-supervisedtrainingresultsvisionrecipesseveraltransformerswork
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

This paper does not describe a novel method. Instead, it studies a straightforward, incremental, yet must-know baseline given the recent progress in computer vision: self-supervised learning for Vision Transformers (ViT). While the training recipes for standard convolutional networks have been highly mature and robust, the recipes for ViT are yet to be built, especially in the self-supervised scenarios where training becomes more challenging. In this work, we go back to basics and investigate the effects of several fundamental components for training self-supervised ViT. We observe that instability is a major issue that degrades accuracy, and it can be hidden by apparently good results. We reveal that these results are indeed partial failure, and they can be improved when training is made more stable. We benchmark ViT results in MoCo v3 and several other self-supervised frameworks, with ablations in various aspects. We discuss the currently positive evidence as well as challenges and open questions. We hope that this work will provide useful data points and experience for future research.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Post-Generation Curation of Synthetic Images via Homogeneous-Heterogeneous Splitting

    cs.LG 2026-07 accept novelty 6.5 of 10

    Post-generation selection via Homogeneous-Heterogeneous real-data splits and a fidelity-diversity score raises synthetic-image utility for classification and segmentation without retraining generators.

  2. Self-Soupervision: Cooking Model Soups without Labels

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Model soups work when ingredient models are trained with different self-supervised objectives or on unlabeled data, improving robustness and transfer without labels.

  3. Hidden in plain sight: VLMs overlook their visual representations

    cs.CV 2025-06 conditional novelty 6.0 of 10

    VLMs perform far worse than their own visual encoders on vision-centric tasks because the language model fails to use accessible visual information and instead follows its language priors.

  4. An Augmentation-Aware Theory for Self-Supervised Contrastive Learning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    An augmentation-aware error bound for contrastive learning decomposes the augmentation gap into a minimum same-class distance and a maximum same-image distance, with a trade-off driven by augmentation strength.

  5. Cut out and Replay: A Simple yet Versatile Strategy for Multi-Label Online Continual Learning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    CUTER replays cropped label-specific object regions instead of whole multi-label images, and regularizes patch-feature graphs to keep the cropping ability alive, improving multi-label online continual learning across ...

  6. DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Training a text encoder to align with frozen DINOv2 features, plus two small adaptation blocks and balanced data curation, yields strong zero-shot classification and open-vocabulary segmentation at a fraction of CLIP'...

  7. Wearable Accelerometer Foundation Models for Health via Knowledge Distillation

    cs.LG 2024-12 conditional novelty 6.0 of 10

    A knowledge-distilled accelerometry encoder, taught by an unsupervised PPG teacher on 20 million minutes of paired wearable data, predicts heart rate, heart-rate variability, demographics, and 46 health conditions fro...

  8. Language Model as Visual Explainer

    cs.CV 2024-12 reject novelty 6.0 of 10

    LVX builds LLM-generated attribute trees to explain any trained image classifier without training the explainer, but its faithfulness metric is directly optimized by the method.

  9. Unsupervised Foundation Model-Agnostic Slide-Level Representation Learning

    cs.CV 2024-11 conditional novelty 6.0 of 10

    COBRA trains a Mamba-2-based slide encoder with contrastive learning across multiple foundation models and magnifications, beating prior slide encoders by +4.4% average AUC on CPTAC with only 3,048 TCGA pretraining slides.

  10. STORM: Slot-based Task-aware Object-centric Representation for robotic Manipulation

    cs.RO 2026-01 conditional novelty 5.0 of 10

    STORM uses a two-stage, text-guided slot attention module on frozen DINOv2 features to improve robot manipulation success and generalization to visual distractors in simulated benchmarks.

  11. Clustering-based Feature Representation Learning for Oracle Bone Inscriptions Detection

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A clustering-based contrastive loss that uses a clean oracle bone font library as an anchor improves character detection in rubbing images by small margins over three detection baselines.

  12. Uncertainty Awareness Enables Efficient Labeling for Cancer Subtyping in Digital Pathology

    cs.CV 2025-06 conditional novelty 5.0 of 10

    An uncertainty-aware active learning method built on SimCLRv2 achieves competitive cancer subtyping accuracy with only 1-10% of expert labels on two histopathology benchmarks.

  13. EmbodiedMAE: A Unified 3D Multi-Modal Representation for Robot Manipulation

    cs.RO 2025-05 conditional novelty 5.0 of 10

    A multi-modal masked autoencoder pre-trained on a new 3D-processed version of the DROID dataset outperforms existing vision foundation models as a robot manipulation backbone.

  14. ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts

    cs.CV 2024-12 conditional novelty 5.0 of 10

    ALoRE aggregates multiple low-rank experts in a Kronecker-product space and merges them into the frozen backbone, reporting top accuracy on FGVC and VTAB-1k with only 0.15M trainable parameters.

  15. Enhancing Parameter-Efficient Fine-Tuning of Vision Transformers through Frequency-Based Adaptation

    cs.CV 2024-11 reject novelty 5.0 of 10

    FreqFit is a frequency-domain filter module that, when inserted between ViT blocks, improves the accuracy of existing PEFT methods on most but not all evaluated benchmarks.

  16. Abnormality-Driven Representation Learning for Radiology Imaging

    cs.CV 2024-11 conditional novelty 5.0 of 10

    A lesion-crop contrastive learning method (LeCL) plus attention-based pooling produces CT representations that, in this study, match or slightly beat existing medical foundation models at a fraction of the training cost.

  17. Self Distillation via Iterative Constructive Perturbations

    cs.LG 2025-05 conditional novelty 4.0 of 10

    A self-distillation framework that aligns features of original and gradient-perturbed inputs reports an accuracy jump from 22.93% to 41.99% on CIFAR-100 and improved SSIM/FID on CUB.

  18. MAL: Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance

    cs.CV 2024-12 conditional novelty 4.0 of 10

    A cluster-masked autoregressive plus multi-task pretraining framework lifts xLSTM vision backbones to 83.4% top-1 accuracy on ImageNet-1K, roughly one point above the ViL-B baseline.

Pith tools