REVIEW 7 cited by
An Empirical Study of Training Self-Supervised Vision Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This paper does not describe a novel method. Instead, it studies a straightforward, incremental, yet must-know baseline given the recent progress in computer vision: self-supervised learning for Vision Transformers (ViT). While the training recipes for standard convolutional networks have been highly mature and robust, the recipes for ViT are yet to be built, especially in the self-supervised scenarios where training becomes more challenging. In this work, we go back to basics and investigate the effects of several fundamental components for training self-supervised ViT. We observe that instability is a major issue that degrades accuracy, and it can be hidden by apparently good results. We reveal that these results are indeed partial failure, and they can be improved when training is made more stable. We benchmark ViT results in MoCo v3 and several other self-supervised frameworks, with ablations in various aspects. We discuss the currently positive evidence as well as challenges and open questions. We hope that this work will provide useful data points and experience for future research.
Forward citations
Cited by 7 Pith papers
-
Post-Generation Curation of Synthetic Images via Homogeneous-Heterogeneous Splitting
Post-generation selection via Homogeneous-Heterogeneous real-data splits and a fidelity-diversity score raises synthetic-image utility for classification and segmentation without retraining generators.
-
Self-Soupervision: Cooking Model Soups without Labels
Model soups work when ingredient models are trained with different self-supervised objectives or on unlabeled data, improving robustness and transfer without labels.
-
Hidden in plain sight: VLMs overlook their visual representations
VLMs perform far worse than their own visual encoders on vision-centric tasks because the language model fails to use accessible visual information and instead follows its language priors.
-
An Augmentation-Aware Theory for Self-Supervised Contrastive Learning
An augmentation-aware error bound for contrastive learning decomposes the augmentation gap into a minimum same-class distance and a maximum same-image distance, with a trade-off driven by augmentation strength.
-
Cut out and Replay: A Simple yet Versatile Strategy for Multi-Label Online Continual Learning
CUTER replays cropped label-specific object regions instead of whole multi-label images, and regularizes patch-feature graphs to keep the cropping ability alive, improving multi-label online continual learning across ...
-
Uncertainty Awareness Enables Efficient Labeling for Cancer Subtyping in Digital Pathology
An uncertainty-aware active learning method built on SimCLRv2 achieves competitive cancer subtyping accuracy with only 1-10% of expert labels on two histopathology benchmarks.
-
Self Distillation via Iterative Constructive Perturbations
A self-distillation framework that aligns features of original and gradient-perturbed inputs reports an accuracy jump from 22.93% to 41.99% on CIFAR-100 and improved SSIM/FID on CUB.
Discussion (0). Sign in to comment.