Pith. sign in

REVIEW 6 cited by

Self-supervised Pretraining of Visual Features in the Wild

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2103.01988 v2 pith:GE7BCFRL submitted 2021-03-02 cs.CV cs.AI

Self-supervised Pretraining of Visual Features in the Wild

Priya Goyal , Mathilde Caron , Benjamin Lefaudeux , Min Xu , Pengchao Wang , Vivek Pai , Mannat Singh , Vitaliy Liptchinsky
show 3 more authors
Ishan Misra Armand Joulin Piotr Bojanowski
This is my paper
classification cs.CV cs.AI
keywords self-supervisedlearningrandomdatasetimagenetimagesmethodsmodel
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Recently, self-supervised learning methods like MoCo, SimCLR, BYOL and SwAV have reduced the gap with supervised methods. These results have been achieved in a control environment, that is the highly curated ImageNet dataset. However, the premise of self-supervised learning is that it can learn from any random image and from any unbounded dataset. In this work, we explore if self-supervision lives to its expectation by training large models on random, uncurated images with no supervision. Our final SElf-supERvised (SEER) model, a RegNetY with 1.3B parameters trained on 1B random images with 512 GPUs achieves 84.2% top-1 accuracy, surpassing the best self-supervised pretrained model by 1% and confirming that self-supervised learning works in a real world setting. Interestingly, we also observe that self-supervised models are good few-shot learners achieving 77.9% top-1 with access to only 10% of ImageNet. Code: https://github.com/facebookresearch/vissl

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Masked Autoencoders Are Scalable Vision Learners

    cs.CV 2021-11 accept novelty 8.0

    Masked autoencoders with asymmetric encoder-decoder and 75% masking ratio enable scalable self-supervised pre-training of vision transformers, achieving 87.8% ImageNet-1K accuracy with ViT-Huge using only unlabeled data.

  2. Emerging Properties in Self-Supervised Vision Transformers

    cs.CV 2021-04 conditional novelty 8.0

    Self-supervised ViTs show emergent semantic segmentation and 78.3% k-NN accuracy on ImageNet; DINO reaches 80.1% linear evaluation with ViT-Base.

  3. Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning

    cs.CV 2025-07 conditional novelty 6.0

    Franca introduces nested Matryoshka clustering and positional disentanglement in a transparent SSL pipeline to deliver open-source vision models competitive with closed proprietary systems.

  4. Vision Transformers Need Registers

    cs.CV 2023-09 unverdicted novelty 6.0

    Adding register tokens to Vision Transformers eliminates high-norm background artifacts and raises state-of-the-art performance on dense visual prediction tasks.

  5. Benchmarking transferability of SSL pretraining to same and different modality segmentation tasks

    cs.CV 2026-05 unverdicted novelty 5.0

    SMIT, which combines masked image modeling with self-distillation, delivers the highest segmentation accuracy, fastest convergence, and best few-shot performance across nine CT and MRI tasks compared to contrastive an...

  6. DINOv2: Learning Robust Visual Features without Supervision

    cs.CV 2023-04 unverdicted novelty 5.0

    Pith review generated a malformed one-line summary.