Pith. sign in

REVIEW 7 cited by

Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.15613 v2 pith:P3AEUVKA submitted 2024-05-24 cs.LG cs.AIcs.CV

Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach

Huy V. Vo , Vasil Khalidov , Timoth\'ee Darcet , Th\'eo Moutakanni , Nikita Smetanin , Marc Szafraniec , Hugo Touvron , Camille Couprie
show 5 more authors
Maxime Oquab Armand Joulin Herv\'e J\'egou Patrick Labatut Piotr Bojanowski
This is my paper
classification cs.LG cs.AIcs.CV
keywords datacurationdatasetslearningself-supervisedtrainedapproachautomatic
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Self-supervised features are the cornerstone of modern machine learning systems. They are typically pre-trained on data collections whose construction and curation typically require extensive human effort. This manual process has some limitations similar to those encountered in supervised learning, e.g., the crowd-sourced selection of data is costly and time-consuming, preventing scaling the dataset size. In this work, we consider the problem of automatic curation of high-quality datasets for self-supervised pre-training. We posit that such datasets should be large, diverse and balanced, and propose a clustering-based approach for building ones satisfying all these criteria. Our method involves successive and hierarchical applications of $k$-means on a large and diverse data repository to obtain clusters that distribute uniformly among data concepts, followed by a hierarchical, balanced sampling step from these clusters. Extensive experiments on three different data domains including web-based images, satellite images and text show that features trained on our automatically curated datasets outperform those trained on uncurated data while being on par or better than ones trained on manually curated data. Code is available at https://github.com/facebookresearch/ssl-data-curation.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DataComp-VLM: Improved Open Datasets for Vision-Language Models

    cs.CV 2026-06 conditional novelty 8.0

    DataComp-VLM benchmark shows instruction-heavy data mixing outperforms filtering for VLM training, with DCVLM-Baseline achieving 63.6% on 33 tasks for 8B models (+5.4pp over FineVision).

  2. OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer

    cs.CV 2026-04 unverdicted novelty 7.0

    OmniShotCut treats shot boundary detection as structured relational prediction via a shot-query Transformer, uses fully synthetic transitions for training data, and releases OmniShotCutBench for evaluation.

  3. DataComp-VLM: Improved Open Datasets for Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 6.0

    DataComp-VLM benchmark shows instruction-heavy data mixtures outperform caption-heavy ones for VLM training, with DCVLM-Baseline reaching 63.6% on 33 tasks using 200B tokens, +5.4pp over FineVision.

  4. SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Models

    cs.CV 2025-12 conditional novelty 6.0

    SigLino distills SigLIP2 and DINOv3 into efficient vision models via asymmetric relation-knowledge distillation, token-balanced batching, and hierarchical data sampling on a new 200M-image corpus, yielding better tran...

  5. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics

    cs.LG 2025-11 conditional novelty 6.0

    LeJEPA derives an optimal isotropic Gaussian target for embeddings and enforces it via sketched regularization to deliver scalable, heuristics-free self-supervised pretraining with 79% ImageNet linear accuracy on ViT-H/14.

  6. MegaStyle: Constructing Diverse and Scalable Style Dataset via Consistent Text-to-Image Style Mapping

    cs.CV 2026-04 unverdicted novelty 5.0

    A scalable pipeline generates an intra-consistent, inter-diverse 1.4M style image dataset from text-to-image models and uses it to train a style encoder and generalizable style transfer model.

  7. Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer

    cs.CV 2025-11 unverdicted novelty 5.0

    Z-Image is an efficient 6B-parameter foundation model for image generation that rivals larger commercial systems in photorealism and bilingual text rendering through a new single-stream diffusion transformer and strea...