Pith. sign in

REVIEW 26 cited by

A Cookbook of Self-Supervised Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.12210 v2 pith:LRW2ZKR4 submitted 2023-04-24 cs.LG cs.CV

Randall Balestriero , Mark Ibrahim , Vlad Sobal , Ari Morcos , Shashank Shekhar , Tom Goldstein , Florian Bordes , Adrien Bardes
show 11 more authors
Gregoire Mialon Yuandong Tian Avi Schwarzschild Andrew Gordon Wilson Jonas Geiping Quentin Garrido Pierre Fernandez Amir Bar Hamed Pirsiavash Yann LeCun Micah Goldblum
This is my paper · ORCID
classification cs.LGcs.CV
keywords learningtrainingbarriercookbookentrymethodsself-supervisedadvance
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art with a high barrier to entry. While many components are familiar, successfully training a SSL method involves a dizzying set of choices from the pretext tasks to training hyper-parameters. Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook. We hope to empower the curious researcher to navigate the terrain of methods, understand the role of the various knobs, and gain the know-how required to explore how delicious SSL can be.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 26 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Self-Soupervision: Cooking Model Soups without Labels

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Model soups work when ingredient models are trained with different self-supervised objectives or on unlabeled data, improving robustness and transfer without labels.

  2. Next-Latent Prediction Transformers Learn Compact World Models

    cs.LG 2025-11 unverdicted novelty 6.0 of 10

    NextLat augments next-token prediction with latent next-state prediction, theoretically converging latents to belief states and showing empirical gains in world modeling, reasoning, planning, and faster inference via ...

  3. Where Do Backdoors Live? A Component-Level Analysis of Backdoor Propagation in Speech Language Models

    cs.CL 2025-10 unverdicted novelty 6.0 of 10

    Backdoors propagate through SLM components with persistence or erasure depending on the targeted part, and poisoned samples are not directly separable from benign ones in shared multitask embeddings.

  4. Multi-Class-Token Transformer for Multitask Self-supervised Music Information Retrieval

    cs.SD 2025-07 conditional novelty 6.0 of 10

    A single ViT with two specialized class tokens, one per self-supervised pretext task, learns both timbre-focused and harmony-focused music representations and outperforms a much larger MLM model on most MIR benchmarks.

  5. XxaCT-NN: Structure Agnostic Multimodal Learning for Materials Science

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Composition plus simulated XRD, fused by cross-attention with a new masked-XRD pretraining objective, reaches near-structure-based property accuracy on 5M materials.

  6. Data Normalization Strategies for EEG Deep Learning

    eess.SP 2025-06 conditional novelty 6.0 of 10

    Window-level, per-channel normalization helps supervised EEG tasks, while minimal or cross-channel window normalization suits contrastive self-supervised learning on EEG.

  7. scSSL-Bench: Benchmarking Self-Supervised Learning for Single-Cell Data

    q-bio.QM 2025-06 conditional novelty 6.0 of 10

    A benchmark of 19 self-supervised learning methods on 9 single-cell datasets shows generic SSL methods outperform specialized frameworks on multi-modal integration and cell typing, while masking is the most effective ...

  8. Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptions

    cs.LG 2025-05 reject novelty 6.0 of 10

    CHARM is a 7M-parameter self-supervised embedding model for multivariate time series that uses channel descriptions to beat specialized baselines on forecasting, classification, and anomaly detection.

  9. ConceptVAE: Self-Supervised Fine-Grained Concept Disentanglement from 2D Echocardiographies

    cs.CV 2025-02 reject novelty 6.0 of 10

    ConceptVAE learns to discretize echocardiograms into fine-grained anatomical concepts and per-concept styles without labels, and reports gains over a VICReg baseline on retrieval, segmentation, and OOD detection.

  10. General Feature Extraction In SAR Target Classification: A Contrastive Learning Approach Across Sensor Types

    eess.SP 2025-02 conditional novelty 6.0 of 10

    A self-supervised Vision Transformer trained on SETHI radar patches, without any MSTAR labels, gives 95.9% few-shot classification accuracy on MSTAR using k-NN with ten labels per class.

  11. Enhancing Content Representation for AR Image Quality Assessment Using Knowledge Distillation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    TransformAR-KD+ uses knowledge distillation with category-aware teachers and cross-attention decoders to improve AR image quality prediction, achieving SRCC 0.8411 on ARIQA.

  12. Transferring self-supervised pre-trained models for SHM data anomaly detection with scarce labeled data

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Autoencoder pre-training on unlabeled bridge monitoring data, followed by fine-tuning on a few hundred labels, raises anomaly detection F1 by 3 to 9 points over supervised training in two real bridge datasets.

  13. PFML: Self-Supervised Learning of Time-Series Data Without Representation Collapse

    cs.LG 2024-11 conditional novelty 6.0 of 10

    PFML predicts statistical functionals of masked time-series frames from unmasked context, yielding non-collapsed representations and performance on par with data2vec across IMU, speech, and EEG tasks.

  14. Chained Recursive Language Models for Multi-Iteration Reasoning

    cs.CL 2026-08 reject novelty 5.0 of 10

    Chained fresh-root model calls with plain-text artifacts improve reported long-context reasoning accuracy over a single-call baseline, but the evidence lacks error bars and compute-matched comparison.

  15. Contrastive Self-Supervised Network Intrusion Detection using Augmented Negative Pairs

    cs.LG 2025-09 conditional novelty 5.0 of 10

    CLAN clusters genuine benign network flows while repelling augmented copies, then classifies new flows by distance to the cluster centroid; on Lycos2017 it reports the highest mean AUROC among compared SSL and anomaly...

  16. Data-Driven Self-Supervised Learning for the Discovery of Solution Singularity for Partial Differential Equations

    math.NA 2025-06 conditional novelty 5.0 of 10

    A density-filtering plus polynomial-fit pipeline detects PDE solution singularities from unlabeled adaptive mesh nodes.

  17. AquaticCLIP: A Vision-Language Foundation Model for Underwater Scene Analysis

    cs.CV 2025-02 conditional novelty 5.0 of 10

    AquaticCLIP adapts CLIP to underwater imagery with a 2M image-text dataset and reports state-of-the-art zero-shot performance across marine classification tasks.

  18. Self-supervised Benchmark Lottery on ImageNet: Do Marginal Improvements Translate to Improvements on Similar Datasets?

    cs.CV 2025-01 conditional novelty 5.0 of 10

    Self-supervised model rankings change substantially on ImageNet variants: DINO and Swav drop on Rendition and Sketch while MoCo and Barlow improve, so ImageNet-only benchmarking is misleading.

  19. BarcodeMamba: State Space Models for Biodiversity Analysis

    cs.LG 2024-12 conditional novelty 5.0 of 10

    A Mamba-2 state space model pretrained on DNA barcodes matches or beats BarcodeBERT on species and genus classification with far fewer parameters, reaching 99.2% seen-species linear probe and 70.2% unseen-species 1-NN...

  20. Weakly Supervised Panoptic Segmentation for Defect-Based Grading of Fresh Produce

    cs.CV 2024-11 conditional novelty 5.0 of 10

    SAM-generated masks train a panoptic segmentation model that counts banana defects exactly in 36.2% of images and estimates relative defect size with r=0.96, at roughly one-tenth the annotation cost.

  21. CITADEL: Continual Anomaly Detection for Enhanced Learning in IoT Intrusion Detection

    cs.CR 2025-08 reject novelty 4.0 of 10

    CITADEL combines self-supervised masked autoencoders with KL-divergence-based memory selection and a hierarchical buffer to detect IoT intrusions without attack labels while retaining old knowledge.

  22. AI-Driven Climate Policy Scenario Generation for Sub-Saharan Africa

    cs.AI 2025-05 conditional novelty 4.0 of 10

    A RAG pipeline using llama3.2-3B and UN COP documents generated 34 policy scenarios for Sub-Saharan Africa, 30 passed author validation, but automated evaluation showed mixed agreement with human judgment.

  23. $\texttt{InfoHier}$: Hierarchical Information Extraction via Encoding and Embedding

    cs.IR 2025-01 reject novelty 4.0 of 10

    A joint SSL and hierarchical clustering framework is proposed, but the paper provides only a visual demonstration and defers theoretical support to future work.

  24. An Empirical Study of Accuracy-Robustness Tradeoff and Training Efficiency in Self-Supervised Learning

    cs.CV 2025-01 conditional novelty 4.0 of 10

    CF-AMC-SSL, applying free adversarial training to 16-crop EMP-SSL, trains in 97 minutes versus 530 for robust EMP-SSL while improving PGD-8 robustness on CIFAR-10 from 28.49% to 33.34%.

  25. Bridging Brain with Foundation Models through Self-Supervised Learning

    cs.LG 2025-06 conditional novelty 3.0 of 10

    A PRISMA-based survey maps self-supervised learning techniques, brain foundation models, datasets, and evaluation protocols for EEG and related neural signals, including a skeptical review of EEG-to-text decoding.

  26. A Primer on Large Language Models and their Limitations

    cs.CL 2024-12 unverdicted novelty 1.0 of 10

    A readable review of LLM concepts and limitations that argues LLM errors are not hallucinations but confident outputs with no regard for truth.

Pith tools