REVIEW 16 cited by
A Cookbook of Self-Supervised Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
A Cookbook of Self-Supervised Learning
read the original abstract
Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art with a high barrier to entry. While many components are familiar, successfully training a SSL method involves a dizzying set of choices from the pretext tasks to training hyper-parameters. Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook. We hope to empower the curious researcher to navigate the terrain of methods, understand the role of the various knobs, and gain the know-how required to explore how delicious SSL can be.
Forward citations
Cited by 16 Pith papers
-
Learn from your own latents and not from tokens: A sample-complexity theory
Latent prediction SSL recovers latent trees from PCFG data with sample complexity constant in hierarchy depth L (up to logs), unlike exponential for token-level or supervised methods.
-
Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets
In generalized contrastive learning with imbalanced classes, optimal representations collapse to class means whose angular geometry is determined by class proportions via convex optimization, and extreme imbalance cau...
-
Toward Parking Spot Occupancy Recognition: A Self-Supervised Approach
A two-stage self-supervised protocol using SimCLR on generic then target unlabeled data, followed by generic-label fine-tuning, reaches 97.8% average accuracy for parking occupancy across three public datasets without...
-
RankUp: Towards High-rank Representations for Large Scale Advertising Recommender Systems
RankUp raises effective rank of representations in deep MetaFormer recommenders via randomized splitting and multi-embeddings, delivering 2-5% GMV gains in production deployments at Weixin.
-
Grounding Hierarchical Vision-Language-Action Models Through Explicit Language-Action Alignment
A contrastive alignment model plus offline preference learning explicitly grounds hierarchical VLA language descriptions to actions and visuals on LanguageTable, achieving performance comparable to fully supervised fi...
-
Rapidly deploying on-device eye tracking by distilling visual foundation models
DistillGaze reduces median gaze error by 58.62% on a 2000+ participant dataset by distilling foundation models into a 256K-parameter on-device model using synthetic labeled data and unlabeled real data.
-
LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
LeJEPA derives an optimal isotropic Gaussian target for embeddings and enforces it via sketched regularization to deliver scalable, heuristics-free self-supervised pretraining with 79% ImageNet linear accuracy on ViT-H/14.
-
Next-Latent Prediction Transformers Learn Compact World Models
NextLat augments next-token prediction with latent next-state prediction, theoretically converging latents to belief states and showing empirical gains in world modeling, reasoning, planning, and faster inference via ...
-
Self-supervised neural operator for solving partial differential equations
Self-supervised neural operator uses Bayesian PINNs to generate training data and a Transformer to learn PDE operators, achieving high accuracy on 1D/2D reaction-diffusion and fluid vibration problems with optional li...
-
Statistical learnability of smooth boundaries via pairwise binary classification with deep ReLU networks
Proves learnability of ordered multiple smooth boundaries in pairwise binary classification via localized deep ReLU networks.
-
Learning General Representation of 12-Lead Electrocardiogram with a Joint-Embedding Predictive Architecture
ECG-JEPA applies a joint-embedding predictive architecture with Cross-Pattern Attention to learn semantic representations from unlabeled 12-lead ECG data and reports state-of-the-art results on diagnostic classificati...
-
Self-Supervised Learning of Plant Image Representations
Domain-specific augmentations and plant-only training data produce stronger self-supervised representations for fine-grained plant recognition than standard SSL pipelines or ImageNet pretraining.
-
Self-Supervised Learning of Plant Image Representations
Domain-adapted augmentations and plant-specific training data improve self-supervised representations for fine-grained plant species recognition over standard SSL pipelines.
-
RankUp: Towards High-rank Representations for Large Scale Advertising Recommender Systems
RankUp enhances representation capacity in deep MetaFormer recommenders via permutation splitting and multi-embeddings, achieving GMV improvements of 2-5% in Weixin production systems.
-
Robust Cross-Domain Generalization Using Unlabeled Target Data with Source-Domain Supervision
Target-informed self-supervised pretraining via masked image modeling and contrastive learning, plus a confidence-aware infusion head, yields over 6% Dice improvement on unlabeled target-domain POCUS images for pediat...
-
There Will Be a Scientific Theory of Deep Learning
A mechanics of the learning process is emerging in deep learning theory, characterized by dynamics, coarse statistics, and falsifiable predictions across idealized settings, limits, laws, hyperparameters, and universa...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.