REVIEW 5 cited by
EMP-SSL: Towards Self-Supervised Learning in One Training Epoch
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Recently, self-supervised learning (SSL) has achieved tremendous success in learning image representation. Despite the empirical success, most self-supervised learning methods are rather "inefficient" learners, typically taking hundreds of training epochs to fully converge. In this work, we show that the key towards efficient self-supervised learning is to increase the number of crops from each image instance. Leveraging one of the state-of-the-art SSL method, we introduce a simplistic form of self-supervised learning method called Extreme-Multi-Patch Self-Supervised-Learning (EMP-SSL) that does not rely on many heuristic techniques for SSL such as weight sharing between the branches, feature-wise normalization, output quantization, and stop gradient, etc, and reduces the training epochs by two orders of magnitude. We show that the proposed method is able to converge to 85.1% on CIFAR-10, 58.5% on CIFAR-100, 38.1% on Tiny ImageNet and 58.5% on ImageNet-100 in just one epoch. Furthermore, the proposed method achieves 91.5% on CIFAR-10, 70.1% on CIFAR-100, 51.5% on Tiny ImageNet and 78.9% on ImageNet-100 with linear probing in less than ten training epochs. In addition, we show that EMP-SSL shows significantly better transferability to out-of-domain datasets compared to baseline SSL methods. We will release the code in https://github.com/tsb0601/EMP-SSL.
Forward citations
Cited by 5 Pith papers
-
A Principled Framework for Multi-View Contrastive Learning
Two multi-view contrastive losses, MV-InfoNCE and MV-DHEL, match InfoNCE's asymptotic optimum while improving accuracy and embedding rank with more views.
-
Replay-free Online Continual Learning with Self-Supervised MultiPatches
Continual MultiPatches (CMP) trains self-supervised online continual learning models on multiple patches of each single example, reaching higher linear probing accuracy than most experience replay baselines without st...
-
Seeing the Whole in the Parts in Self-Supervised Representation Learning
CO-SSL, which aligns pre-pooling local representations with the global image embedding, reaches 71.5% ImageNet-1K top-1 at 100 epochs and improves robustness over BYOL.
-
Improving Pre-trained Self-Supervised Embeddings Through Effective Entropy Maximization
Continuing SSL pre-training for ten epochs with a criterion that maximizes one-dimensional marginal entropies and minimizes pairwise covariances improves downstream ImageNet classification, especially with few labels.
-
An Empirical Study of Accuracy-Robustness Tradeoff and Training Efficiency in Self-Supervised Learning
CF-AMC-SSL, applying free adversarial training to 16-crop EMP-SSL, trains in 97 minutes versus 530 for robust EMP-SSL while improving PGD-8 robustness on CIFAR-10 from 28.49% to 33.34%.
Discussion (0). Continue with ORCID to comment.