REVIEW 6 cited by
Exploring Simple Siamese Representation Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Siamese networks have become a common structure in various recent models for unsupervised visual representation learning. These models maximize the similarity between two augmentations of one image, subject to certain conditions for avoiding collapsing solutions. In this paper, we report surprising empirical results that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing. We provide a hypothesis on the implication of stop-gradient, and further show proof-of-concept experiments verifying it. Our "SimSiam" method achieves competitive results on ImageNet and downstream tasks. We hope this simple baseline will motivate people to rethink the roles of Siamese architectures for unsupervised representation learning. Code will be made available.
Forward citations
Cited by 6 Pith papers
-
Transformers as Unrolled Inference in Probabilistic Laplacian Eigenmaps: An Interpretation and Potential Improvements
Transformers can be viewed as unrolled inference in a probabilistic Laplacian Eigenmaps model, and replacing the attention matrix by attention minus identity improves validation performance.
-
scSSL-Bench: Benchmarking Self-Supervised Learning for Single-Cell Data
A benchmark of 19 self-supervised learning methods on 9 single-cell datasets shows generic SSL methods outperform specialized frameworks on multi-modal integration and cell typing, while masking is the most effective ...
-
FRAME: Pre-Training Video Feature Representations via Anticipation and Memory
FRAME distills DINO and CLIP features into a compact video encoder with a memory module and future-frame prediction, outperforming image-based and self-supervised video baselines on dense video tasks.
-
Contrastive Self-Supervised Network Intrusion Detection using Augmented Negative Pairs
CLAN clusters genuine benign network flows while repelling augmented copies, then classifies new flows by distance to the cluster centroid; on Lycos2017 it reports the highest mean AUROC among compared SSL and anomaly...
-
Leveraging neural network interatomic potentials for a foundation model of chemistry
Using embeddings from a pretrained neural network interatomic potential as features for small machine learning models gives competitive or better property predictions than end-to-end deep networks, especially with lim...
-
Examination of PCA Utilisation for Multilabel Classifier of Multispectral Images
On SSL4EO-S12-ML, PCA preprocessing improves fine-tuned DINOv2 multi-label classifiers (F1 62.97 vs 31.16) but degrades ResNet50 and most frozen-encoder configurations.
Discussion (0). Sign in to comment.