RECORD
On the Importance of Asymmetry for Siamese Representation Learning
Computer Vision and Pattern Recognition · Submitted 1 Apr 2022
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Our key insight is keeping a relatively lower variance in target than source generally benefits learning.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Many recent self-supervised frameworks for visual representation learning are based on certain forms of Siamese networks. Such networks are conceptually symmetric with two parallel encoders, but often practically asymmetric as numerous mechanisms are devised to break the symmetry. In this work, we conduct a formal study on the importance of asymmetry by explicitly distinguishing the two encoders within the network -- one produces source encodings and the other targets. Our key insight is keeping a relatively lower variance in target than source generally benefits learning. This is empirically justified by our results from five case studies covering different variance-oriented designs, and is aligned with our preliminary theoretical analysis on the baseline. Moreover, we find the improvements from asymmetric designs generalize well to longer training schedules, multiple other frameworks and newer backbones. Finally, the combined effect of several asymmetric designs achieves a state-of-the-art accuracy on ImageNet linear probing and competitive results on downstream transfer. We hope our exploration will inspire more research in exploiting asymmetry for Siamese representation learning.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Many recent self-supervised frameworks for visual representation learning are based on certain forms of Siamese networks. | background 1.00 | 0.03 | 0.01 | asserted 0.99 |
| s1 | Such networks are conceptually symmetric with two parallel encoders, but often practically asymmetric as numerous mechanisms are devised to break the symmetry. | background 0.99 | 0.07 | 0.03 | asserted 0.95 |
| s2 | In this work, we conduct a formal study on the importance of asymmetry by explicitly distinguishing the two encoders within the network -- one produces source encodings and the other targets. | contribution 0.86 | 0.94 | 0.06 | asserted 0.90 |
| s3 | Our key insight is keeping a relatively lower variance in target than source generally benefits learning. | interpretation 0.59 | 0.86 | 0.88 | asserted 0.82 |
| s4 | This is empirically justified by our results from five case studies covering different variance-oriented designs, and is aligned with our preliminary theoretical analysis on the baseline. | result 0.84 | 0.96 | 0.17 | asserted 0.97 |
| s5 | Moreover, we find the improvements from asymmetric designs generalize well to longer training schedules, multiple other frameworks and newer backbones. | result 1.00 | 0.96 | 0.12 | asserted 1.00 |
| s6 | Finally, the combined effect of several asymmetric designs achieves a state-of-the-art accuracy on ImageNet linear probing and competitive results on downstream transfer. | result 1.00 | 0.95 | 0.24 | asserted 1.00 |
| s7 | We hope our exploration will inspire more research in exploiting asymmetry for Siamese representation learning. | future work 1.00 | 0.07 | 0.03 | unclear 0.38 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Many recent self-supervised frameworks for visual representation learning are based on certain forms of Siamese networks. Such networks are conceptually symmetric with two parallel encoders, but often practically asymmetric as numerous mechanisms are devised to break the symmetry. In this work, we conduct a formal study on the importance of asymmetry by explicitly distinguishing the two encoders within the network -- one produces source encodings and the other targets. Our key insight is keeping a relatively lower variance in target than source generally benefits learning. This is empirically justified by our results from five case studies covering different variance-oriented designs, and is aligned with our preliminary theoretical analysis on the baseline. Moreover, we find the improvements from asymmetric designs generalize well to longer training schedules, multiple other frameworks and newer backbones. Finally, the combined effect of several asymmetric designs achieves a state-of-the-art accuracy on ImageNet linear probing and competitive results on downstream transfer. We hope our exploration will inspire more research in exploiting asymmetry for Siamese representation learning.
Human review
How the literature uses it
3 arXiv papers cite this in 5 sentences
Papers citing it · 1 sentence without a confident label
Cites it as support 1 paper
-
However, it is actually consistent with the theoretical analysis in [ref] that the in-training (source) network of contrastive learning requires high variance of input distribution, where the masking strategy already introduces much higher variance than the full input for the momentum (target) network.
Masked Image Modeling with Denoising Contrast · Analysis of Asymmetric Designs · 1.00
Cites it as background 2 papers
-
First introduced in [ref] , the siamese network [ref] is one promising approach among many self-supervised learning approaches and outperforms supervised counterparts in many visual benchmarks.
Masked Siamese ConvNets · Introduction · 0.83
-
It becomes the widely used technique in practices for most of fields ranging from optimization [ref] , reinforcement learning [ref] , knowledge distillation [ref] , recent semi-supervised learning frameworks [ref] , and self-supervised learning methods [ref] .
On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learning · Related Works · 0.95
What it cites · 87 sentences without a confident label
Uses the method or tool of 10 papers
-
We find a simple change of InfoNCE [ref] temperature from $0.1$ to $0.2$ can roughly compensate for this gap.
Representation Learning with Contrastive Predictive Coding · More Implementation Details · 0.82
-
Inspired by SimCLR [ref] , each MoCo v2 encoder further uses a projection head (projector), which is a 2-layer MLP without Batch Normalization (BN) [ref] in-between.
A Simple Framework for Contrastive Learning of Visual Representations · Case Studies for Source-Target Variance · 0.87
-
By default on ImageNet, we use LARS [ref] optimizer with batch size 4096, initial learning rate $lr=$1.6 (linearly scaled [ref] ), weight decay 0 and train the classifier for 90 epochs with a half-cycle cosine schedule following SimSiam [ref] .
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour · Implementation Details · 1.00
Compares with 6 papers
-
Compared to vanilla MoCo v2 [ref] , our baseline is generally better in linear probing on ImageNet [ref] (detailed in <ref>).
Improved Baselines with Momentum Contrastive Learning · Case Studies for Source-Target Variance · 0.98
-
Our model performs competitively against the most recent NNCLR [ref] ), achieving best on 5 tasks but lags behind on ones with iconic images.
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations · Transfer Learning · 1.00
-
We find our model indeed achieves state-of-the-art (56.8%), slightly better than SwAV [ref] which also used .
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments · Transfer Learning · 1.00
Cites as background 6 papers
-
Deviating from the original meaning of `Siamese', we explicitly mark the two encoders within the network functionally different: a source encoder and a target encoder.[Depending on the context, source has also been referred as query/online/student; and target as key/teacher in the literature [ref] .] The source encoder generates source encodings, and updates its weights via normal gradient-based optimization like in supervised learning.
Bootstrap your own latent: A new approach to self-supervised Learning · Introduction · 0.82
-
Despite different motivations and formulations, many recent un-/self-supervised methods for visual representation learning [ref] are based on certain forms of Siamese networks [ref] .
A Simple Framework for Contrastive Learning of Visual Representations · Introduction · 0.97
-
Despite different motivations and formulations, many recent un-/self-supervised methods for visual representation learning [ref] are based on certain forms of Siamese networks [ref] .
Barlow Twins: Self-Supervised Learning via Redundancy Reduction · Introduction · 0.97
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.