Pith. sign in

REVIEW 2 major objections 4 minor 73 references

This paper argues that the instability of multi-crop training in predictor-based Siamese self-supervised learning (BYOL, SimSiam, MoCo v3) is caused by a single shared predictor being forced to align representations of very different view t

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:04 UTC pith:3VCQT6VV

load-bearing objection Genuinely useful SSL recipe with consistent gains, but the paper's central claim about shared-predictor interference is not directly tested. the 2 major comments →

arxiv 2602.05845 v2 pith:3VCQT6VV submitted 2026-02-05 cs.CV

Self-Supervised Learning with a Multi-Task Latent Space Objective

Pierre-Fran\c{c}ois De Plaen , Abhishek Jha , Luc Van Gool , Tinne Tuytelaars , Marc Proesmans This is my paper
classification cs.CV
keywords self-supervised learningSiamese networksmulti-croppredictor-based SSLBYOLSimSiamMoCo v3cutout views
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that a single shared predictor — the small MLP head that aligns online and target representations in BYOL, SimSiam, and MoCo v3 — is why the multi-crop augmentation strategy (adding multiple small local crops) destabilizes these methods. The authors' central move is to treat each spatial transformation (global crop, local crop, masked cutout) as a separate alignment task solved by its own dedicated predictor over a shared encoder. They show that this per-view-type decoupling stabilizes multi-crop training and yields consistent linear-evaluation gains of roughly 3.8–4 points across the three frameworks with a ResNet-50 on ImageNet. Adding asymmetric cutout views — where the online view is partially masked and the target is complete — improves performance further, giving a multi-task BYOL of 76.7% linear accuracy on ImageNet in 800 epochs, exceeding the 1000-epoch BYOL baseline of 74.3%. If correct, the paper provides both a mechanistic explanation for a well-known training failure and a simple, backbone-agnostic recipe that turns spatial augmentations into complementary supervision signals.

Core claim

The paper's central claim is that the predictor in asymmetric Siamese architectures must specialize by view type. Because global and local crops differ strongly in scale and content, a single predictor is asked to solve heterogeneous alignment tasks at once, and the resulting interference makes optimization unstable. Splitting the predictor into one head per view type — global, local, and cutout — removes that interference without changing the loss, the backbone, or the EMA/stop-gradient mechanics. The loss becomes a weighted sum over view types of the squared L2 distance between each view-specific prediction and the global target representation. The authors further show that spatial transfo

What carries the argument

View-specific prediction heads — one MLP per view type, sharing the backbone and projection head. Each view type (global, local, cutout) defines its own alignment task, and its predictor maps the online representation to the target representation space. This removes the cross-view interference that the paper identifies as the cause of multi-crop instability, and it makes each spatial transformation a controllable, additive supervision signal.

Load-bearing premise

The paper attributes multi-crop instability to the shared predictor based on a comparison of MoCo v2 vs v3; if the gains actually come from the added parameters, changed BatchNorm statistics, or implicit regularization of the per-view heads, the interference explanation collapses even if the empirical fix still works.

What would settle it

Train a naive multi-crop BYOL where the shared predictor is widened to match the total parameter count of the per-view predictors, keeping all other settings (including BatchNorm statistics) identical. If this widened shared predictor reproduces the multi-crop gains and stability, the shared-predictor interference explanation is wrong; if it remains unstable and underperforms, the per-view specialization is the causal factor.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multi-crop, previously unstable for predictor-based Siamese methods, becomes a usable and beneficial augmentation for BYOL, SimSiam, and MoCo v3, with consistent ~4-point linear-evaluation gains on ImageNet (ResNet-50, 200-epoch training).
  • Adding an asymmetric cutout view to the multi-task objective further improves representations: multi-task BYOL reaches 76.7% linear accuracy on ImageNet (ResNet-50, 800 epochs), surpassing the 1000-epoch BYOL baseline (74.3%) and matching or exceeding clustering methods like SwAV and DINO.
  • The improvements transfer across backbones (ViT-S and ViT-B) and to dense tasks: multi-task BYOL outperforms supervised pre-training on COCO object detection and segmentation (41.8 vs 39.0 AP detection; 38.0 vs 35.4 AP segmentation).
  • The formulation is computationally efficient: the 200-epoch multi-task model (72 h on 4×A100) outperforms the 1000-epoch baseline (219 h) by 1.3%, roughly a 3× wall-clock efficiency gain.
  • Semi-supervised fine-tuning with 1% and 10% of ImageNet labels shows multi-task BYOL at or near the state of the art, matching a compressive BYOL variant and outperforming standard BYOL.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the shared-predictor interference story is right, per-view predictors should be viewed not as a fix but as a design principle: any new spatial or geometric transformation can be added as a new task with its own head, without retuning the optimization. This suggests a modular, expandable pre-training recipe.
  • The paper's own conditioning idea — feeding view-specific information (e.g., cutout coordinates) into the predictor — points toward a general joint-embedding predictive architecture, where the predictor becomes a conditional model rather than a fixed map. That could be tested directly.
  • The framework's success with a single masked-view task on images suggests analogous asymmetric tasks for other modalities — masked frames in video, masked patches in point clouds — where the target remains complete.
  • Because spatial augmentations dominate the learning signal, a natural extension is to ablate which spatial tasks are most complementary; the paper's own negative results (rotation, patch shuffling, CutMix-like blending) already map a boundary of what works.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes a multi-task formulation of predictor-based Siamese SSL in which each spatial view type (global, local, cutout) is handled by its own predictor head on top of a shared encoder. It argues that multi-crop instability in BYOL, SimSiam, and MoCo v3 is caused by a shared predictor forced to align heterogeneous views simultaneously, and that view-specific predictors stabilize training, yielding linear-evaluation gains of 3.8–4.0 points. It further introduces asymmetric cutout views as a complementary spatial task. Experiments on ImageNet with ResNet-50 and ViT-S/B and transfer to COCO report consistent gains over the corresponding baselines, with code released.

Significance. If the causal claim held, the paper would offer a simple, broadly applicable fix for multi-crop in predictor-based SSL and a clean design principle. The manuscript has notable strengths: code release; controlled within-framework comparisons in Tables 1 and 2; a useful ablation of view composition in Table 5; a timing analysis in Appendix C; and transparent negative results in Appendix D. However, the central explanation is not yet supported because the key baseline—naive multi-crop with a shared predictor under the paper's own settings—is missing. The empirical recipe may be valuable, but as written the main attribution requires additional experiments.

major comments (2)
  1. [Sec. 3.2, Sec. 4.1, Table 1] The central claim that 'the shared predictor itself is the source of instability' is not tested by the paper's controlled experiments. Table 1 compares the per-view-predictor multi-crop variant against the two-view baseline, not against a multi-crop run with a shared predictor under the same 200-epoch settings. The only evidence cited in Sec. 3.2 is the MoCo v2-vs-v3 comparison, but those architectures differ by both the removal of the memory bank and the introduction of the predictor+EMA, so the predictor is not isolated. The reported +3.8–4.0 points could therefore be attributed to the added local views, the extra predictor parameters, per-head BatchNorm statistics, or implicit regularization. Please add a naive shared-predictor multi-crop baseline for at least BYOL and MoCo v3 under the exact recipes used here, together with a training-stability diagnostic (e.g., loss trajectory or gr
  2. [Sec. 4.3, Appendix C, Table 9] The 'matched compute' comparison for the cutout increment is not supported by the reported timings. Table 9 gives 21:35 per epoch for multi-task (2 global + 2 local + 1 cutout) versus 19:51 for multi-predictor multi-crop (2 global + 4 local), so the multi-task run is about 8.5% more expensive per epoch despite using one fewer view. Thus the +0.9–1.4 point gain from cutout in Table 2 may be confounded with compute or implementation cost. Please provide a compute-matched comparison (e.g., equal per-epoch FLOPs/wall-clock via adjusted view budget or batch size) or explicitly soften the claim.
minor comments (4)
  1. [Sec. 3.2] The sentence 'Since BYOL and MoCo v3 differ only in their loss objectives' is inaccurate: BYOL and MoCo v3 differ in loss and in several recipe details, and the relevant comparison in this passage is between MoCo v2 and v3. Please correct or clarify.
  2. [Eq. (1)] State explicitly that the target z_glob is detached (stop-gradient/EMA) for all view types. The text implies this, but the equation and surrounding notation do not specify it.
  3. [Sec. 4.2] The phrase 'close to random performance (3.4%)' is imprecise because random-chance top-1 on ImageNet is 0.1%. Please describe 3.4% as a collapsed or weak-feature baseline rather than random.
  4. [Table 6] The cutout-only row is trained for 100 epochs while the caption says 'Models are trained for 200 epochs, except the cutout-only variant.' Please note the epoch count directly in the table for readability.

Circularity Check

0 steps flagged

No circularity: empirical gains are measured, not derived from fitted inputs; causal explanation is confounded but not definitionally circular.

full rationale

The paper's central contribution is an empirical architecture change: one predictor per view type plus optional cutout views. The loss in Eq. (1) is a straightforward weighted MSE over view-specific predictors and does not encode or presuppose the reported gains. The +3.8-4.0 linear-evaluation improvements and the 76.7% ImageNet result are measured outputs of pre-training runs, not quantities recovered from fitted parameters or from the paper's own equations. The explanatory claim that a shared predictor is the source of multi-crop instability is inferred from the MoCo v2-vs-v3 comparison and from cited prior reports, not from the method's definition; even if that causal attribution is confounded or untested, that is a correctness/validity concern, not circularity. The paper does not define 'shared predictor interference' as the observed failure, nor does it derive the gains from that claim. The per-view predictor weights are hand-chosen equal (λ_glob = λ_loc = λ_cutout) rather than fitted to force the reported numbers. The only self-citation that appears, ADM [15] in Related Work, is a non-load-bearing example of a non-contrastive method and does not support any load-bearing premise. There is no uniqueness theorem imported from prior work, no fitted input renamed as a prediction, and no ansatz smuggled in via citation. The method is validated against external baselines and standard benchmarks, so the derivation chain is self-contained in the sense relevant to circularity analysis.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

No physically invented entities. The view-specific predictors are architectural components, not postulates with independent evidence requirements. The central claim rests on two hand-chosen hyperparameters (equal weights, cutout size range) and on the unverified causal hypothesis about the shared predictor.

free parameters (2)
  • view task weights λ_v = 1/3 each (equal)
    Eq. (1) weights global, local, cutout losses equally; no ablation or tuning is reported, so the contribution of each task to the total loss is a hand-chosen setting that could affect the balance and results.
  • cutout area range = 20%–40% of image area
    Appendix A: random cutout area sampled uniformly between 20% and 40% of image area. This range is chosen by hand (following Torchvision's RandomResizedCrop sampler) and is an augmentation hyperparameter of the new view type.
axioms (3)
  • domain assumption The representation of a global crop (z_glob) is a valid and sufficient regression target for local and cutout views
    Eq. (1) uses the same global target for all view types. The paper does not compare against per-view targets; if local/cutout representations should align to their own contextual target, the multi-task framing would be different.
  • ad hoc to paper Multi-crop instability in predictor-based methods is caused by interference in the shared predictor
    Sec. 3.2 infers this from the MoCo v2-vs-v3 comparison and from the success of the fix, but never runs a naive multi-crop shared-predictor baseline in the same controlled setting. This is the paper's central hypothesis, assumed as the explanation.
  • domain assumption Spatial augmentations dominate the SSL learning signal
    Sec. 5.2 and Tab. 6 motivate cutout views by showing cropping alone is strong and removing crop collapses accuracy. This supports the design choice to add spatial tasks, but it is an empirical claim about the SSL objective, not a derivation.

pith-pipeline@v1.3.0-alltime-deepseek · 16498 in / 10627 out tokens · 106343 ms · 2026-08-03T04:04:07.650725+00:00 · methodology

0 comments
read the original abstract

We propose a multi-task formulation of self-predictive Siamese SSL in which each spatial transformation defines a distinct latent-space alignment task, solved by a dedicated predictor over a shared encoder. This perspective directly explains a long-standing failure of multi-crop training in self-predictive methods such as BYOL, SimSiam, and MoCo v3: a shared predictor is forced to solve heterogeneous alignment tasks simultaneously, leading to unstable optimization. Assigning one predictor per view type resolves this interference, unlocking linear evaluation gains of 3.8-4\% across frameworks. This perspective also suggests a principled way to enrich pre-training by introducing additional spatial transformations as complementary tasks. We demonstrate this by introducing asymmetric cutout views, in which a masked online view is aligned with a complete target, forming a semantic inpainting objective. The resulting framework is stable, backbone-agnostic, and consistently improves the performance of ResNet and ViT models on ImageNet and COCO.

Figures

Figures reproduced from arXiv: 2602.05845 by Abhishek Jha, Luc Van Gool, Marc Proesmans, Pierre-Fran\c{c}ois De Plaen, Tinne Tuytelaars.

Figure 1
Figure 1. Figure 1: Overview of our predictor-based Siamese SSL framework. Naive multi-crop (left) is unstable under a shared predictor. Using [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed framework. Each image is augmented into multiple spatial views: global (views A), local (views [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Asymmetric and symmetric cutout. Image from Ima [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

73 extracted references · 14 linked inside Pith

  1. [1]

    Masked siamese net- works for label-efficient learning

    Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bo- janowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese net- works for label-efficient learning. InECCV, pages 456–473. Springer, 2022. 2

    2022

  2. [2]

    Self-supervised learning from images with a joint-embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bo- janowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. InCVPR, pages 15619–15629, 2023. 1, 2, 3, 6

    2023

  3. [3]

    See- ing the whole in the parts in self-supervised representation learning.arXiv preprint arXiv:2501.02860, 2025

    Arthur Aubret, C ´eline Teuli`ere, and Jochen Triesch. See- ing the whole in the parts in self-supervised representation learning.arXiv preprint arXiv:2501.02860, 2025. 2, 3

    Pith/arXiv arXiv 2025

  4. [4]

    Multimae: Multi-modal multi-task masked autoen- coders

    Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi-task masked autoen- coders. InECCV, pages 348–367. Springer, 2022. 3

    2022

  5. [5]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. InICLR, 2021. 2, 6

    2021

  6. [6]

    Vi- creg: Variance-invariance-covariance regularization for self- supervised learning

    Adrien Bardes, Jean Ponce, and Yann LeCun. Vi- creg: Variance-invariance-covariance regularization for self- supervised learning. InICLR, 2021. 2

    2021

  7. [7]

    Deep clustering for unsupervised learning of visual features

    Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. InECCV, pages 132–149, 2018. 2

    2018

  8. [8]

    Unsupervised learn- ing of visual features by contrasting cluster assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Pi- otr Bojanowski, and Armand Joulin. Unsupervised learn- ing of visual features by contrasting cluster assignments. NeurIPS, 33:9912–9924, 2020. 1, 2, 3, 6, 12

    2020

  9. [9]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, pages 9650–9660, 2021. 1, 2, 3, 6, 12, 13

    2021

  10. [10]

    A simple framework for contrastive learn- ing of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learn- ing of visual representations. InICML, pages 1597–1607. PmLR, 2020. 1, 2, 3, 4, 6, 8, 13

    2020

  11. [11]

    Exploring simple siamese rep- resentation learning

    Xinlei Chen and Kaiming He. Exploring simple siamese rep- resentation learning. InCVPR, pages 15750–15758, 2021. 1, 2, 3, 6

    2021

  12. [12]

    Improved baselines with momentum contrastive learning

    Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020. 4, 6

    Pith/arXiv arXiv 2003

  13. [13]

    An empiri- cal study of training self-supervised vision transformers

    Xinlei Chen, Saining Xie, and Kaiming He. An empiri- cal study of training self-supervised vision transformers. In ICCV, pages 9640–9649, 2021. 1, 2, 3, 6

    2021

  14. [14]

    Cluster and predict la- tent patches for improved masked image modeling.CoRR,

    Timoth ´ee Darcet, Federico Baldassarre, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Cluster and predict la- tent patches for improved masked image modeling.CoRR,

  15. [15]

    Adversarial dependence minimiza- tion.arXiv preprint arXiv:2502.03227, 2025

    Pierre-Franc ¸ois De Plaen, Tinne Tuytelaars, Marc Proes- mans, and Luc Van Gool. Adversarial dependence minimiza- tion.arXiv preprint arXiv:2502.03227, 2025. 2

    Pith/arXiv arXiv 2025

  16. [16]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InCVPR, pages 248–255. IEEE, 2009. 5, 6

    2009

  17. [17]

    Improved regular- ization of convolutional neural networks with cutout.arXiv preprint arXiv:1708.04552, 2017

    Terrance DeVries and Graham W Taylor. Improved regular- ization of convolutional neural networks with cutout.arXiv preprint arXiv:1708.04552, 2017. 1, 4

    Pith/arXiv arXiv 2017

  18. [18]

    Multi-task self- supervised visual learning

    Carl Doersch and Andrew Zisserman. Multi-task self- supervised visual learning. InICCV, pages 2051–2060,

    2051

  19. [19]

    Unsuper- vised visual representation learning by context prediction

    Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsuper- vised visual representation learning by context prediction. In ICCV, pages 1422–1430, 2015. 2

    2015

  20. [20]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InICLR, 2021. 2, 6

    2021

  21. [21]

    Whitening for self-supervised representation learning

    Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. InICML, pages 3015–3024. PMLR, 2021. 2

    2021

  22. [22]

    Multimodal masked autoen- coders learn transferable representations.arXiv preprint arXiv:2205.14204, 2022

    Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel. Multimodal masked autoen- coders learn transferable representations.arXiv preprint arXiv:2205.14204, 2022. 3

    Pith/arXiv arXiv 2022

  23. [23]

    Un- supervised representation learning by predicting image rota- tions

    Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Un- supervised representation learning by predicting image rota- tions. InICLR, 2018. 2

    2018

  24. [24]

    Accurate, large mini- batch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677, 2017

    Priya Goyal, Piotr Doll ´ar, Ross Girshick, Pieter Noord- huis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large mini- batch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677, 2017. 6

    Pith/arXiv arXiv 2017

  25. [25]

    Bootstrap your own latent-a new ap- proach to self-supervised learning.NeurIPS, 33:21271– 21284, 2020

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new ap- proach to self-supervised learning.NeurIPS, 33:21271– 21284, 2020. 1, 2, 3, 6, 8, 13

    2020

  26. [26]

    Unsupervised multi-task feature learning on point clouds

    Kaveh Hassani and Mike Haley. Unsupervised multi-task feature learning on point clouds. InICCV, pages 8160–8171,

  27. [27]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InCVPR, pages 770–778, 2016. 5, 6

    2016

  28. [28]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. InCVPR, pages 9729–9738, 2020. 1, 2, 3, 6

    2020

  29. [29]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InCVPR, pages 16000–16009, 2022. 1, 2, 6, 7

    2022

  30. [30]

    Batch normalization: Accelerating deep network training by reducing internal co- variate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal co- variate shift. InICML, pages 448–456. PMLR, 2015. 6

    2015

  31. [31]

    Multi- modal contrastive masked autoencoders: A two-stage pro- gressive pre-training approach for rgbd datasets

    Muhammad Abdullah Jamal and Omid Mohareri. Multi- modal contrastive masked autoencoders: A two-stage pro- gressive pre-training approach for rgbd datasets. InCVPR, pages 17947–17957, 2025. 3 9

    2025

  32. [32]

    Adam: A method for stochastic opti- mization.arXiv preprint arXiv:1412.6980, 2014

    Diederik P Kingma. Adam: A method for stochastic opti- mization.arXiv preprint arXiv:1412.6980, 2014. 6

    Pith/arXiv arXiv 2014

  33. [33]

    Global-local self- distillation for visual representation learning

    Tim Lebailly and Tinne Tuytelaars. Global-local self- distillation for visual representation learning. InWACV, pages 1441–1450, 2023. 3

    2023

  34. [34]

    Compressive visual representations

    Kuang-Huei Lee, Anurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. NeurIPS, 34:19538–19552, 2021. 6, 13

    2021

  35. [35]

    Continuous control with deep reinforcement learning.arXiv preprint arXiv:1509.02971, 2015

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning.arXiv preprint arXiv:1509.02971, 2015. 6

    Pith/arXiv arXiv 2015

  36. [36]

    Ms2l: Multi-task self-supervised learning for skeleton based action recognition

    Lilang Lin, Sijie Song, Wenhan Yang, and Jiaying Liu. Ms2l: Multi-task self-supervised learning for skeleton based action recognition. InProceedings of the 28th ACM international conference on multimedia, pages 2490–2498, 2020. 3

    2020

  37. [37]

    A closer look at benchmarking self-supervised pre- training with image classification.IJCV, pages 1–13, 2025

    Markus Marks, Manuel Knott, Neehar Kondapaneni, Eli- jah Cole, Thijs Defraeye, Fernando Perez-Cruz, and Pietro Perona. A closer look at benchmarking self-supervised pre- training with image classification.IJCV, pages 1–13, 2025. 7

    2025

  38. [38]

    Object-aware cropping for self- supervised learning.arXiv preprint arXiv:2112.00319, 2021

    Shlok Mishra, Anshul Shah, Ankan Bansal, Abhyuday Jagannatha, Janit Anjaria, Abhishek Sharma, David Ja- cobs, and Dilip Krishnan. Object-aware cropping for self- supervised learning.arXiv preprint arXiv:2112.00319, 2021. 3

    Pith/arXiv arXiv 2021

  39. [39]

    An embedding-dynamic approach to self-supervised learning

    Suhong Moon, Domas Buracas, Seunghyun Park, Jinkyu Kim, and John Canny. An embedding-dynamic approach to self-supervised learning. InWACV, pages 2750–2758, 2023. 2, 3

    2023

  40. [40]

    Augmentations vs algorithms: What works in self- supervised learning.arXiv preprint arXiv:2403.05726, 2024

    Warren Morningstar, Alex Bijamov, Chris Duvarney, Luke Friedman, Neha Kalibhat, Luyang Liu, Philip Mans- field, Renan Rojas-Gomez, Karan Singhal, Bradley Green, et al. Augmentations vs algorithms: What works in self- supervised learning.arXiv preprint arXiv:2403.05726, 2024. 1, 2, 3, 4

    Pith/arXiv arXiv 2024

  41. [41]

    You don’t need domain-specific data augmentations when scaling self- supervised learning.NeurIPS, 37:116106–116125, 2024

    Th ´eo Moutakanni, Maxime Oquab, Marc Szafraniec, Maria Vakalopoulou, and Piotr Bojanowski. You don’t need domain-specific data augmentations when scaling self- supervised learning.NeurIPS, 37:116106–116125, 2024. 3, 8

    2024

  42. [42]

    Unsupervised learning of visual representations by solving jigsaw puzzles

    Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. InECCV, pages 69–84. Springer, 2016. 2

    2016

  43. [43]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 1, 8

    Pith/arXiv arXiv 2023

  44. [44]

    Context encoders: Feature learning by inpainting

    Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. InCVPR, pages 2536–2544, 2016. 2

    2016

  45. [45]

    Dinov3.arXiv preprint arXiv:2508.10104, 2025

    Oriane Sim ´eoni, Huy V V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Micha ¨el Ramamonjisoa, et al. Dinov3.arXiv preprint arXiv:2508.10104, 2025. 1

    Pith/arXiv arXiv 2025

  46. [46]

    Branching out for better byol

    Azad Singh and Deepak Mishra. Branching out for better byol. InNeurIPS 2021 Workshop on Self-Supervised Learn- ing: Theory and Practice, 2021. 3

    2021

  47. [47]

    Un- derstanding self-supervised learning dynamics without con- trastive pairs

    Yuandong Tian, Xinlei Chen, and Surya Ganguli. Un- derstanding self-supervised learning dynamics without con- trastive pairs. InICML, pages 10268–10278. PMLR, 2021. 1, 4

    2021

  48. [48]

    Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Hol- ger Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on ima- genet? InFirst Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022, 2022. 6

    2022

  49. [49]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. InICML, pages 10347–10357. PMLR, 2021. 6

    2021

  50. [50]

    How to train state-of-the-art mod- els using torchvision’s latest primitives.https : / / pytorch

    Vasilis Vryniotis. How to train state-of-the-art mod- els using torchvision’s latest primitives.https : / / pytorch . org / blog / how - to - train - state - of - the - art - models - using - torchvision - latest-primitives/, 2021. 6

    2021

  51. [51]

    Adaptive multi-head contrastive learning

    Lei Wang, Piotr Koniusz, Tom Gedeon, and Liang Zheng. Adaptive multi-head contrastive learning. InEuropean Con- ference on Computer Vision, pages 404–421. Springer, 2024. 3

    2024

  52. [52]

    Dense contrastive learning for self-supervised visual pre-training

    Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. InCVPR, pages 3024–3033, 2021. 3

    2021

  53. [53]

    On the importance of asymmetry for siamese representation learning

    Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen. On the importance of asymmetry for siamese representation learning. InCVPR, pages 16570– 16579, 2022. 4

    2022

  54. [54]

    Region similarity representation learn- ing

    Tete Xiao, Colorado J Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learn- ing. InICCV, pages 10539–10548, 2021. 3

    2021

  55. [55]

    Detco: Unsu- pervised contrastive learning for object detection

    Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsu- pervised contrastive learning for object detection. InICCV, pages 8392–8401, 2021. 3

    2021

  56. [56]

    Simmim: A simple framework for masked image modeling

    Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. InCVPR, pages 9653–9663, 2022. 2, 6

    2022

  57. [57]

    Comae: Single model hybrid pre-training on small-scale rgb- d datasets

    Jiange Yang, Sheng Guo, Gangshan Wu, and Limin Wang. Comae: Single model hybrid pre-training on small-scale rgb- d datasets. InAAAI, pages 3145–3154, 2023. 3

    2023

  58. [58]

    Large batch training of convolutional networks.arXiv preprint arXiv:1708.03888, 2017

    Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks.arXiv preprint arXiv:1708.03888, 2017. 6

    Pith/arXiv arXiv 2017

  59. [59]

    Cutmix: Regu- larization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu- larization strategy to train strong classifiers with localizable features. InICCV, pages 6023–6032, 2019. 3, 14

    2019

  60. [60]

    Barlow twins: Self-supervised learning via redundancy reduction

    Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St´ephane Deny. Barlow twins: Self-supervised learning via redundancy reduction. InICML, pages 12310–12320. PMLR, 2021. 1, 2, 6 10

    2021

  61. [61]

    S4l: Self-supervised semi-supervised learning

    Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lu- cas Beyer. S4l: Self-supervised semi-supervised learning. In ICCV, pages 1476–1485, 2019. 13

    2019

  62. [62]

    mixup: Beyond empirical risk minimiza- tion.arXiv preprint arXiv:1710.09412, 2017

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion.arXiv preprint arXiv:1710.09412, 2017. 3

    Pith/arXiv arXiv 2017

  63. [63]

    Color- ful image colorization

    Richard Zhang, Phillip Isola, and Alexei A Efros. Color- ful image colorization. InECCV, pages 649–666. Springer,

  64. [64]

    Leverage your local and global represen- tations: A new self-supervised learning strategy

    Tong Zhang, Congpei Qiu, Wei Ke, Sabine S ¨usstrunk, and Mathieu Salzmann. Leverage your local and global represen- tations: A new self-supervised learning strategy. InCVPR, pages 16580–16589, 2022. 2

    2022

  65. [65]

    Random erasing data augmentation

    Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. InAAAI, pages 13001–13008, 2020. 3

    2020

  66. [66]

    Image bert pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. InICLR, 2021. 2 11 Supplementary Material A. Experimental Settings In this section, we detail the implementation settings re- quired to reproduce the results reported in Secs. 4.1 to 4.4, 5.1 and 5.2. Unless specified otherwi...

    2021

  67. [67]

    For SimSiam and MoCo v3, the minimum area is set to 20%

    random resized cropping: a random patch of the image is selected, with an area uniformly sampled between 8% and 100% of that of the original image, and an aspect ratio logarithmically sampled between 3/4 and 4/3. For SimSiam and MoCo v3, the minimum area is set to 20%

  68. [68]

    random horizontal flipping with a probability of 50%

  69. [69]

    BYOL and MoCo v3 use the ranges(0.4,0.4,0.2,0.1), while SimSiam uses (0.4,0.4,0.4,0.1)

    random color jitter: brightness, contrast, saturation, and hue are perturbed with random offsets uniformly sampled for each image. BYOL and MoCo v3 use the ranges(0.4,0.4,0.2,0.1), while SimSiam uses (0.4,0.4,0.4,0.1)

  70. [70]

    random grayscale with a probability of 20%

  71. [71]

    In BYOL and MoCo v3, the transformation is applied with 100% probability for the first view and 10% for the second; in SimSiam, both views use a 50% probability

    random Gaussian blur: the image is blurred with a Gaus- sian blur kernel of size 23 and a standard deviation uni- formly sampled in[0.1,2]. In BYOL and MoCo v3, the transformation is applied with 100% probability for the first view and 10% for the second; in SimSiam, both views use a 50% probability

  72. [72]

    random solarization with probability 20% in the second view, for BYOL and MoCo v3

  73. [73]

    Multi-predictor multi-crop and multi-task strategies

    color normalization: finally, we normalize the color channels by subtracting the per-channel mean and divid- ing by the per-channel standard deviation estimated on the ImageNet training set. Multi-predictor multi-crop and multi-task strategies. All multi-crop and multi-task experiments reuse the hyper- parameters of their respective 2-view baselines to en...