Pith. sign in

REVIEW 3 major objections 5 minor 46 references

By shifting the anti-collapse force to a persistent embedding table, IConE keeps self-supervised training stable at batch size 1.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 18:06 UTC pith:5WPW3EII

load-bearing objection Practical extension of instance discrimination for small-batch SSL; solid experiments, but the B=1 claim is undercut by unspecified BatchNorm handling in the 3D ResNet. the 3 major comments →

arxiv 2603.15263 v2 pith:5WPW3EII submitted 2026-03-16 cs.CV cs.LG

IConE: Batch Independent Collapse Prevention for Self-Supervised Representation Learning

Konstantinos Almpanakis , Anna Kreshuk This is my paper
classification cs.CV cs.LG
keywords self-supervised learningjoint-embedding architecturerepresentation collapsesmall-batch traininginstance discriminationbiomedical imagingclass imbalancedimensional collapse
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces IConE, a self-supervised learning method that removes the batch-size dependence of collapse prevention. Instead of using in-batch negatives or batch statistics, IConE maintains a learnable embedding table with one anchor per training instance, regularized so the anchors stay spread on the hypersphere. The encoder only receives attractive gradients, pulling views together and toward their instance anchor, while all repulsive pressure is applied to the embedding table across the whole dataset. This makes training stable at batch sizes from 1 to 64, outperforming contrastive and non-contrastive baselines on 2D and 3D biomedical benchmarks, especially under severe class imbalance. If correct, this means small-batch regimes, common in high-dimensional scientific imaging, no longer require fundamentally different self-supervised objectives.

Core claim

IConE decouples the two objectives of joint-embedding learning: alignment is enforced in the encoder space through view-view and view-instance attraction, while uniformity is enforced in a persistent auxiliary embedding table via a squared-hinge diversity loss on the full Gram matrix. Because the encoder gradient contains only the attractive terms (Lvi and Lvv), the anti-collapse mechanism is completely externalized to the embedding table, making it mathematically independent of batch composition. Empirically, this yields nearly flat linear-probe accuracy across batch sizes 1 to 64, preserves high effective rank and RankMe/LiDAR scores where baselines collapse, and maintains minority-class a

What carries the argument

The central object is the instance embedding table E ∈ R^(N×d), a persistent set of learnable anchors, one per training sample. Diversity is enforced by a squared-hinge loss on the off-diagonal Gram matrix of the normalized table, Ldiv = (1/(N(N-1))) Σ_{i≠j} [max(0, G_ij)]^2. Its role is to provide a deterministic, dataset-level repulsive geometry that stays well-spread even when the current batch contains a single instance, while the encoder optimizes only the attractive view-view and view-anchor terms, as shown in Equation (6).

Load-bearing premise

The load-bearing premise is that minimizing the squared-hinge diversity loss over the embedding table yields a globally well-spread set of anchors, and that attracting the encoder only to these anchors is enough to keep the encoder's own representations non-collapsed on unseen data, even though N > d makes perfect orthogonality impossible.

What would settle it

Train IConE with batch size 1 on a dataset where the number of instances exceeds the embedding dimension, using a backbone with no batch normalization, and measure the effective rank of the encoder outputs on a held-out set. If the rank drops toward 1 despite the anchor table remaining well-spread, the claimed transfer of anti-collapse geometry from anchors to encoder fails; a more direct test is to freeze the well-spread anchor table and train only the Lvv term, which should collapse, showing that the attractive term itself cannot prevent collapse without the anchor-alignment signal.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Self-supervised learning can be made stable at batch size 1 without changing the backbone or adding a momentum encoder, enabling training on single high-dimensional volumes that saturate GPU memory.
  • The anti-collapse mechanism becomes a property of the dataset rather than of any batch, so class-imbalanced data no longer loses minority-class features merely because those classes are absent from small batches.
  • The same decoupling transfers to supervised learning, where a class-prototype table outperforms standard cross-entropy and supervised contrastive loss in small-batch and low-data regimes.
  • The gradient structure implies that the encoder is never directly repelled by other instances, which removes the temperature hyperparameter and the need for a large negative pool.
  • The diversity regularizer is modular: replacing it with variance-covariance or sketched-isotropic-Gaussian penalties preserves the batch-independence and scales better for larger datasets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct testable consequence of Equation (6) is that the encoder's representation rank should be controlled entirely by the anchor geometry; if the anchor table is well-spread, the encoder should stay high-rank even when trained on a single repeated instance, a prediction that could be checked on synthetic data with N much larger than d.
  • Because instances get fixed anchors and the encoder is trained to map views to those anchors, the method implicitly assumes the augmentation manifold of each instance is small enough to be represented by a single point; on datasets with very high intra-class diversity, a single anchor per instance may force the encoder to average incompatible views, which would show up as degraded linear-probe per
  • The paper's own limitation section notes the O(N^2) cost of the full-table Gram computation and proposes VCReg/SIGReg as scalable substitutes, but the B=1 results for the 3D ResNet leave batch-normalization statistics unspecified; since batch normalization typically computes statistics over the B×V forward passes, the effective batch at B=1 is just two views, and how that behaves is not analyzed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. IConE proposes a self-supervised learning framework for small-batch and data-scarce regimes. It augments a standard joint-embedding setup with a learnable per-instance embedding table E, regularized by an explicit diversity loss Ldiv that penalizes positive off-diagonal entries of the normalized Gram matrix. The encoder is trained only through attractive terms: view-view consistency Lvv and view-instance alignment Lvi, so its loss gradient is formally independent of batch size (Eq. 6). The paper presents linear-probe results on 2D MedMNIST (ViT-S) and 3D MedMNIST (3D ResNet-18) for batch sizes 1–64, representation-geometry metrics, class-imbalance experiments, a supervised extension, plus synthetic ablations isolating the role of each loss component.

Significance. If the claims hold, IConE is a useful addition to small-batch SSL, especially for memory-constrained 3D biomedical data. The paper is thorough in several respects: identical backbones/augmentations across baselines, five seeds, detailed hyperparameters, per-dataset results, regularizer-modularity ablations, and explicit GPU memory/wall-clock scaling measurements. The synthetic experiment cleanly shows that all three loss terms are necessary. The core idea of externalizing the anti-collapse mechanism to a persistent parameter space is simple and structurally appealing. However, the strength of the central empirical claim depends on resolving the unspecified BatchNorm behavior in the 3D architecture and on adding the missing no-BN BYOL baseline; the paper also overstates the guarantee provided by Ldiv.

major comments (3)
  1. [Sec. 4.2 / Fig. 2B / Appendix A.1] The 3D encoder is a 3D ResNet-18, which contains BatchNorm3d layers. The manuscript never states how batch statistics are handled at B=1. Since V=2 views are generated, the effective BN batch is 2 samples — both are augmentations of the same instance. If train-mode BN is used, the encoder's activations and gradients depend on this 2-view batch, so the 'batch-independent' claim is not established for the 3D experiments; the reported B=1 stability in Fig. 2B and Fig. 4 cannot be attributed solely to the decoupled loss in Eq. (6). If running statistics were frozen, or BN replaced by a batch-independent normalization, that is a major unreported implementation change. Please specify the exact BN configuration for all batch sizes and either provide experiments with a batch-independent normalization or restrict the claim to the loss function.
  2. [Sec. 4.2 / Appendix A.4] The BYOL baseline is implemented with its standard BatchNorm. Richemond et al. [34] is cited in the related work but never included in the experiments; removing BN is precisely what makes BYOL more robust to small batches. Without a BYOL-no-BN comparison (or BYOL with a batch-independent normalization), the claim that IConE outperforms 'strong' baselines throughout the small-batch regime is incomplete. The results in Fig. 2 could be at least partially driven by the baseline's internal batch-statistic dependence rather than by IConE's decoupling mechanism.
  3. [Sec. 3.3, Eq. (3)] The text after Eq. (7) states that Ldiv 'guarantees a non-collapsed target geometry.' This is not justified. Ldiv is a squared hinge on positive off-diagonal Gram entries; it does not constrain negative correlations or directly maximize rank. When N > d, strict orthogonality is impossible, and there exist configurations with low effective rank that satisfy zero positive correlations (e.g., vectors lying in a subspace with all pairwise inner products non-positive). No formal argument shows that minimizing Eq. (3) yields a well-spread full-rank anchor set, nor that this property transfers to encoder representations on unseen instances. The empirical ablations support the mechanism, but the wording should be softened from 'guarantees' to 'encourages', or a rigorous statement with assumptions should be added.
minor comments (5)
  1. [Sec. 2] The method name is inconsistently typeset as 'SwA V' instead of 'SwAV' on multiple occasions.
  2. [Fig. 4] The 'No data' labels are unclear; specify that the loss is undefined at B=1 for SimCLR, VICReg, SwAV, MoCo-v2, and BYOL.
  3. [Appendix E] Typo: 'reducing linea probe accuracy' should be 'linear probe accuracy'.
  4. [Appendix C.1] The SIGReg formula uses t without stating the integration grid; the numerical approximation is mentioned but no step size or range is given. A short sentence describing the grid would improve reproducibility.
  5. [Abstract / Contributions] The phrase 'state-of-the-art' is used without comparison to recent large-scale SSL models; the evidence supports strong performance on a specific MedMNIST benchmark set, and the claim should be scoped accordingly.

Circularity Check

0 steps flagged

No load-bearing circularity: IConE's batch-agnostic anti-collapse is an explicit loss decomposition (Eq. 6), not a fitted prediction; the only self-citation ([7]) is non-load-bearing related-work framing.

full rationale

The central derivation is self-contained. Equation (5) defines L = Lvi + Lvv + Ldiv, and Eq. (6) states ∇θL = ∇θLvi + ∇θLvv, so the encoder gradient receives no repulsive term. This is a direct consequence of the loss definition, not a fitted or post-hoc prediction. Ldiv (Eq. 3) is optimized over the persistent table E and contributes only to ∇ei L (Eq. 7); therefore the claimed batch-size independence of the anti-collapse mechanism is structural rather than circular. No hyperparameter is tuned to produce the reported batch-size curves, and the geometric metrics (rank, RankMe, LiDAR, uniformity) are measured on the learned representations rather than imposed as targets in the loss. The closest thing to a self-citation is Ref. [7] in Section 2.1, used to support the general statement that batch-statistic SSL methods "require significant modification to function effectively [7, 23]"; it is related-work framing, not a load-bearing premise of IConE's derivation. The unspecified BatchNorm behavior at B=1 in the 3D ResNet-18 (Appendix A.1) is an empirical reproducibility/correctness gap, but it is not a circularity: it does not make Eq. (6) equivalent to an input or rename a fitted quantity as a prediction.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 1 invented entities

The central claim rests on the design of the embedding table and its regularizer. The only free hyperparameters are standard choices (number of views, init scale). No constant is fitted to make results match; the main auxiliary assumptions are domain-level and the sufficiency of the hinge regularizer.

free parameters (3)
  • number_of_views = 2
    Chosen as default after an ablation (Appendix C.2); set to 2 to minimize memory; not fitted but a design choice.
  • initialization_scale = 0.02
    Default from Appendix A.3; ablated in C.3 and robust across 0.02-0.50.
  • hinge_threshold_for_div = 0
    The squared hinge in Eq. (3) penalizes only positive off-diagonal correlations; the threshold 0 is chosen by hand without analysis.
axioms (3)
  • domain assumption The stochastic augmentation family T produces views whose shared semantics are relevant to downstream tasks.
    Standard SSL assumption invoked in Sec 3.1; if augmentations destroy class semantics, instance alignment to anchors would not transfer to label-relevant features.
  • domain assumption The encoder has sufficient capacity to learn an injective mapping from instances to anchors.
    At B=1 the encoder is trained with only attractive gradients; the method relies on the encoder being able to separate instances, which is not guaranteed for overlapping or extremely similar inputs.
  • ad hoc to paper The hinge regularizer on the normalized table is a sufficient anti-collapse mechanism.
    Ldiv only penalizes positive correlations; it is an ad hoc design choice and the paper's theoretical motivation is qualitative (uniformity from Wang & Isola), not a proof that the encoder avoids collapse.
invented entities (1)
  • Learnable instance embedding table E ∈ R^{N×d} no independent evidence
    purpose: Provides persistent, globally regularized anchors that define a dataset-level target geometry for the encoder, replacing batch-level negative sampling or variance statistics.
    The table is an architectural construct used only during training; its ability to prevent collapse is established only through within-paper experiments and ablations, not through external predictions.

pith-pipeline@v1.3.0-alltime-deepseek · 23449 in / 13447 out tokens · 132580 ms · 2026-08-02T18:06:43.497504+00:00 · methodology

0 comments
read the original abstract

Self-supervised learning (SSL) has revolutionized representation learning, with Joint-Embedding Architectures (JEAs) emerging as an effective approach for capturing semantic features. Existing JEAs rely on implicit or explicit batch interaction -- via negative sampling or statistical regularization -- to prevent representation collapse. This reliance becomes problematic in regimes where batch sizes must be small, such as high-dimensional scientific data, where memory constraints and class imbalance make large, well-balanced batches infeasible. We introduce IConE (Instance-Contrasted Embeddings), a framework that decouples collapse prevention from the training batch size. Rather than enforcing diversity through batch statistics, IConE maintains a global set of learnable auxiliary instance embeddings regularized by an explicit diversity objective. This transfers the anti-collapse mechanism from the transient batch to a dataset-level embedding space, allowing stable training even when batch statistics are unreliable, down to batch size 1. Across diverse 2D and 3D biomedical modalities, IConE outperforms strong contrastive and non-contrastive baselines throughout the small-batch regime (from B=1 to B=64) and demonstrates marked robustness to severe class imbalance. Geometric analysis shows that IConE preserves high intrinsic dimensionality in the learned representations, preventing the collapse observed in existing JEAs as batch sizes shrink.

Figures

Figures reproduced from arXiv: 2603.15263 by Anna Kreshuk, Konstantinos Almpanakis.

Figure 1
Figure 1. Figure 1: Decoupling invariance and anti-collapse gradients with IConE. Left: [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Batch-size stability. Mean and standard deviation linear-probe top-1 balanced accuracy aggregated over (A) 2D datasets and (B) 3D datasets. 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Performance Drop from Max to Min Batch Size (%) IConE SwAV DIET DINO MoCov2 BYOL SimCLR VICReg Method Batch Size Dependency (Lower = More Stable) 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Correlation(Batch Size, Performance) 0.66 0.68 … view at source ↗
Figure 3
Figure 3. Figure 3: Batch sensitivity analysis. Left [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: UMAP visualization of learned representations. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Representation geometry analysis. (A) Alignment vs. uniformity loss for 3D datasets, colored by downstream accuracy. (B) Accuracy vs. effective rank (3D). (C) RankMe vs. batch size (2D). (D) LiDAR vs. batch size (2D). IConE achieves superior geometry metrics across all measures. (A) Alignment vs. Uniformity. Following Wang & Isola [39], we plot alignment loss (how close positive pairs are) against uniformi… view at source ↗
Figure 6
Figure 6. Figure 6: Robustness to class imbalance. (A) Overall top-1 balanced accuracy vs. imbalance ratio. (B) Minority class accuracy vs. imbalance ratio. IConE maintains superior performance on both metrics, with particularly strong advantages on minority classes at high imbalance ratios. Results are averaged over all 2D and 3D datasets. 4.4 Robustness to Class Imbalance Class imbalance poses a critical challenge for batch… view at source ↗
Figure 7
Figure 7. Figure 7: Per-dataset 2D results. Linear probe balanced accuracy vs. batch size for each 2D MedMNIST dataset. In most datasets, IConE maintains consistent performance across all batch sizes, while other baselines show characteristic degradation patterns at small batch sizes. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Per-dataset 3D results. Linear probe balanced accuracy vs. batch size for each 3D MedMNIST dataset. IConE’s batch-agnostic design provides consistent advantages across all volumetric imaging tasks. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Joint batch size and dataset size analysis. [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Peak GPU memory (GB) across batch sizes and dataset sizes for all evaluated SSL methods on BloodMNIST. Each panel corresponds to a different batch size. IConE-SIGreg and IConE-VCReg match the memory footprint of competitive baselines across all settings, whereas the default IConE variant incurs increasing overhead as N grows. N=500 N=1k N=2k N=5k 0 2 4 6 8 Training Time (hours) Batch size = 1 N=500 N=1k N… view at source ↗
Figure 11
Figure 11. Figure 11: Wall-clock training time (hours) across batch sizes and dataset sizes for all evaluated SSL methods on BloodMNIST. The default IConE variant exhibits super-linear growth in training time with dataset size due to its O(N 2 ) diversity regularisation, while IConE-SIGreg and IConE￾VCReg scale comparably to the remaining baselines. D Representation Quality Metrics Beyond downstream task performance, we provid… view at source ↗
Figure 12
Figure 12. Figure 12: Representation quality metrics on 2D MedMNIST datasets. [PITH_FULL_IMAGE:figures/full_fig_p029_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Representation quality metrics on 3D MedMNIST datasets. [PITH_FULL_IMAGE:figures/full_fig_p030_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Final representations for each ablation variant. [PITH_FULL_IMAGE:figures/full_fig_p030_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Training evolution across all ablation variants. [PITH_FULL_IMAGE:figures/full_fig_p031_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Classification accuracy during training. [PITH_FULL_IMAGE:figures/full_fig_p032_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Alignment and uniformity metrics [39] during training. (a) Alignment loss (lower = tighter same-class clusters). (b) Uniformity loss (lower = more spread). Full IConE achieves the best trade-off with both low alignment and low uniformity. Without Lvv, alignment suffers; without Ldiv, uniformity suffers. Alignment-Uniformity Trade-off [PITH_FULL_IMAGE:figures/full_fig_p032_17.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 14 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2210.07277 (2022)

    Assran, M., Balestriero, R., Duval, Q., Bordes, F., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., Ballas, N.: The hidden uniform cluster prior in self-supervised learning. arXiv preprint arXiv:2210.07277 (2022)

    Pith/arXiv arXiv 2022

  2. [2]

    In: 2024 IEEE International Symposium on Biomedical Imaging (ISBI)

    Ayzenberg, L., Giryes, R., Greenspan, H.: Dinov2 based self supervised learning for few shot medical image segmentation. In: 2024 IEEE International Symposium on Biomedical Imaging (ISBI). pp. 1–5. IEEE (2024)

    2024

  3. [3]

    arXiv preprint arXiv:2302.10260 (2023)

    Balestriero, R.: Unsupervised learning on a diet: Datum index as target free of self-supervision, reconstruction, projector head. arXiv preprint arXiv:2302.10260 (2023)

    Pith/arXiv arXiv 2023

  4. [4]

    arXiv preprint arXiv:2511.08544 (2025)

    Balestriero, R., LeCun, Y .: Lejepa: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544 (2025)

    Pith/arXiv arXiv 2025

  5. [5]

    arXiv preprint arXiv:2105.04906 (2021)

    Bardes, A., Ponce, J., LeCun, Y .: Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 (2021)

    Pith/arXiv arXiv 2021

  6. [6]

    Journal of Rock Mechanics and Geotechnical Engineering (2025)

    Brondolo, F., Beaussant, S.: Dinov2 rocks geological image analysis: Classification, seg- mentation, and interpretability. Journal of Rock Mechanics and Geotechnical Engineering (2025)

    2025

  7. [7]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Buglakova, E., Archit, A., D’Imprima, E., Mahamid, J., Pape, C., Kreshuk, A.: Tiling artifacts and trade-offs of feature normalization in the segmentation of large biological images. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13109– 13118 (2025)

    2025

  8. [8]

    Advances in neural information processing systems33, 9912–9924 (2020)

    Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems33, 9912–9924 (2020)

    2020

  9. [9]

    In: Proceedings of the IEEE/CVF interna- tional conference on computer vision

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF interna- tional conference on computer vision. pp. 9650–9660 (2021)

    2021

  10. [10]

    Advances in Neural Information Processing Systems35, 33860–33875 (2022)

    Chen, C., Zhang, J., Xu, Y ., Chen, L., Duan, J., Chen, Y ., Tran, S., Zeng, B., Chilimbi, T.: Why do we need large batchsizes in contrastive learning? a gradient-bias perspective. Advances in Neural Information Processing Systems35, 33860–33875 (2022)

    2022

  11. [11]

    In: International conference on machine learning

    Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PmLR (2020)

    2020

  12. [12]

    arXiv preprint arXiv:2003.04297 (2020)

    Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 (2020)

    Pith/arXiv arXiv 2003

  13. [13]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Chen, X., Xie, S., He, K.: An empirical study of training self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9640–9649 (2021)

    2021

  14. [14]

    arXiv preprint arXiv:2602.06159 (2026) 15

    Chen, X., Zhang, C., Fu, C., Yang, Z., Zhou, K., Zhang, Y ., He, J., Zhang, Y ., Sun, M., Wang, Z., et al.: Driving with dino: Vision foundation features as a unified bridge for sim-to-real generation in autonomous driving. arXiv preprint arXiv:2602.06159 (2026) 15

    arXiv 2026

  15. [15]

    In: 2009 IEEE conference on computer vision and pattern recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)

    2009

  16. [16]

    arXiv preprint arXiv:2010.11929 (2020)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., De- hghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    Pith/arXiv arXiv 2010

  17. [17]

    In: International conference on machine learning

    Ermolov, A., Siarohin, A., Sangineto, E., Sebe, N.: Whitening for self-supervised represen- tation learning. In: International conference on machine learning. pp. 3015–3024. PMLR (2021)

    2021

  18. [18]

    In: International conference on machine learning

    Garrido, Q., Balestriero, R., Najman, L., Lecun, Y .: Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. In: International conference on machine learning. pp. 10929–10974. PMLR (2023)

    2023

  19. [19]

    Advances in neural information processing systems33, 21271–21284 (2020)

    Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al.: Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems33, 21271–21284 (2020)

    2020

  20. [20]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    He, K., Chen, X., Xie, S., Li, Y ., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16000–16009 (2022)

    2022

  21. [21]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    He, K., Fan, H., Wu, Y ., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9729–9738 (2020)

    2020

  22. [22]

    In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)

    2016

  23. [23]

    Advances in neural information processing systems30(2017)

    Ioffe, S.: Batch renormalization: Towards reducing minibatch dependence in batch-normalized models. Advances in neural information processing systems30(2017)

    2017

  24. [24]

    In: Inter- national conference on machine learning

    Jiang, Z., Chen, T., Mortazavi, B.J., Wang, Z.: Self-damaging contrastive learning. In: Inter- national conference on machine learning. pp. 4927–4939. PMLR (2021)

    2021

  25. [25]

    arXiv preprint arXiv:1609.04836 (2016)

    Keskar, N.S., Mudigere, D., Nocedal, J., Smelyanskiy, M., Tang, P.T.P.: On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 (2016)

    Pith/arXiv arXiv 2016

  26. [26]

    Advances in neural information processing systems33, 18661–18673 (2020)

    Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y ., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. Advances in neural information processing systems33, 18661–18673 (2020)

    2020

  27. [27]

    PloS one20(8), e0329273 (2025)

    Kuang, Y ., Guan, J., Liu, H., Chen, F., Wang, Z., Wang, W.: Piccl: A lightweight multiview contrastive learning framework for image classification. PloS one20(8), e0329273 (2025)

    2025

  28. [28]

    PLOS Computational Biology21(12), e1013828 (2025)

    Moutakanni, T., Couprie, C., Yi, S., Doron, M., Chen, Z.S., Moshkov, N., Gardes, E., Caron, M., Touvron, H., Joulin, A., et al.: Cell-dino: Self-supervised image-based embeddings for cell fluorescent microscopy. PLOS Computational Biology21(12), e1013828 (2025)

    2025

  29. [29]

    In: International Conference on Multimedia Modeling

    Neuschmied, H., Winter, M., Bailer, W.: Improving few-shot object detection using visual explanations of dinov2 features. In: International Conference on Multimedia Modeling. pp. 59–71. Springer (2026)

    2026

  30. [30]

    arXiv preprint arXiv:1807.03748 (2018)

    Oord, A.v.d., Li, Y ., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)

    Pith/arXiv arXiv 2018

  31. [31]

    arXiv preprint arXiv:2304.07193 (2023)

    Oquab, M., Darcet, T., Moutakanni, T., V o, H., Szafraniec, M., Khalidov, V ., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

    Pith/arXiv arXiv 2023

  32. [32]

    Proceedings of the National Academy of Sciences117(40), 24652– 24663 (2020) 16

    Papyan, V ., Han, X., Donoho, D.L.: Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences117(40), 24652– 24663 (2020) 16

    2020

  33. [33]

    In: DAGM German Conference on Pattern Recognition

    Pototzky, D., Sultan, A., Schmidt-Thieme, L.: Fastsiam: Resource-efficient self-supervised learning on a single gpu. In: DAGM German Conference on Pattern Recognition. pp. 53–67. Springer (2022)

    2022

  34. [34]

    arXiv preprint arXiv:2010.10241 (2020)

    Richemond, P.H., Grill, J.B., Altché, F., Tallec, C., Strub, F., Brock, A., Smith, S., De, S., Pascanu, R., Piot, B., et al.: Byol works even without batch statistics. arXiv preprint arXiv:2010.10241 (2020)

    Pith/arXiv arXiv 2010

  35. [35]

    In: 2007 15th European signal processing conference

    Roy, O., Vetterli, M.: The effective rank: A measure of effective dimensionality. In: 2007 15th European signal processing conference. pp. 606–610. IEEE (2007)

    2007

  36. [36]

    arXiv preprint arXiv:2508.10104 (2025)

    Siméoni, O., V o, H.V ., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V ., Szafraniec, M., Yi, S., Ramamonjisoa, M., et al.: Dinov3. arXiv preprint arXiv:2508.10104 (2025)

    Pith/arXiv arXiv 2025

  37. [37]

    arXiv preprint arXiv:2312.04000 (2023)

    Thilak, V ., Huang, C., Saremi, O., Dinh, L., Goh, H., Nakkiran, P., Susskind, J.M., Littwin, E.: Lidar: Sensing linear probing performance in joint embedding ssl architectures. arXiv preprint arXiv:2312.04000 (2023)

    Pith/arXiv arXiv 2023

  38. [38]

    arXiv preprint arXiv:2505.12477 (2025)

    Van Assel, H., Ibrahim, M., Biancalani, T., Regev, A., Balestriero, R.: Joint embedding vs reconstruction: Provable benefits of latent space prediction for self supervised learning. arXiv preprint arXiv:2505.12477 (2025)

    arXiv 2025

  39. [39]

    In: International conference on machine learning

    Wang, T., Isola, P.: Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In: International conference on machine learning. pp. 9929–9939. PMLR (2020)

    2020

  40. [40]

    arXiv preprint arXiv:2203.13457 (2022)

    Wang, Y ., Zhang, Q., Wang, Y ., Yang, J., Lin, Z.: Chaos is a ladder: A new theoretical under- standing of contrastive learning via augmentation overlap. arXiv preprint arXiv:2203.13457 (2022)

    Pith/arXiv arXiv 2022

  41. [41]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Wu, Z., Xiong, Y ., Yu, S.X., Lin, D.: Unsupervised feature learning via non-parametric instance discrimination. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3733–3742 (2018)

    2018

  42. [42]

    Scientific Data 10(1), 41 (2023)

    Yang, J., Shi, R., Wei, D., Liu, Z., Zhao, L., Ke, B., Pfister, H., Ni, B.: Medmnist v2-a large- scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data 10(1), 41 (2023)

    2023

  43. [43]

    In: European conference on computer vision

    Yeh, C.H., Hong, C.Y ., Hsu, Y .C., Liu, T.L., Chen, Y ., LeCun, Y .: Decoupled contrastive learning. In: European conference on computer vision. pp. 668–684. Springer (2022)

    2022

  44. [44]

    In: International Conference on Machine Learning

    Yuan, Z., Wu, Y ., Qiu, Z.H., Du, X., Zhang, L., Zhou, D., Yang, T.: Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. In: International Conference on Machine Learning. pp. 25760–25782. PMLR (2022)

    2022

  45. [45]

    In: International conference on machine learning

    Zbontar, J., Jing, L., Misra, I., LeCun, Y ., Deny, S.: Barlow twins: Self-supervised learning via redundancy reduction. In: International conference on machine learning. pp. 12310–12320. PMLR (2021)

    2021

  46. [46]

    arXiv preprint arXiv:2311.09974 (2023) 17 A Training Details We provide complete implementation details to ensure reproducibility

    Zhang, J., Shen, L., Liu, P.: From pretext to purpose: Batch-adaptive self-supervised learning. arXiv preprint arXiv:2311.09974 (2023) 17 A Training Details We provide complete implementation details to ensure reproducibility. All experiments were conducted using PyTorch. A.1 Architecture Specifications 2D Experiments.We use a Vision Transformer Small (Vi...

    Pith/arXiv arXiv 2023