Pith. sign in

REVIEW 3 major objections 5 minor 33 references

Three Necessary Principles for Self-Supervised Visual Representation Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Self-supervised visual learning needs three non-overlapping objectives: observe, predict, regularize.

desk verdict A clean, honest three-principle formalization of SSL with correct theorems, but the title's 'necessary' claim rests on a small, batch-size-sensitive gap at one scale. read the letter →

arxiv 2608.08309 v1 pith:TA344VAW submitted 2026-08-08 cs.CV cs.AIcs.LG

Nikos Giakoumoglou , Paschalis Giakoumoglou , Tania Stathaki This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords self-supervisedlearningrepresentationenergy-basedmodelsrepresentationalcollapsejoint-embeddingpredictivearchitecturescontrastiveregularizationvisualtransformers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that learning useful visual representations without labels requires a training signal made of three non-overlapping pieces: observation (semantic invariance across augmented views), prediction (latent prediction of masked patches), and regularization (explicit prevention of representational collapse). It formalizes these as three conditions on an energy-based compatibility function and shows that any two without the third leave a degenerate or impoverished solution available. The load-bearing formal result is that under negative-free alignment, an encoder with observation and prediction but no explicit regularizer can collapse to a constant and achieve zero training loss, so regularization is provably necessary rather than a heuristic. At the small scale studied, removing any one principle produces a clear, ordered drop in accuracy, and every major self-supervised method is recovered as a special case of the same energy decomposition.

What carries the argument

The central object is the energy decomposition $F_w(x,y)=\alpha F_w^{\mathrm{proj}}(x,y)+\beta F_w^{\mathrm{pred}}(x,y)+\gamma\Omega(f_\theta)$: a projector term realizes the observation principle (Definition 4), a predictor term realizes the prediction principle (Definition 5), and a geometric regularizer $\Omega$ on the batch distribution of encoder outputs realizes the regularization principle (Definition 6). The projector makes same-image view pairs close after global pooling; the predictor regresses latent representations of masked patches from context; the regularizer is strictly positive at every Dirac mass, so it is minimized only by full-rank representation distributions. The formal work is done by Theorem 8 (constant-encoder collapse under $\gamma=0$ with MSE alignment), Remark 9 (gradient decay of contrastive repulsion), Theorem 15 (uniform invariance gradient plus spatially local prediction gradient), and Theorem 17 (momentum encoder tracks the online fixed point).

What would settle it

Train the same three-objective decomposition at a scale large enough to separate the hypotheses, for instance ViT-Base on ImageNet-1k with an 800-epoch schedule, and compare three configurations: observation plus regularization without prediction, prediction plus regularization without observation, and all three. If the prediction-free model matches the full model on patch-retrieval spatial recall@5, Proposition 10's structural necessity fails; if the observation-free model matches on linear-probe accuracy, Proposition 11 fails. A second check: if NT-Xent without explicit regularization at batch size 4096 matches or exceeds the regularized model's accuracy, then Remark 9's self-limiting claim is empirically bounded at scale.

Watch

Extended reading notes

Core claim

The paper's central claim is that representational completeness in self-supervised visual learning has three necessary and mutually non-redundant components, formalized as Definitions 4, 5, and 6: observation, prediction, and regularization. The argument is carried by a unified energy $F_w(x,y)=\alpha F_w^{\mathrm{proj}}(x,y)+\beta F_w^{\mathrm{pred}}(x,y)+\gamma\Omega(f_\theta)$, whose coefficients are zeroed to recover invariance methods, predictive methods, and combined methods. Theorem 8 proves that with $\gamma=0$ and mean-squared alignment the constant encoder is a global minimizer achieving zero loss, so observation plus prediction cannot stand alone in the negative-free regime; Remark 9 shows contrastive repulsion is self-limiting because its gradient decays as representations homogenize. Theorem 15 proves the invariance and prediction gradients are structurally disjoint at the encoder output under a mean-pool projector, and Theorem 17 proves a momentum encoder converges to the same fixed point as the online encoder, adding no collapse guarantee at convergence. The paper reads all major SSL methods as special cases of this decomposition and validates the necessity claims with controlled ViT-Tiny/STL-10 experiments, including patch-level retrieval for the spatial consequence of prediction.

Load-bearing premise

The load-bearing premise is that the empirical ordering measured at one small scale (ViT-Tiny on STL-10, 200 epochs) reflects the structure of the learning problem rather than the particular architecture, batch size, and hyperparameters; the paper itself leaves open whether a larger encoder could learn spatial structure from observation alone or cross-view invariance from prediction alone.

Editorial extensions

If this is right

  • Under negative-free alignment, an explicit regularizer is formally necessary: without it the constant encoder is a global minimizer, so any negative-free SSL method that omits a geometric term carries a guaranteed degenerate solution.
  • Contrastive losses give only self-limiting collapse resistance; their anti-collapse gradient weakens as representations homogenize and as batch size shrinks, so they cannot replace an explicit regularizer with a guaranteed positive gradient at every step.
  • Observation and prediction supply complementary training signals at the encoder output: one uniform across all patch tokens, one local to context patches, so combining them is not redundant at the studied scale.
  • Momentum encoders and stop-gradient mechanisms stabilize training but do not determine the converged representation; the fixed point is set by the explicit regularizer.
  • Every major self-supervised method can be classified by which of the three principles it instantiates, turning architectural heuristics into a single energy-based design space.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the decomposition scales, SSL architecture design could become modular: swap regularizers and predictors independently, and diagnose failures by which principle is missing, rather than tuning implicit mechanisms.
  • A direct testable extension is the same ablation at larger scale, for example ViT-Base or ViT-Large on ImageNet-1k with longer schedules; the paper itself leaves open whether observation alone can acquire spatial structure, or prediction alone cross-view invariance, at scale. If either gap closes, 'necessary' would weaken to 'useful at this scale.'
  • The batch-size sensitivity result suggests a quantitative prediction not tested at scale: with very large batches the contrastive signal's implicit resistance should approach an explicit regularizer, shrinking the accuracy gap between the contrastive-only and fully regularized configurations.
  • The authors' own caveat that effective rank is a diagnostic rather than a quality score implies that representations satisfying all three principles should still be evaluated by downstream transfer; a dense probe could turn the structural necessity of prediction into a measured result.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that a complete self-supervised visual representation learning signal requires three non-overlapping objectives: observation (cross-view invariance), prediction (latent spatial prediction), and regularization (anti-collapse). It formalizes these as an energy decomposition in Equation (3), proves Theorem 8 (the constant encoder is a global minimizer under negative-free alignment with no regularization), Theorem 15 (gradient complementarity of invariance and prediction at the encoder output under a mean-pool projector), and Theorem 17 (the momentum encoder converges to the online encoder's fixed point and provides no asymptotic collapse guarantee), and validates the claims with ViT-Tiny/STL-10 experiments, including batch-size sensitivity and patch-level retrieval. The paper also presents a taxonomy of prior SSL methods as special cases of the proposed decomposition.

Significance. If the central claims hold, the paper provides a useful conceptual decomposition of SSL objectives and a clean formal statement of when collapse is unavoidable: under negative-free alignment, observation plus prediction without explicit regularization admits the constant encoder as a global minimizer. The theorems are parameter-free consequences of the stated definitions, the proofs are clear, and the paper pairs each theoretical claim with a controlled experiment, including a patch-retrieval evaluation that directly targets the spatial content of prediction. The authors also state explicit caveats about scale, which is commendable. At the same time, the title-level 'necessary' claim is supported mainly by a single small-scale empirical ordering, and some of the reported effective-rank numbers are inconsistent with the paper's own collapse narrative, so the current evidence does not fully justify the title as written.

major comments (3)
  1. [Section 6, Table 1, and Appendix G/Table 6] The empirical claim that 'no pair substitutes for the third' is carried in the contrastive regime by the 3.4-point linear-probe gap between row H (51.6) and row J (55.0) in Table 1. The paper reports means over five seeds but no variance, confidence interval, or significance test, and Appendix G/Table 6 shows that the H-versus-J gap shrinks monotonically with batch size: 8.6 points at B=128, 5.6 at B=256, and 3.4 at B=512. Since Theorem 8 does not apply to NT-Xent and Remark 9 explicitly identifies the contrastive anti-collapse signal as batch-size dependent, this single hyperparameter slice is load-bearing for the title-level 'necessary' wording. Please add uncertainty and significance measures, test larger batch sizes, and either strengthen the evidence or narrow the claims to the studied configuration.
  2. [Table 1, rows G and I, and Appendix E] The effective-rank values in Table 1 contradict the accompanying narrative. Appendix E defines effective rank near 1 as collapse onto a single direction and values approaching D' as full rank; yet row G (MSE+JEPA, no regularizer) reports 164.5, which is high rank, while row I (MSE+JEPA+SIGReg) reports 9.4, which is near collapse. The table caption and Section 6 describe row G as collapsing under Theorem 8 and row I as the regularized run that avoids collapse. Similarly, row H without regularization (167.7) has higher effective rank than row J with SIGReg (87.5). Please reconcile the table with Figure 1 and the Appendix E definition, or report the underlying covariance spectra if the scalar effective-rank measure is being used in a way that the text does not define.
  3. [Section 3.1, Propositions 10 and 11] The necessity arguments for prediction and observation are structural-by-construction rather than optimization-theoretic: no term in the objective depends on patch positions or cross-view alignment, so the omitted signal is never provided. The paper itself concedes in Section 3.1 that 'whether a larger encoder could acquire spatial structure from observation alone, or cross-view invariance from prediction alone, remains open.' Combined with the fact that Theorem 8 establishes necessity only for the negative-free case, the title and abstract overclaim relative to what is proven. Please qualify the title and the normative conclusions, or provide a formal statement of the sense in which 'necessary' is meant within the stated protocol.
minor comments (5)
  1. [Theorem 17, note] The reference to 'Equation (8)' is dangling: no Equation (8) is defined in the main text, where equation numbers jump from Equation (3) to Equation (13). Please correct the cross-reference.
  2. [Title and abstract] The abstract's qualified phrase 'at the scale we study, no pair substitutes for the third' is in tension with the unqualified title 'Three Necessary Principles for Self-Supervised Visual Representation Learning.' The title should carry the same qualification as the abstract.
  3. [Table 5 caption] Using the same dagger symbol for 'redundant implicit mechanism in the method name' and 'implicit collapse prevention only in the Reg column' is confusing; please use distinct symbols or a clearer legend.
  4. [Section 6, row I discussion] The sentence attributing row I's low effective rank to a gradient-magnitude imbalance between MSE and SIGReg is plausible, but it is difficult to reconcile with the claim that the regularizer prevents dimensional collapse. Please state explicitly whether the low effective rank of row I represents a failure of this particular regularizer/objective combination or a failure of the regularization principle.
  5. [Figure 3 caption] The caption says the figure 'verifies Theorem 15,' but the figure shows per-patch gradient norms at selected checkpoints. A sentence explaining how uniformity and locality are measured from these norms would make the verification transparent.

Circularity Check

1 steps flagged · score 2.0 of 10

Only minor definitional circularity in the structural-necessity claims; the formal theorems and empirical ordering are self-contained.

  1. self definitional [Abstract and Section 3.1 (Propositions 10-11)]
    "Dropping prediction withholds the spatial training signal by construction; dropping observation forfeits cross-view semantic invariance by construction; at the scale we study, no pair substitutes for the third. ... Propositions 10 and 11 are structural, not optimization-theoretic: they assert the omitted signal is never provided, not that no encoder could acquire the property indirectly."

    The necessity of prediction and observation is established by definition: the prediction principle (Definition 5) is defined as the presence of an L_pred term, and observation (Definition 4) as the presence of a cross-view F_proj term. Proposition 10 then observes that without L_pred no term depends on patch positions, and Proposition 11 that without F_proj no term enforces cross-view closeness. These conclusions follow immediately from the definitions and from Equation (3), so the abstract's 'no pair substitutes' is partly a restatement of how the principles were defined rather than an empirical or derived result.

full rationale

The paper's load-bearing formal results are parameter-free consequences of the stated objectives: Theorem 8 proves the constant encoder is a global minimizer of alpha*LMSE + beta*Lpred at gamma=0 by direct construction, and Theorem 15 is a chain-rule calculation under a mean-pool projector. The empirical ordering in Table 1 is measured from a controlled ablation, not obtained by fitting parameters to those accuracies; no fitted constant is renamed as a prediction. The 'structural necessity' arguments for prediction and observation are the only definitional element: they assert that removing the term defined as providing that signal removes the signal, which is true by construction. The paper explicitly acknowledges this ('structural, not optimization-theoretic' and 'remains open'), so the circularity is minor and localized. Self-citations (e.g., [17-20]) are not load-bearing for the proofs or the necessity claims. Overall the central claims retain independent content, so the score is low.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The theoretical proofs are self-contained on top of standard calculus and the stated loss definitions. The three domain assumptions above are explicit in the paper: momentum convergence, mean-pool projector, and MSE alignment. The central 'necessary at the scale we study' claim additionally relies on transfer from one small benchmark, which the authors list as an open question. There are no invented physical entities; the three principles are conceptual labels, not new objects.

free parameters (6)
  • alpha (invariance loss weight) = 0.1 for NT-Xent, 1.0 for MSE
    Hand-chosen hyperparameter. The relative empirical importance of observation versus prediction and the 'no pair substitutes' ordering could shift with this weight.
  • beta (prediction loss weight) = 1.0
    Hand-chosen hyperparameter. The prediction principle's marginal gains are measured at this fixed weight.
  • gamma (regularization loss weight) = 0.05 for SIGReg, 0.02 for VCReg, 1.0 for RDMReg
    Hand-chosen hyperparameters. Row I shows the regularizer's effectiveness depends on this weight relative to the invariance loss; the claim of regularization necessity is demonstrated at this operating point.
  • NT-Xent temperature tau = 0.07
    Hand-chosen hyperparameter. The self-limiting contrastive argument in Remark 9 is temperature-sensitive through the softmax denominator.
  • Momentum coefficient m = 0.996
    Hand-chosen hyperparameter. The momentum-encoder redundancy experiment and the convergence plot depend on this EMA rate.
  • Batch size B = 512
    Hand-chosen hyperparameter. The contrastive anti-collapse signal scales with the number of in-batch negatives, so the empirical claim that 'no pair substitutes for the third' is demonstrated at this batch size, with Table 6 showing batch-size sensitivity.
assumptions (4)
  • domain assumption Theorem 17 assumes the online encoder theta(t) converges to a fixed point theta* under the training objective.
    The momentum-encoder redundancy conclusion is conditional on convergence, which the paper notes is not guaranteed in general. The proof analyzes a decoupled low-pass filter, not the full coupled stop-gradient dynamics.
  • domain assumption Theorem 15 assumes a mean-pool projector architecture, where h_psi acts on the pooled token s_x.
    The uniform-gradient result relies on d s_x / d f_theta(x)(i) = (1/N) I_D. For projectors that act on individual patch tokens, the uniform-gradient statement does not apply.
  • domain assumption Theorem 8 assumes negative-free alignment, Linv = LMSE.
    The collapse theorem is proven for MSE matching, not for NT-Xent. For the contrastive case the paper gives only the informal gradient-decay argument of Remark 9, not a theorem.
  • domain assumption The 'necessity' of observation and prediction is assumed to generalize from the studied protocol to the general statement in the title.
    The paper explicitly leaves open whether larger encoders could acquire spatial structure from observation alone or cross-view invariance from prediction alone. The title's universal claim depends on this unproven transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Three Necessary Principles for Self-Supervised Visual Representation Learning." pith.science (2026). https://pith.science/paper/TA344VAW

@misc{pith2026260808309,
  author       = {Pith},
  title        = {Pith review of: Three Necessary Principles for Self-Supervised Visual Representation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TA344VAW}},
  note         = {Machine review of arXiv:2608.08309}
}
read the original abstract

We argue that learning visual representations without labels requires a training signal jointly complete across three non-overlapping objectives: semantic invariance across augmented views, patch-level spatial prediction, and representational non-degeneracy. We formalize these as the observation, prediction, and regularization principles and prove (i) that combining observation and prediction without regularization admits the constant encoder as a global minimizer under negative-free alignment; (ii) that the two objectives are gradient-complementary and structurally non-conflicting at the encoder output; and (iii) that the momentum encoder converges to the same fixed point as the online encoder and provides no collapse guarantee at convergence. Contrastive alignment provides only self-limiting collapse resistance, formalized via an explicit gradient-decay argument. Dropping prediction withholds the spatial training signal by construction; dropping observation forfeits cross-view semantic invariance by construction; at the scale we study, no pair substitutes for the third. Every major self-supervised method is a special case of a single unified energy decomposition. We pair every theoretical claim with a controlled experiment, including a patch-retrieval evaluation for the spatial consequence of prediction.

Figures

Figures reproduced from arXiv: 2608.08309 by the authors.

Figure 1
Figure 1. Eigenspectrum of the projector-output covariance, verifying The [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Marginal gain in linear￾probe accuracy from adding each principle, verifying Propositions 7, 10, and 11. Each bar is the accuracy gained by adding one principle to the pair that omits it, relative to our proposed model (row J, 55.0%). No principle is redundant at this scale, and observation contributes the largest gain. Collapse without regularization (Theo￾rem 8). The direct signature of Theorem 8 is a degenerate r… view at source ↗
Figure 3
Figure 3. Per-patch gradient norm at the encoder output, verifying Theorem 15. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Momentum-encoder redundancy, verifying Theorem 17. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Architecture and gradient flows for the three-principle decomposition. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Effective rank over training. Negative-free alignment without regularization stays low-rank; the contrastive term raises rank but does not stabilize it; the explicit regularizer holds it in a healthy band. The MSE + SIGReg run (row I) stabilizes at low rank, reflecting…
Figure 7
Figure 7. Figure 7: Linear-probe accuracy over training for every strict subset. [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]
Figure 8
Figure 8. Figure 8: Linear-probe accuracy for four regularizer choices. [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 17 canonical work pages

  1. [1]

    arXiv preprint arXiv:2506.09985 (2025)

    Assran, M., Bardes, A., Fan, D., Garrido, Q., Howes, R., Komeili, M., Muckley, M., Rizvi, A., Roberts, C., Sinha, K., Zholus, A., Arnaud, S., Gejji, A., Martin, A., Robert Hogan, F., Dugas, D., Bojanowski, P., Khalidov, V., Labatut, P., Massa, F., Szafraniec, M., Krishnakumar, K., Li, Y., Ma, X., Chandar, S., Meier, F., LeCun, Y., Rabbat, M., Ballas, N.: ...

    arXiv 2025

  2. [2]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., Ballas, N.: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15619–15629 (2023)

    work page 2023

  3. [3]

    In: International conference on machine learning

    Baevski, A., Hsu, W.N., Xu, Q., Babu, A., Gu, J., Auli, M.: Data2Vec: A Gen- eral Framework for Self-Supervised Learning in Speech, Vision and Language. In: International conference on machine learning. pp. 1298–1312. PMLR (2022)

    work page 2022

  4. [4]

    Balestriero, R., LeCun, Y.: LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics (2025),https://arxiv.org/abs/2511.08544

    arXiv 2025

  5. [5]

    In: International Conference on Learning Representations (2022),https: //openreview.net/forum?id=p-BhZSz59o4

    Bao, H., Dong, L., Piao, S., Wei, F.: BEit: BERT pre-training of image trans- formers. In: International Conference on Learning Representations (2022),https: //openreview.net/forum?id=p-BhZSz59o4

    2022

  6. [6]

    arXiv preprint arXiv:2404.08471 (2024)

    Bardes, A., Garrido, Q., Ponce, J., Rabbat, M., LeCun, Y., Assran, M., Ballas, N.: Revisiting Feature Prediction for Learning Visual Representations from Video. arXiv preprint arXiv:2404.08471 (2024)

    arXiv 2024

  7. [7]

    In: International Conference on Learning Representations (2022),https://openreview.net/forum?id=xm6YD62D1Ub

    Bardes, A., Ponce, J., LeCun, Y.: VICReg: Variance-Invariance-Covariance Regu- larization for Self-Supervised Learning. In: International Conference on Learning Representations (2022),https://openreview.net/forum?id=xm6YD62D1Ub

    work page 2022

  8. [8]

    In: Proceedings of the International Conference on Computer Vision (ICCV) (2021)

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the International Conference on Computer Vision (ICCV) (2021)

    2021

Show all 33 references
  1. [9]

    In: International conference on machine learning

    Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A Simple Framework for Con- trastive Learning of Visual Representations. In: International conference on machine learning. pp. 1597–1607. PmLR (2020)

    2020

  2. [10]

    In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition

    Chen, X., He, K.: Exploring Simple Siamese Representation Learning. In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15750–15758 (2021)

    2021

  3. [11]

    In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)

    Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05). vol. 1, pp. 539–546 vol. 1 (2005).https://doi.org/10.1109/C...

    2005 doi

  4. [12]

    In: Proceedings of the fourteenth international conference on artificial intelligence and statistics

    Coates, A., Ng, A., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics. pp. 215–223. JMLR Workshop and Conference Proceedings (2011) 16 N. Giakoumo...

    2011

  5. [13]

    CoRRabs/2306.02572(2023), https://doi.org/10.48550/arXiv.2306.02572

    Dawid, A., LeCun, Y.: Introduction to Latent Variable Energy-Based Models: A Path Towards Autonomous Machine Intelligence. CoRRabs/2306.02572(2023), https://doi.org/10.48550/arXiv.2306.02572

    2023 doi arXiv

  6. [14]

    Ermolov, A., Siarohin, A., Sangineto, E., Sebe, N.: Whitening for self-supervised representation learning (2021),https://openreview.net/forum?id=3Wp8HM2CNdR

    2021

  7. [15]

    In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J

    Garrido, Q., Balestriero, R., Najman, L., Lecun, Y.: RankMe: Assessing the down- stream performance of pretrained self-supervised representations by their rank. In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (eds.) Proceedings of the 40th Inte...

    2023

  8. [16]

    Garrido, Q., Chen, Y., Bardes, A., Najman, L., Lecun, Y.: On the duality between contrastive and non-contrastive self-supervised learning (2023).https://doi.org/ https://doi.org/10.48550/arXiv.2206.02574 , https://arxiv.org/abs/2206. 02574

    2023 doi arXiv

  9. [17]

    Giakoumoglou, N., Stathaki, T.: Relational Representation Distillation (2024), https://arxiv.org/abs/2407.12073

    2024 arXiv

  10. [18]

    Giakoumoglou, N., Stathaki, T.: SynCo: Synthetic Hard Negatives for Contrastive Visual Representation Learning (2024),https://arxiv.org/abs/2410.02401

    2024 arXiv

  11. [19]

    In: 2025 IEEE International Conference on Image Processing (ICIP)

    Giakoumoglou, N., Stathaki, T.: Cluster Contrast for Unsupervised Visual Repre- sentation Learning. In: 2025 IEEE International Conference on Image Processing (ICIP). pp. 133–138. IEEE (2025)

    2025

  12. [20]

    Giakoumoglou, N., Stathaki, T., Gkelias, A.: A Review on Discriminative Self- supervised Learning Methods in Computer Vision (2025),https://arxiv.org/ abs/2405.04969

    2025 arXiv

  13. [21]

    Advances in neural information processing systems33, 21271–21284 (2020)

    Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al.: Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning. Advances in neural information processing systems33, 21271...

    2020

  14. [22]

    arXiv:2111.06377 (2021)

    He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. arXiv:2111.06377 (2021)

    2021 arXiv

  15. [23]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum Contrast for Unsupervised Visual Representation Learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9729–9738 (2020)

    2020

  16. [24]

    In: International Conference on Learning Representations (2022),https://openreview.net/forum?id=YevsQ05DEN7

    Jing, L., Vincent, P., LeCun, Y., Tian, Y.: Understanding dimensional collapse in contrastive self-supervised learning. In: International Conference on Learning Representations (2022),https://openreview.net/forum?id=YevsQ05DEN7

    2022

  17. [25]

    Kuang, Y., Dagade, Y., Rudner, T.G.J., Balestriero, R., LeCun, Y.: Rectified LPJEPA: Joint-Embedding Predictive Architectures with Sparse and Maximum- Entropy Representations (2026),https://arxiv.org/abs/2602.01456

    2026 arXiv

  18. [26]

    2, 2022-06-27

    LeCun, Y., et al.: A Path Towards Autonomous Machine Intelligence Version 0.9. 2, 2022-06-27. Open Review62(1), 1–62 (2022)

    2022

  19. [27]

    Advances in neural information processing sys- tems37, 2348–2377 (2024)

    Mo, S., Tong, S.: Connecting Joint-Embedding Predictive Architecture with Con- trastive Self-Supervised Learning. Advances in neural information processing sys- tems37, 2348–2377 (2024)

    2024

  20. [28]

    van den Oord, A., Li, Y., Vinyals, O.: Representation Learning with Contrastive Predictive Coding (2019),https://arxiv.org/abs/1807.03748

    2019 arXiv

  21. [29]

    Transactions on Ma- chine Learning Research (2024),https://openreview.net/forum?id=a68SUt6zFt, featured Certification

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y., Li, S.W., Misra, I., Rabbat, M., Sharma, V., Three Necessary Principles for SSL 17 Syn...

    2024

  22. [30]

    In: Meila, M., Zhang, T

    Tian, Y., Chen, X., Ganguli, S.: Understanding self-supervised learning dynam- ics without contrastive pairs. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning. Proceedings of Ma- chine Learning Research, vol. 139, pp. 10268–1...

    2021

  23. [31]

    In: III, H.D., Singh, A

    Wang, T., Isola, P.: Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In: III, H.D., Singh, A. (eds.) Pro- ceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 11...

    2020

  24. [32]

    Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: Self-supervised learning via redundancy reduction (2021),https://arxiv.org/abs/2103.03230

    2021 arXiv

  25. [33]

    Giakoumoglou et al

    Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., Kong, T.: ibot: Image bert pre-training with online tokenizer (2022),https://arxiv.org/abs/2111.07832 18 N. Giakoumoglou et al. A Energy-Based Training Strategies We recall the two standard strategies for training an ...

    2022 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.