Pith. sign in

REVIEW 5 major objections 7 minor 50 references

Deterministic JEPA fails on text because masked language rarely has one coherent latent target, so squared-error prediction collapses into a centroid over alternatives.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 19:54 UTC pith:ECOZCWS5

load-bearing objection Clean mechanism paper: pure text JEPA collapses in the order their theory predicts; the main caveat is whether their T-JEPA recipe is fair or collapse-tuned. the 5 major comments →

arxiv 2607.23531 v1 pith:ECOZCWS5 submitted 2026-07-26 cs.CL

The JEPA Paradox in Language: The Geometry of Linguistic Alternatives

Anh Trac Duc Dinh , Khang Nhat Hoang Vo This is my paper
classification cs.CL
keywords JEPAconditional concentrationself-supervised learningtext representation learningrepresentation collapsecentroid degeneracymasked predictionlatent prediction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Joint-embedding predictive architectures learn by matching latent representations of missing content from context. That recipe works for images, video, and audio, yet has not become a standard way to train text encoders. This paper argues the gap is geometric, not a matter of capacity or tuning: useful deterministic latent prediction needs conditional concentration—given a context and target location, the target embedding should sit near a single meaningful point. Local image patches often meet that bar through spatial continuity; masked text often does not, because many valid token or span completions can point in different directions. Under squared error the optimal predictor is then a centroid over incompatible alternatives, which creates pressure toward either irreducible noise or loss-reducing collapse. Matched image and text experiments show the predicted order: mutual-information saturation and elevated target variance come first, then train–validation instability, effective-rank degeneration, cosine collapse, and near-chance transfer. The point is not that predictive learning is impossible for language, but that text-compatible objectives must keep multiple plausible completions alive rather than compress them into one latent point.

Core claim

Deterministic squared-error JEPA is mismatched to token-level language because masked text often lacks conditional concentration. When several valid completions remain separated in representation space, the Bayes-optimal predictor is a centroid over those alternatives; limited predictability and elevated irreducible variance then precede effective-rank degeneration, cosine collapse, and poor downstream transfer. Matched image JEPA training does not follow this failure chain, and the same text pattern repeats across independent data seeds.

What carries the argument

Conditional concentration—the requirement that, given context and target location, the target representation lie near a single meaningful point. The argument ties three necessary conditions together: predictability, non-collapse, and low conditional variance. Their joint failure under squared-error latent prediction produces centroid degeneracy: the model either absorbs irreducible multimodal variance or reduces loss by merging the distinctions it should preserve.

Load-bearing premise

That the authors' text JEPA recipe—span masking, model sizes, data scale, and a long warmup chosen to make collapse dynamics observable—is a fair test of the objective itself, not a setup that manufactures the predicted failure.

What would settle it

A pure squared-error latent-prediction text JEPA, with no generative anchor, that still shows rising context–target mutual information, non-degenerate effective rank, irreducible target variance falling because completions truly concentrate rather than because the encoder compresses, and strong frozen transfer on classification and retrieval—matching the healthy image-JEPA diagnostic pattern.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Text-compatible JEPA objectives must preserve multiple plausible completions—via distributional, mixture, contrastive, or semantic-level targets—rather than regress to one latent point.
  • Marginal collapse-prevention alone is not enough on text: keeping covariance healthy does not guarantee predictability or avoid centroid degeneracy.
  • JEPA-style gains reported when a generative language loss is kept alongside latent prediction are consistent with the thesis, because the generative term resolves token ambiguity first.
  • Diagnostics should track the ordered chain—MI saturation and high conditional variance before rank and cosine collapse—not only final downstream scores.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Moving targets from tokens to higher-level spans or meanings may restore conditional concentration even under squared error, if those targets truly share a center.
  • The same geometry likely limits other deterministic latent regressors on open-ended discrete sequences, not only JEPA-style predictors.
  • A direct next experiment the paper motivates but does not run is a mixture- or set-valued latent predictor that can represent several live completions without averaging them.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The manuscript argues that deterministic squared-error latent prediction (JEPA) is structurally mismatched to token-level language because masked text lacks conditional concentration: multiple plausible completions have separated target representations, so the Bayes-optimal predictor is a centroid over heterogeneous alternatives. Three conditions — predictability, non-collapse, low conditional variance — are formalized via standard tools (DPI, Lipschitz concentration, bias–variance decomposition, separation lower bounds; App. B). Empirically, a constructed text JEPA (T-JEPA, BERT-Large on 100K C4 sentences) is compared to I-JEPA (ViT-H on 100K ImageNet) under matched protocols, tracking an InfoNCE MI proxy, effective rank, cosine similarity, train–val loss, and an estimated conditional target variance. T-JEPA shows MI saturation and elevated variance before loss divergence, rank degeneration, cosine collapse, and chance-level/zero downstream transfer (Table 2); the chain reproduces across five data seeds (App. D). App. J positions the result against LLM-JEPA/DLLM-JEPA (JEPA succeeds only as an auxiliary to a generative anchor), and App. K proves SIGReg cannot fix predictability or centroid degeneracy, with an explicit falsifiable prediction.

Significance. If the central claim holds, the paper gives a clean, mechanistic answer to a genuinely open question — why JEPA-style latent prediction has transferred to images, video, and audio but not become a standard text-encoder objective — and it does so with several practices worth crediting: an explicit falsifiable prediction (Consequence IV, App. K: T-JEPA+SIGReg should pass geometric collapse tests while still failing MI/transfer), five-seed replication of the diagnostic chain (App. D), a reconciliation with the strongest apparent counterexamples (LLM-JEPA's γ=0 ablation, App. J), and open disclosure of the diagnostic-schedule tuning (§5.1, App. F). The theoretical apparatus is elementary (conditional-mean optimality, total variance, DPI), but it is correctly stated and used to organize a coherent five-metric diagnostic protocol rather than to smuggle in the conclusion. The empirical scale is small and largely single-configuration, so the result is best read as a well-supported mechanistic hypothesis rather than a settled law; within that scope it would be of clear interest to the SSL and representation-learning communities.

major comments (5)
  1. [§2 / App. J] §2 and throughout: the manuscript never reconciles its thesis with data2vec (Baevski et al., 2022), which is cited once as an 'important precursor' but is, functionally, deterministic EMA-teacher latent prediction at masked positions trained successfully on text — nearly the same objective family as T-JEPA, with strong downstream results. This is the most direct empirical counterexample to the claim that squared-error latent prediction is mismatched to token-level language, and App. J's treatment of LLM-JEPA/DLLM-JEPA does not cover it (data2vec has no generative anchor). The paper needs a dedicated analysis: is the difference the target (average of top-K layers vs. position-specific token states), mask ratio (data2vec uses much higher masking), block masking, training scale/duration, or normalization of targets? Without this, a reader cannot tell whether T-JEPA diagnoses an objective–ge
  2. [§5.4 and App. C.3] The cross-modal quantitative comparison of irreducible variance ('roughly double I-JEPA's level', supporting Consequence III) rests on two estimators that measure different things. For text, dVar is dispersion of EMA-encoder embeddings of oracle-sampled span completions weighted by oracle probabilities (semantic alternatives). For images, dVar is dispersion under Gaussian pixel noise (σ_aug=0.2) and sign flips — i.e., encoder smoothness to augmentation, not the conditional uncertainty of masked-patch content given context. A perfectly smooth image encoder yields near-zero image dVar regardless of how ambiguous the masked patch actually is. The text estimator conditions on genuine alternative completions; the image estimator does not. The qualitative conclusion (text variance elevated pre-collapse, collapsing only with rank degeneration) may still hold, but the numeric cross-modality gap
  3. [§5.1 / Table 1 / App. F] The temporal failure signature in Table 1 — the paper's central empirical claim — is demonstrated only under the 15-epoch diagnostic schedule that §5.1 states 'was chosen to make collapse dynamics observable.' App. F argues the standardized 10-epoch schedule collapses 'faster,' but supports this only with downstream endpoints (Table 2), not with the five diagnostic trajectories under that schedule. The ordering (MI saturation and high dVar precede rank collapse) is therefore established in exactly one, explicitly engineered optimization regime. Since the ordering — not merely the collapse — is what distinguishes this paper's mechanism from generic optimization instability, the manuscript should show the diagnostic curves (MI, dVar, erank, cosine, loss) under the standardized schedule and ideally one or two intermediate schedules, demonstrating that the ordering is schedule-invariant even
  4. [§5.2–5.3 / App. C.2] The effective-rank evidence has two unresolved issues. (1) Both models sit at erank ≈ 4.7–4.85 (out of d=1024/1280) before T-JEPA's collapse; characterizing 4.66 as 'non-trivial rank' (§5.2) is misleading — both spectra are extremely concentrated, and the I-JEPA/T-JEPA contrast is 4.85 vs. 1.57, not full-rank vs. collapsed. (2) The late-training divergence — training erank inflating to 494–545 while validation erank falls to 1.57 (§5.3) — is interpreted as 'the training objective gamed by overfitting to noise,' but the estimation protocol (batch size / number of representations per covariance estimate, computed 'every 10 training steps' per App. C.2) is not reported, and small-sample covariance estimates can produce exactly this kind of train/val spectral divergence. Please report the estimator details, confidence intervals across the five seeds, and temper the causal gloss until the est
  5. [§4.1 / App. C.1] The MI proxy Î = log N − L_InfoNCE uses a fixed dot-product critic s(z_C, z_T) = z_C^⊤ z_T with no learned projection (App. C.1). A low value then certifies only that target identity is not linearly decodable from the context representation under this particular geometry — not that I(z_C; z_T) is small; the encoder could store predictive information in a form invisible to a bilinear critic. The argument is comparative (T-JEPA plateau vs. I-JEPA rise under the same critic), which mitigates but does not eliminate the concern, since the two encoders have different dimensionalities and masking geometries. At minimum, report the proxy with a stronger (e.g., learned projection / MLP) critic, and soften language such as 'MI saturation' to reflect that this is a lower-bound proxy under a restricted critic class. Relatedly, the 1/N inside the log in Eq. (60) is nonstandard — please reconcile with
minor comments (7)
  1. [Table 2] Table 2: T-JEPA IMDB accuracy 50.02 with F1 66.68 is the signature of single-class (majority) prediction; please report macro-F1 or note this explicitly so the F1 column is not read as evidence of residual signal.
  2. [§5.1 vs. App. F] §5.1 states T-JEPA EMA momentum τ = 0.996 while App. F specifies τ = 0.9996 for the downstream runs. These are different protocols and that is fine, but the discrepancy should be flagged in one place to avoid appearing to be an inconsistency.
  3. [App. D / §5.5] App. D: 'it's objective' and 'each seed has it's own' — grammatical errors (its). Also in the main text, §5.5 refers to 'Sections 5.2–4.3' (should be 5.2–5.4).
  4. [App. G] Terminology alternates between 'centroid degeneracy' (Abstract, Consequence III) and 'centroid degeneration' (App. G); please standardize.
  5. [App. G] App. G: the claim that T-JEPA token embeddings span 'a t-SNE range of ~0.003' is not interpretable without the overall plot scale, since t-SNE axes are in arbitrary units; report the ratio to the full embedding extent or use a scale-invariant statistic.
  6. [App. B.1.2] The conditional perplexity range 8–32 (App. B.1.2) is stated without a citation or measurement; please source it or label it as an assumption.
  7. [§5.5] BYOL_M vs. BYOL_R (§5.5): the Replace setting keeps one view as the uncorrupted original, so the two-view task is substantially easier independent of conditional ambiguity; the interpretation as support for the ambiguity mechanism should acknowledge this confound.

Circularity Check

0 steps flagged

No load-bearing circularity: MSE/conditional-mean theory and empirical failure ordering are independent of their inputs; only mild experimental-design self-containment.

full rationale

The paper’s central chain is: (i) squared-error JEPA is Bayes-optimal at the conditional mean (Eq. 6, Prop. B.12); (ii) under a separation assumption on plausible text completions, that mean is a non-representative centroid and residual variance is bounded below (Props. B.5, B.16–B.17); (iii) collapse is then an available low-variance escape (Prop. B.10); (iv) matched I-JEPA vs T-JEPA runs should therefore show MI saturation and elevated dVar before rank/cosine collapse and poor transfer. Steps (i)–(iii) are standard bias–variance and DPI arguments under explicitly stated assumptions; they do not define the empirical diagnostics to equal the conclusion, nor do they import a uniqueness theorem from the authors. Step (iv) is tested with held-out MI proxy, effective rank, cosine stats, irreducible-variance estimator, multi-seed C4 draws, and frozen-encoder downstream tasks—quantities not fitted to force the ordering. The only mild self-containment note is methodological, not definitional: the long warmup “was chosen to make collapse dynamics observable” (§5.1), but App. F reports the same collapse under the shorter standardized schedule, so the result is not manufactured by that choice. Author-overlapping citations (e.g. TI-JEPA) appear only as related work, not as load-bearing premises. Score 1 rather than 0 only for that schedule caveat; no circular step meets the quote-and-reduce bar.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 3 invented entities

The central claim rests on standard MSE/information facts plus domain assumptions about image locality vs lexical ambiguity, and on the authors' operationalization of T-JEPA and its diagnostics. Free parameters are training and metric knobs that affect when collapse is observed; axioms are the separation/Lipschitz/non-degeneracy conditions in App. B; invented entities are conceptual labels (conditional concentration, centroid degeneracy) and the T-JEPA construct itself, not new physical objects.

free parameters (6)
  • Diagnostic LR schedule / 10-epoch warmup to 1e-3 then cosine to 1e-6 over 15 epochs = warmup 2e-4→1e-3 over 10 epochs; 15 epochs total
    Explicitly chosen so collapse stages are temporally resolvable (§5.1); changes timing of the reported sequence.
  • EMA momentum τ = 0.996 (diagnostics); 0.9996 (downstream T-JEPA)
    Controls target encoder lag; standard JEPA knob affecting stability vs collapse speed.
  • Text span masking budget and span geometry = 1–5 spans, length 1–5; ~15–20% tokens downstream
    Defines how multimodal the conditional targets are; core experimental choice for T-JEPA.
  • Conditional variance estimator K and oracle threshold = K=16; ε_min=0.001
    Number and filtering of MLM completions determine dVar magnitude used to validate Argument III.
  • InfoNCE proxy queue size and temperature = N=2048; τ=0.1
    Sets scale of MI proxy; comparative only but used as Stage-1 evidence.
  • Data scale and backbone pair = 100K diagnostics; 3M downstream C4; d=1280 vs 1024
    100K ImageNet vs 100K C4; ViT-H/16 vs BERT-Large—capacity and domain choices that bound external validity.
axioms (6)
  • standard math Data processing inequality: latent I(z_C; z_T) cannot exceed raw context–target information (Lemma B.1).
    Used to bound predictability independent of encoder capacity (App. B.1).
  • standard math Squared-error optimal predictor is the conditional mean (Eq. 6; Prop. B.12).
    Load-bearing for centroid degeneracy claim throughout §§3–4.
  • domain assumption Natural images have bounded local variation and locally Lipschitz target maps, yielding spatial concentration (Prop. B.3, Assumption 3).
    Justifies why I-JEPA enjoys low conditional variance; not proved from data in-paper.
  • domain assumption Target separation: plausible token completions remain at least Δ apart in target representation space unless the encoder collapses them (Assumption 1 / 4).
    Needed for non-concentration and variance lower bounds (Props. B.5, B.17); if false a priori, text could be conditionally concentrated without collapse.
  • domain assumption Non-degenerate visual targets: masked image targets retain content variation beyond position (Assumption 2).
    Supplies anti-collapse pressure for images (Prop. B.8).
  • ad hoc to paper T-JEPA with EMA + stop-gradient + span masking is the appropriate pure-text counterpart to I-JEPA for testing the paradox.
    Defines the experimental object; alternative text JEPA designs might change outcomes.
invented entities (3)
  • Conditional concentration (as JEPA suitability criterion) independent evidence
    purpose: Name the geometric requirement that target latents cluster around one meaningful point given context and location.
    Organizing concept for the image–text divide; operationalized via MI proxy and dVar rather than measured as a single established statistic in prior work.
  • Centroid degeneracy independent evidence
    purpose: Label the failure mode where the MSE optimum averages incompatible linguistic alternatives.
    Direct consequence of conditional mean under multimodal targets; supported by theory and variance/collapse diagnostics.
  • T-JEPA (this paper's text JEPA instantiation) no independent evidence
    purpose: Provide a pure latent-prediction text baseline matched to I-JEPA-style training.
    Experimental construct unrelated to prior T-JEPA names; no external standard implementation.

pith-pipeline@v1.2.0-grok45-kimik3 · 30630 in / 4320 out tokens · 94219 ms · 2026-07-30T19:54:50.977441+00:00 · methodology

0 comments
read the original abstract

Joint-Embedding Predictive Architectures (JEPAs) are effective for images, video, and audio, yet deterministic JEPA-style latent prediction has not become a standard objective for text encoders. We argue that this gap reflects a mismatch between squared-error latent prediction and the conditional structure of language. The key requirement is conditional concentration: given a context and target location, the target representation should lie near a single meaningful point. Local image prediction often satisfies this through spatial continuity, whereas masked text can admit multiple valid token or span completions whose representations need not share a coherent center. We formalize this mismatch through three conditions---predictability, non-collapse, and low conditional variance---and show how their failure creates centroid degeneracy and collapse pressure in text. Matched I-JEPA and T-JEPA experiments reveal the predicted sequence: mutual-information saturation and elevated target variance precede train--validation instability, effective-rank degeneration, cosine collapse, and poor downstream transfer. The same pattern appears across five independent data seeds, indicating that it is not a sampling artifact. These results do not rule out predictive learning for language; they show that text-compatible JEPA objectives must preserve multiple plausible completions rather than compress them into a single latent point.

Figures

Figures reproduced from arXiv: 2607.23531 by Anh Trac Duc Dinh, Khang Nhat Hoang Vo.

Figure 1
Figure 1. Figure 1: Conditional concentration explains the image-text divide in JEPA. In images, spatial con [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Context–target predictability and representation rank for I-JEPA and T-JEPA. Left: In [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Loss and Effective Rank for I-JEPA and T-JEPA (training and validation, log scale). [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Irreducible variance Var( d z ∗ | zC , pj ) for I-JEPA and T-JEPA (log and raw scale). T￾JEPA’s irreducible variance remains persistently higher than I-JEPA’s throughout training and col￾lapses to near-zero only after representational collapse at step ∼15,000. The red dashed line marks T-JEPA’s instability point (step 9,340) [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: T-JEPA validation MI proxy (left) and effective rank (right, log scale) across 5 seeds ( [PITH_FULL_IMAGE:figures/full_fig_p024_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: T-JEPA validation loss (left) and effective rank (right, both log scale) across 5 seeds ( [PITH_FULL_IMAGE:figures/full_fig_p024_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Irreducible variance Var( d z ∗ | zC , pj ) for T-JEPA (log scale, left; raw scale, right) across 5 seeds (0–4), each independently re-sampling 100K C4 sentences, showing the elevated conditional￾variance floor is not seed-specific. E SUPPLEMENTARY METRIC FOR ARGUMENT II (SEED 42) Cosine similarity [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Pairwise cosine-similarity statistics for T-JEPA across 5 seeds ( [PITH_FULL_IMAGE:figures/full_fig_p025_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Pairwise cosine similarity of I-JEPA and T-JEPA representations. Left: mean pairwise cosine similarity over training. T-JEPA rises to ≈1.0 after the instability point at step 9,340, while I-JEPA remains bounded away from 1. Centre: standard deviation and 95th percentile of validation cosine similarities. After collapse, T-JEPA has vanishing dispersion and near-unit upper quantiles. Right: late-training his… view at source ↗
Figure 10
Figure 10. Figure 10: t-SNE visualization of representation geometry for BERT and T-JEPA. [PITH_FULL_IMAGE:figures/full_fig_p026_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: 3D activation surfaces across self-supervised text encoders. Each surface shows last hidden-state activations for a held-out sentence, with token index on one axis and sampled hidden dimensions on the other. BERT exhibits strong token-wise and dimension-wise variation. VICReg, Barlow Twins, and BYOL retain visible activation structure, with variation across both tokens and dimensions. T-JEPA produces a ne… view at source ↗
Figure 12
Figure 12. Figure 12: Activation heatmaps across self-supervised text encoders. Each heatmap shows last hidden-state activations as a token-by-dimension matrix for the same held-out sentence. BERT and the non-JEPA SSL baselines retain structured variation across tokens and dimensions, whereas T￾JEPA is nearly uniform and low-amplitude, matching the effective-rank and cosine-collapse results. Qualitative structure. A non-degene… view at source ↗
Figure 13
Figure 13. Figure 13: Per-step representation heatmaps of the T-JEPA context encoder’s last hidden state for the [PITH_FULL_IMAGE:figures/full_fig_p028_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 1 canonical work pages

  1. [1]

    2006 , isbn =

    Elements of Information Theory , author =. 2006 , isbn =

    2006

  2. [2]

    Shannon, C. E. , title =. The Bell System Technical Journal , year =

  3. [3]

    Proceedings of the 36th International Conference on Machine Learning , pages =

    On Variational Bounds of Mutual Information , author =. Proceedings of the 36th International Conference on Machine Learning , pages =. 2019 , editor =

    2019

  4. [4]

    2018 , eprint =

    Representation Learning with Contrastive Predictive Coding , author =. 2018 , eprint =

    2018

  5. [5]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

    He, Kaiming and Fan, Haoqi and Wu, Yuxin and Xie, Saining and Girshick, Ross , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

  6. [6]

    A Path Towards Autonomous Machine Intelligence , author =

  7. [7]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

    Assran, Mahmoud and Duval, Quentin and Misra, Ishan and Bojanowski, Piotr and Vincent, Pascal and Rabbat, Michael and LeCun, Yann and Ballas, Nicolas , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

  8. [8]

    International Conference on Learning Representations , year =

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author =. International Conference on Learning Representations , year =

  9. [9]

    2019 , address =

    Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina , booktitle =. 2019 , address =

    2019

  10. [10]

    Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Li, Kai and Fei-Fei, Li , booktitle =

  11. [11]

    Journal of Machine Learning Research , volume =

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , author =. Journal of Machine Learning Research , volume =

  12. [12]

    Transactions on Machine Learning Research , year =

    Revisiting Feature Prediction for Learning Visual Representations from Video , author =. Transactions on Machine Learning Research , year =

  13. [13]

    IEEE International Conference on Multimedia and Expo , year =

    Tuncay, Ludovic and Labb. IEEE International Conference on Multimedia and Expo , year =

  14. [14]

    Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning , booktitle =

    Grill, Jean-Bastien and Strub, Florian and Altch. Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning , booktitle =

  15. [15]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Chen, Ting and Kornblith, Simon and Norouzi, Mohammad and Hinton, Geoffrey , title =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , volume =

    2020

  16. [16]

    Proceedings of the 38th International Conference on Machine Learning , pages =

    Barlow Twins: Self-Supervised Learning via Redundancy Reduction , author =. Proceedings of the 38th International Conference on Machine Learning , pages =. 2021 , volume =

    2021

  17. [17]

    Bardes, Adrien and Ponce, Jean and LeCun, Yann , booktitle =

  18. [18]

    2511.08544 , archivePrefix =

    Balestriero, Randall and LeCun, Yann , year =. 2511.08544 , archivePrefix =

    Pith/arXiv arXiv

  19. [19]

    2603.19312 , archivePrefix =

    Maes, Lucas and Le Lidec, Quentin and Scieur, Damien and LeCun, Yann and Balestriero, Randall , year =. 2603.19312 , archivePrefix =

    Pith/arXiv arXiv

  20. [20]

    OpenAI Blog , volume =

    Language Models are Unsupervised Multitask Learners , author =. OpenAI Blog , volume =

  21. [21]

    and Kaiser,

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser,. Attention is All You Need , booktitle =

  22. [22]

    Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D. and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Wint...

  23. [23]

    International Conference on Learning Representations , year =

    Understanding Dimensional Collapse in Contrastive Self-Supervised Learning , author =. International Conference on Learning Representations , year =

  24. [24]

    Proceedings of the 2015 Conference on Innovations in Theoretical Computer Science , pages =

    Feige, Uriel , title =. Proceedings of the 2015 Conference on Innovations in Theoretical Computer Science , pages =. 2015 , publisher =

    2015

  25. [25]

    2020 , address =

    Lewis, Mike and Liu, Yinhan and Goyal, Naman and Ghazvininejad, Marjan and Mohamed, Abdelrahman and Levy, Omer and Stoyanov, Veselin and Zettlemoyer, Luke , booktitle =. 2020 , address =

    2020

  26. [26]

    and Manning, Christopher D

    Clark, Kevin and Luong, Minh-Thang and Le, Quoc V. and Manning, Christopher D. , booktitle =

  27. [27]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

    Masked Autoencoders Are Scalable Vision Learners , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

  28. [28]

    Bao, Hangbo and Dong, Li and Piao, Songhao and Wei, Furu , booktitle =

  29. [29]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

    Exploring Simple Siamese Representation Learning , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

  30. [30]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =

    Emerging Properties in Self-Supervised Vision Transformers , author =. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =

  31. [31]

    Zhou, Jinghao and Wei, Chen and Wang, Huiyu and Shen, Wei and Xie, Cihang and Yuille, Alan and Kong, Tao , booktitle =

  32. [32]

    European Conference on Computer Vision , year =

    Masked Siamese Networks for Label-Efficient Learning , author =. European Conference on Computer Vision , year =

  33. [33]

    2022 , volume =

    Baevski, Alexei and Hsu, Wei-Ning and Xu, Qiantong and Babu, Arun and Gu, Jiatao and Auli, Michael , booktitle =. 2022 , volume =

    2022

  34. [34]

    Le, Trong and Thai, Phat and Nguyen, Sang and Hua, Minh and Pham, Ngan and Bui, Thang and Quan, Tho and Bui, Tuan , booktitle =

  35. [35]

    and Daly, Raymond E

    Maas, Andrew L. and Daly, Raymond E. and Pham, Peter T. and Huang, Dan and Ng, Andrew Y. and Potts, Christopher , title =. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , month =. 2011 , address =

    2011

  36. [36]

    and Angeli, Gabor and Potts, Christopher and Manning, Christopher D

    Bowman, Samuel R. and Angeli, Gabor and Potts, Christopher and Manning, Christopher D. A large annotated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 2015. doi:10.18653/v1/D15-1075

    doi:10.18653/v1/d15-1075 2015

  37. [37]

    MTEB : Massive Text Embedding Benchmark

    Muennighoff, Niklas and Tazi, Nouamane and Magne, Loic and Reimers, Nils. MTEB : Massive Text Embedding Benchmark. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023. doi:10.18653/v1/2023.eacl-main.148

    doi:10.18653/v1/2023.eacl-main.148 2023

  38. [38]

    2506.09985 , archivePrefix =

    Assran, Mido and Bardes, Adrien and Fan, David and Garrido, Quentin and Howes, Russell and Komeili, Mojtaba and Muckley, Matthew and Rizvi, Ammar and Roberts, Claire and Sinha, Koustuv and Zholus, Artem and Arnaud, Sergio and Gejji, Abha and Martin, Ada and Hogan, Francois Robert and Dugas, Daniel and Bojanowski, Piotr and Khalidov, Vasil and Labatut, Pat...

    Pith/arXiv arXiv

  39. [39]

    2311.15830 , archivePrefix =

    Fei, Zhengcong and Fan, Mingyuan and Huang, Junshi , year =. 2311.15830 , archivePrefix =

    Pith/arXiv arXiv

  40. [40]

    Vo, Khang H. N. and Nguyen, Duc P. T. and Nguyen, Thong T. and Quan, Tho T. , booktitle =

  41. [41]

    2512.10942 , archivePrefix =

    Chen, Delong and Shukor, Mustafa and Moutakanni, Theo and Chung, Willy and Yu, Jade and Kasarla, Tejaswi and Bolourchi, Allen and LeCun, Yann and Fung, Pascale , year =. 2512.10942 , archivePrefix =

    arXiv

  42. [42]

    Advances in Neural Information Processing Systems , volume =

    Connecting Joint-Embedding Predictive Architecture with Contrastive Self-Supervised Learning , author =. Advances in Neural Information Processing Systems , volume =

  43. [43]

    Hai Huang and Yann LeCun and Randall Balestriero , booktitle=

  44. [44]

    Sangdae Nam , booktitle=

  45. [45]

    Machine Learning , year =

    Rich Caruana , title =. Machine Learning , year =. doi:10.1023/A:1007379606734 , issn =

    doi:10.1023/a:1007379606734

  46. [46]

    International Conference on Learning Representations , year=

    Reinforcement Learning with Unsupervised Auxiliary Tasks , author=. International Conference on Learning Representations , year=

  47. [47]

    Neural Generation of Regular Expressions from Natural Language with Minimal Domain Knowledge

    Locascio, Nicholas and Narasimhan, Karthik and DeLeon, Eduardo and Kushman, Nate and Barzilay, Regina. Neural Generation of Regular Expressions from Natural Language with Minimal Domain Knowledge. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016. doi:10.18653/v1/D16-1197

    work page doi:10.18653/v1/d16-1197 2016

  48. [48]

    2021 , eprint=

    Training Verifiers to Solve Math Word Problems , author=. 2021 , eprint=

    2021

  49. [49]

    Proceedings of the 32nd ACM International Conference on Advances in Geographic Information Systems , pages =

    Li, Lihuan and Xue, Hao and Song, Yang and Salim, Flora , title =. Proceedings of the 32nd ACM International Conference on Advances in Geographic Information Systems , pages =. 2024 , isbn =. doi:10.1145/3678717.3691271 , abstract =

    arXiv 2024

  50. [50]

    The Thirteenth International Conference on Learning Representations , year =

    Hugo Thimonier and Jos. The Thirteenth International Conference on Learning Representations , year =