Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Bootleg shows that self-supervised visual representations improve sharply when a student predicts a teacher's hidden-layer embeddings at four depths instead of its final output or raw pixels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 18:05 UTC pith:XJM42ZHX

load-bearing objection Strong empirical recipe, but the paper's own Table 4 shows the hidden-layer target is a minor ingredient in the final model — the causal story is over-sold. the 3 major comments →

arxiv 2603.15553 v2 pith:XJM42ZHX submitted 2026-03-16 cs.CV cs.LG

Self-Distillation of Hidden Layers for Self-Supervised Representation Learning

Scott C. Lowe , Anthony Fuller , Sageev Oore , Evan Shelhamer , Graham W. Taylor This is my paper
classification cs.CV cs.LG
keywords self-supervised learningmasked image modelingself-distillationvision transformerhidden layersfeature predictionmasking strategyfrozen evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Bootleg proposes that the best self-supervised training signal for vision transformers sits between pixels and the network's final layer. The paper trains a student to predict, from a masked view of an image, the embeddings of the masked patches at four evenly spaced transformer blocks of an exponential-moving-average teacher (blocks 1, 4, 8, and 12). This 'hidden-self-distillation' objective is shown to be more stable than I-JEPA's final-layer distillation and to produce frozen-encoder features that beat MAE (masked autoencoder, pixel reconstruction), I-JEPA, data2vec 2.0, and CrossMAE by large margins on classification and segmentation probes. If correct, the recipe gives a practical, augmentation-free, batch-size-independent pretraining method that improves over the dominant generative and predictive self-supervised baselines.

Core claim

The paper's central claim is that the abstraction level of the distillation target determines the quality of the learned representation, and the optimum is neither raw input nor the final layer but several intermediate layers at once. Bootleg frames SSL as a hierarchical prediction task: given visible patches, the student reconstructs, from the same predictor, the teacher's representations at multiple depths — capturing low-, mid-, and high-level features simultaneously. In extensive ablations, the method shows that spaced-out, concatenated, individually z-scored targets from blocks {1,4,8,12} outperform single hidden targets, consecutive targets, averaged targets (data2vec-style), and the I

What carries the argument

Hidden-layer self-distillation with a multi-block masking strategy, under the method name Bootleg. A student vision transformer (ViT) sees a sparse set of image patches selected by four large, contiguous, overlapping rectangular masks; an EMA teacher encodes the full image; the student's predictor must take the masked patches' embeddings at blocks 1, 4, 8, and 12 of the teacher, z-score them per layer, concatenate them, and match them with an L2 loss. The two load-bearing pieces are (1) the diverse target set spanning the abstraction hierarchy and (2) the mask geometry: the paper shows that if masks are small or scattered (MAE-style random, green noise, inverse blocks), hidden-target distill

Load-bearing premise

The whole gain rests on the masking strategy: with MAE-style random or small masks, hidden-target training collapses at block 4 or deeper (Sec. 3.2, Table 5), so the success is not attributable to hidden targets alone but to the specific choice of large contiguous multi-block masks, whose adequacy is determined empirically per dataset rather than by a general principle.

What would settle it

An apples-to-apples replication that changes only the targets from {1,4,8,12} to {12} (same masking, predictor, registers, and hyperparameters) should show a large drop on ImageNet frozen probes — the paper's prototype ablations predict roughly 5 points on X-Blk; if the gap vanished, the multi-hidden-layer claim would be falsified. A second check targets the masking dependency: train on a dataset whose objects are small or scattered (e.g., medical or document images) where neighboring-patch correlation differs; the paper's own hypothesis predicts the contiguous-block strategy needs re-tuning,

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Frozen (probe-only) Bootleg encoders beat MAE, I-JEPA, data2vec 2.0, and CrossMAE at every ViT size on ImageNet-1k and iNaturalist-21 classification, with the widest margins at the smallest model size.
  • The pretraining transfers to dense prediction: Bootleg outperforms I-JEPA by roughly 10 mIoU points on ADE20K and Cityscapes frozen probes, and is best at ViT-S and ViT-B for fine-tuning ImageNet and ADE.
  • Replacing the final-layer target with hidden-layer targets is the single largest improvement over I-JEPA (+7.0 X-Blk on ImageNet-1k, +5.7 Lin on ADE), larger than any other recipe change such as masking or predictor size.
  • Hidden-self-distillation also improves other masked-image-modeling frameworks (MAE, CrossMAE, data2vec 2.0) once their masking is changed to large contiguous blocks — a gain of +6 to +12 points — showing the principle is general among single-view SSL methods.
  • Training is more stable: across 3 random seeds, Bootleg's probe accuracy standard deviation is far smaller than I-JEPA's (e.g., 0.3–0.8 vs 2.1–2.9 on ImageNet probes).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The mask-shape dependency suggests the real mechanism is preventing a spatial shortcut: when masked patches sit adjacent to seen ones, their correlated deep activations let the predictor cheat, so the contiguous-block requirement may need re-tuning per domain; a principled, data-dependent masking rule would be a natural next step the paper leaves open.
  • Because Bootleg needs no augmentations or batch interactions, the same hidden-distillation recipe should transfer to non-image modalities (audio, time series, point clouds) where final-layer self-distillation instability has also been reported; testing a {1,4} spacing on shallower stacks is a cheap falsifiable extension.
  • The abstraction-gradient view suggests hidden targets act as an implicit stabilizer: early-layer targets are more stimulus-driven and ground the training against collapse, which could be transplanted into other self-distillation settings (contrastive or multimodal) whose final-layer EMA targets are known to be unstable.
  • The information-bottleneck argument implies that distilling many layers into a single-layer student bottleneck forces genuine compression; a direct test would be measuring whether linear-probe transfer at intermediate student layers improves as target diversity increases.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Bootleg, a single-view masked self-supervised learning method for vision transformers. A teacher encoder (EMA of the student) processes the full image, while the student sees masked patches and a predictor is trained to regress per-patch, z-scored target embeddings collected from multiple hidden blocks of the teacher (e.g., blocks 1, 4, 8, 12 for ViT-S/B). Bootleg also includes block-masking improvements, CLS/register tokens, an enlarged predictor, and modified transforms. The authors report large frozen-probe gains over MAE, I-JEPA, and data2vec on ImageNet-1k, iNaturalist-21, VTAB, ADE20K, and Cityscapes, with the largest margins at ViT-S. They frame hidden-layer self-distillation as the core mechanism that makes the method work, and include extensive ablations, seed-variance analysis, and representational analyses.

Significance. If the empirical recipe holds, Bootleg is a substantial practical contribution to single-view, augmentation-light SSL: it improves frozen-encoder representations across classification and dense prediction, with a smaller-batch/no-negatives setup, and the paper includes welcome evidence on seed variance (Table 13) and a careful decomposition of differences from I-JEPA (Table 4). The manuscript is transparent about implementation details, masking bugs in prior code, and acknowledged limitations. However, the conceptual claim that hidden-layer prediction is the main driver is not actually supported by the authors' own ablations; the gains appear to be a joint effect of several interacting components. The paper needs a major reframing and additional analysis to make the central mechanistic claim commensurate with the evidence.

major comments (3)
  1. [Sec. 6.3/Table 4] The target-set row is the only ablation that directly isolates the paper's central mechanism within the final recipe. Changing I-JEPA's target from block 12 to {1,4,8,12} yields +7.0 X-Blk and +5.7 mIoU ('only with'), but removing hidden targets from Bootleg (74.4→73.9 and 26.6→25.2) costs only ~0.5 and 1.4 points ('only without'). The corresponding 'without' deltas for masking (+1.8), predictor (+1.9), CLS (+0.3), registers (+0.5), and hyperparameters (+1.2) are comparable or larger. Thus the headline Bootleg-versus-I-JEPA margin is not attributable mainly to hidden-layer self-distillation; it is a joint product of the whole recipe, including fixes to I-JEPA masking bugs. The abstract, Sec. 4, and Sec. 6.2 nonetheless call hidden targets 'the core mechanism' and 'the main conceptual difference.' This overstates what Table 4 supports. Please reframe the contribution as the full Bootleg r
  2. [Sec. 3.1-3.2, Table 5] The stability/grounding argument is not supported. Sec. 3.1 claims early-layer targets provide grounding because they are less processed, but Sec. 3.2 and Fig. 3 show that with MAE-style random masks, hidden-target self-distillation collapses for block 4 and deeper, and changing the masking strategy alone stabilizes training. Table 5 similarly shows random, green-noise, and inverse-block masks degrade or collapse even with Bootleg's targets. Hence the factor that determines stability in these experiments is mask block size/contiguity, not the depth or multiplicity of targets. The 'stimulus-driven grounding' explanation should either be tested (e.g., target entropy, gradient variance across mask/target configurations) or softened; as written, the mechanism is asserted rather than demonstrated.
  3. [Appx. C / Table 1] The comparison to I-JEPA is partly a comparison against the published implementation with known bugs documented in Appx. C: off-by-one mask placement, missing edge rows/columns, and visible-context truncation bias. Bootleg fixes these bugs as part of the method, so the large Table 1 margins conflate the new method with bug fixes. Table 4's 'only with' masking row (+5.9 X-Blk, +4.5 mIoU) quantifies this confound. The paper should present a fixed-masking I-JEPA (without hidden targets) as a primary baseline in Table 1, or state clearly in the abstract and headline that the gains are relative to the published I-JEPA implementation, not to an ideal I-JEPA with corrected masking.
minor comments (5)
  1. [Table 4] Target-row arithmetic: 'Only without' X-Blk is listed as +0.4, but 74.4−73.9 = 0.5. Please check rounding.
  2. [Sec. 6.3] The text calls changing targets 'the largest single improvement,' but AvgΔ is +3.6 for both Targets and Masking; it should be 'one of the largest.'
  3. [Abstract] '+10% vs I-JEPA' should be qualified: the >10-point margins occur at ViT-S on the X-Blk probe; ViT-B and ViT-L margins in Table 1 are roughly +6.8 and +8.3 X-Blk.
  4. [Appx. E.1.1] The text refers to 'five probe types' but lists four (Patch, CLS, X-Attn, X-Blk), and the main tables report X-Blk but not X-Attn. Clarify which probes are actually used.
  5. [Table 13] The seed-variance analysis is at 300 epochs, whereas the headline Table 1 uses 600-epoch single-seed models. State this explicitly near the seed-variance paragraph.

Circularity Check

0 steps flagged

No significant circularity: Bootleg's benchmark results are measured externally, its ablations quantify component contributions rather than assume them, and the EMA self-distillation loop is the method under test, not a fitted prediction.

full rationale

The paper does not contain a derivation whose 'prediction' reduces to a fitted constant or to a self-citation. Bootleg's training target is the EMA teacher's hidden-layer embeddings, which is self-referential by design, but this is the proposed method itself, not a claimed derivation of an independently defined result. Downstream evaluations on IN-1k, iNat21, ADE20K, Cityscapes, and VTAB are external benchmarks, and the baselines (MAE, I-JEPA, data2vec 2.0, CrossMAE) are trained or taken from released checkpoints independently of the paper's claims. The target-choice rule (every 4th block) is presented as an empirical rule-of-thumb, not as a consequence of a first-principles theorem. The ablation in Table 4 quantifies the marginal contribution of the multi-block target set in the final recipe (e.g., X-Blk 74.4 vs. 73.9 and ADE20K Lin 26.6 vs. 25.2, i.e. +0.4/+1.4 when removed); this undercuts the strength of the causal claim that hidden targets alone drive the improvements, but it is an evidentiary weakness, not circularity. There are no uniqueness theorems imported from the authors, no ansatz smuggled via self-citation, and no renaming of a known result. The self-citations that exist (e.g., Fuller et al. 2022, Safari et al. 2025) are used for domain-application context and are not load-bearing for the method's validity. The paper's own limitations section acknowledges that the masking strategy is empirically chosen and that no general principle yet selects it, which further confirms that no circular step is being hidden. Overall, the paper is self-contained against external benchmarks and its central claims are empirically testable; the circularity pass finds no specific reduction of any result to its own inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper's central contribution is an empirical training recipe with several tuned hyperparameters: the target layer set, masking strategy, EMA momentum, and normalization. The most fragile load-bearing choice is the masking configuration, since the paper itself shows that other masking strategies lead to instability or collapse. No new theoretical entities are introduced.

free parameters (5)
  • Target layer set L = {1,4,8,12} for 12-block ViT; {1,4,8,12,16,20,24} for ViT-L
    Chosen by ablation (Sec 6.1, Appx G.2.1). The 'every 4th block' rule is empirical; consecutive or averaged targets lower performance.
  • Masking configuration = 4 mask rectangles, mask area fraction [0.16,0.183], aspect ratio [0.667,1.5], seen rate ~29%
    Selected via sensitivity sweeps (Appx G.3). The central claim depends on this masking because random masks lead to collapse (Sec 3.2, 6.4).
  • EMA momentum = 0.9985
    Taken from hyperparameter search (Appx B); no sensitivity analysis is reported for this value.
  • Target standardization = per-patch per-layer z-score before concatenation
    Chosen over joint standardization; Appx H.4 shows separate standardization is important, but it is inherited from MAE/I-JEPA rather than derived.
  • Probe hyperparameter selection = best of 25 or 121 learning-rate/weight-decay configs
    Reported downstream numbers are best-over-sweep rather than typical or median, which inflates the absolute margins somewhat.
axioms (4)
  • domain assumption EMA teacher weights provide a stable, learnable target for hidden-layer prediction.
    Standard for BYOL/I-JEPA; not proven for hidden layers. Invoked in Sec 4 where the teacher-encoder is an EMA of the student.
  • domain assumption Hidden layers of a ViT progress from low-level to high-level abstraction, so spaced targets cover complementary levels.
    Sec 3.1 cites prior work (Zeiler & Fergus, Raghu, Park, etc.); this is required for the claim that blocks 1,4,8,12 provide diverse signal.
  • domain assumption Predicting hidden representations is a legitimate SSL objective that does not require additional collapse prevention.
    Central to the method; empirically shown only under block masking. Random masks cause collapse (Sec 3.2, Table 5), so the assumption is not universally valid.
  • ad hoc to paper Per-patch z-scoring of target embeddings prevents collapse and is a sufficient normalization.
    Inherited from MAE/I-JEPA; Appx H.4 shows joint standardization hurts, but no mechanism is provided for why separate z-scoring works.

pith-pipeline@v1.3.0-alltime-deepseek · 49572 in / 12149 out tokens · 122429 ms · 2026-08-02T18:05:01.278758+00:00 · methodology

0 comments
read the original abstract

The landscape of self-supervised learning (SSL) is currently dominated by generative approaches (e.g. MAE) that reconstruct raw low-level data, and predictive approaches (e.g. I-JEPA) that predict high-level abstract embeddings. While generative methods are stable due to their reliable training targets based on ground-truth data, they are computationally inefficient for high-redundancy modalities like imagery, and their training objective does not prioritize learning high-level, conceptual features. Conversely, predictive methods often suffer from training instability due to their reliance on the non-stationary targets of final-layer self-distillation. We introduce Bootleg, a method that bridges this divide by tasking the model with predicting latent representations from multiple hidden layers of a teacher. This hierarchical objective forces the model to capture features at varying levels of abstraction simultaneously. We demonstrate Bootleg significantly outperforms comparable baselines (+10% vs. I-JEPA) on frozen probe classification of ImageNet-1K, iNaturalist-21, and VTAB, and semantic segmentation of ADE20K, Cityscapes, and COCO-Stuff.

Figures

Figures reproduced from arXiv: 2603.15553 by Anthony Fuller, Evan Shelhamer, Graham W. Taylor, Sageev Oore, Scott C. Lowe.

Figure 1
Figure 1. Figure 1: Multi-layer self-distillation with Bootleg. The teacher-encoder (blue), student-encoder (green), and predictor (orange) are ViTs, made of repeated transformer blocks. A schematic of a single transformer block is overlaid (bottom right). The teacher-encoder is an EMA of the student-encoder, and processes the full image. The student-encoder sees a subset of the image and must create embeddings of them to fac… view at source ↗
Figure 2
Figure 2. Figure 2: Bridging I-JEPA and MAE with targets across hidden layers. Left: We train ViT-S with I-JEPA, except for the target which we change to be a hidden layer (𝑥-axis) of the teacher-encoder instead of its final output (blue curve) or in addition to the final output (red). We plot the frozen attentive probe top-1 accuracy on IN-1k (𝑦-axis) for the respective encoders. Middle: Similar, but for ViT-B to verify at l… view at source ↗
Figure 3
Figure 3. Figure 3: Accompanies Fig. 2 of the main text [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Sample masks generated by I-JEPA and Bootleg’s masking strategies. Visible tokens are shown in blue. Prediction masks are shown in green/yellow/orange/red with one colour per mask rectangle. These prediction masks can overlap, leading to brighter shades of orange/yellow. Unused tokens (seen by the teacher-encoder, but not by either the student-encoder or predictor) are shown in black. For both I-JEPA and B… view at source ↗
Figure 5
Figure 5. Figure 5: Rates at which each token position within the 14 ×14 grid is visible and presented to the student-encoder. Top row: I-JEPA masking. Bottom row: Bootleg masking. We show the change in visibility of the tokens as the per-GPU batch size is increased from 4 (left) to 1024 (right). Red marks on the colorbars indicate the least and most frequent any token is visible for a given plot. Note that token visibility i… view at source ↗
Figure 6
Figure 6. Figure 6: Token occurrence rates versus per-GPU batch size for I-JEPA and Bootleg masking strategies. Due to stochastic overlap of the predictor masks, the visible tokens need to be truncated to be the same length across all samples on the same GPU. Top left: Fraction of tokens which are visible (shown to the encoder) decreases as the batch size increases. This is because the target length for the visible tokens is … view at source ↗
Figure 6
Figure 6. Figure 6: Inter-layer Pearson correlation matrices for Bootleg ViT-S/16 and ViT-B/16 encoders. Each cell shows the mean correlation between embedding vectors at corresponding spatial positions across 10,000 images. Correlation decays with layer distance, and the final block is highly distinct from early blocks [PITH_FULL_IMAGE:figures/full_fig_p038_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Inter-layer CKA similarity matrices. CKA captures shared representational geometry and yields higher values than Pearson correlation for distant layer pairs, but the two measures are strongly rank-correlated (𝜌 ≈ 0.89). 0 2 4 6 8 10 12 Layer distance k 0.0 0.2 0.4 0.6 0.8 1.0 Similarity ViT-S/16 Pearson r CKA 0 2 4 6 8 10 12 Layer distance k Similarity ViT-B/16 Pearson r CKA [PITH_FULL_IMAGE:figures/full_… view at source ↗
Figure 8
Figure 8. Figure 8: Off-diagonal decay of Pearson correlation and CKA as a function of layer distance 𝑘. CKA decays more slowly, remaining above 0.15 even at the maximum distance, while Pearson correlation drops near zero. Error bars show standard deviation across all layer pairs at each distance. Tok b1 b2 b3 b4 b5 b6 b7 b8 b9 b10 b11 b12 Final 0.0 0.2 0.4 0.6 0.8 1.0 P e arso n r with targ et la y er ViT-S/16 Target: b1 Tar… view at source ↗
Figure 9
Figure 9. Figure 9: Pearson correlation of each encoder layer with the four Bootleg distillation target layers (blocks 1, 4, 8, and 12). Each target’s profile peaks at its own block and decays with distance, confirming that the targets capture non-redundant representations. 39 [PITH_FULL_IMAGE:figures/full_fig_p039_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Azimuthally averaged spatial correlation vs. distance for selected encoder layers. Early layers exhibit localized corre￾lations that decay with distance. Block 12 shows near-uniform high correlation, indicating a globally coherent representation. The post-LayerNorm final output partially restores spatial structure. 40 [PITH_FULL_IMAGE:figures/full_fig_p040_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Learn from your own latents and not from tokens: A sample-complexity theory

    cs.LG 2026-05 unverdicted novelty 7.0

    Latent prediction SSL recovers latent trees from PCFG data with sample complexity constant in hierarchy depth L (up to logs), unlike exponential for token-level or supervised methods.

Reference graph

Works this paper leans on

79 extracted references · 6 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [3]

    V-JEPA 2 : Self-supervised video models enable understanding, prediction and planning

    Mahmoud Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, X...

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.09985 2025

  3. [4]

    MultiMAE : Multi-modal multi-task masked autoencoders

    Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE : Multi-modal multi-task masked autoencoders. In Shai Avidan, Gabriel Brostow, Moustapha Ciss \'e , Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision -- ECCV 2022, pp.\ 348--367, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-19836-6. doi:10.1007/978-3-031-1983...

    work page doi:10.1007/978-3-031-19836-6 2022

  4. [5]

    data2vec: A general framework for self-supervised learning in speech, vision and language

    Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162...

    2022

  5. [6]

    Efficient self-supervised learning with contextualized target representations for vision, speech and language

    Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, vol...

    2023

  6. [7]

    BEiT : BERT pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT : BERT pre-training of image transformers. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=p-BhZSz59o4

    2022

  7. [8]

    VICR eg: Variance-invariance-covariance regularization for self-supervised learning

    Adrien Bardes, Jean Ponce, and Yann LeCun. VICR eg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=xm6YD62D1Ub

    2022

  8. [9]

    Revisiting feature prediction for learning visual representations from video

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. doi:10.48550/arxiv.2404.08471. Featured Certification

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.08471 2024

  9. [10]

    Network dissection: Quantifying interpretability of deep visual representations

    David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 3319--3327, 2017. doi:10.1109/CVPR.2017.354

    doi:10.1109/cvpr.2017.354 2017

  10. [11]

    Perception encoder: The best visual embeddings are not at the output of the network

    Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Abdul Rasheed, Junke Wang, Marco Monteiro, Hu Xu, Shiyu Dong, Nikhila Ravi, Shang-Wen Li, Piotr Dollar, and Christoph Feichtenhofer. Perception encoder: The best visual embeddings are not at the output of the network. In The...

    2025

  11. [12]

    Unsupervised learning of visual features by contrasting cluster assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 9912--9924. Curran Associates, Inc., 2020. UR...

    2020

  12. [14]

    VL-JEPA : Joint embedding predictive architecture for vision-language

    Delong Chen, Mustafa Shukor, Theo Moutakanni, Willy Chung, Jade Yu, Tejaswi Kasarla, Yejin Bang, Allen Bolourchi, Yann LeCun, and Pascale Fung. VL-JEPA : Joint embedding predictive architecture for vision-language. arXiv preprint arXiv:2512.10942, 2025. doi:10.48550/arxiv.2512.10942

    doi:10.48550/arxiv.2512.10942 2025

  13. [15]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.\ 1597--1607. PMLR, 13--18 Jul 2020 a . URL https...

    2020

  14. [16]

    Exploring simple siamese representation learning

    Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 15745--15753, 2021. doi:10.1109/CVPR46437.2021.01549

    arXiv 2021

  15. [17]

    Improved baselines with momentum contrastive learning

    Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020 b . doi:10.48550/arxiv.2003.04297

    Pith/arXiv arXiv doi:10.48550/arxiv.2003.04297 2003

  16. [19]

    Whatever next? P redictive brains, situated agents, and the future of cognitive science

    Andy Clark. Whatever next? P redictive brains, situated agents, and the future of cognitive science. Behav Brain Sci, 36 0 (3): 0 181--204, May 2013. doi:10.1017/s0140525x12000477

    doi:10.1017/s0140525x12000477 2013

  17. [20]

    SatMAE : Pre-training transformers for temporal and multi-spectral satellite imagery

    Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon. SatMAE : Pre-training transformers for temporal and multi-spectral satellite imagery. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 19...

    2022

  18. [21]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 3213--3223, June 2016

    2016

  19. [22]

    Vision transformers need registers

    Timoth \'e e Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=2dnO3LLiJ1

    2024

  20. [23]

    Cluster and predict latents patches for improved masked image modeling

    Timoth \'e e Darcet, Federico Baldassarre, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Cluster and predict latents patches for improved masked image modeling. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=Ycmz7qJxUQ

    2025

  21. [24]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol...

    doi:10.18653/v1/ 2019

  22. [25]

    PeCo : Perceptual codebook for BERT pre-training of vision transformers

    Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, Nenghai Yu, and Baining Guo. PeCo : Perceptual codebook for BERT pre-training of vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 552--560, Jun 2023. doi:10.1609/aaai.v37i1.25130

    work page doi:10.1609/aaai.v37i1.25130 2023

  23. [26]

    Brain-JEPA : Brain dynamics foundation model with gradient positioning and spatiotemporal masking

    Zijian Dong, Ruilin Li, Yilei Wu, Thuan Tinh Nguyen, Joanna Su Xian Chong, Fang Ji, Nathanael Ren Jie Tong, Christopher Li Hsian Chen, and Juan Helen Zhou. Brain-JEPA : Brain dynamics foundation model with gradient positioning and spatiotemporal masking. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances ...

    doi:10.52202/079017-2732 2024

  24. [27]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https...

    2021

  25. [28]

    Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C. Mozer. H ead2 T oe: Utilizing intermediate representations for better transfer learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machi...

    2022

  26. [29]

    A-JEPA : Joint-embedding predictive architecture can listen

    Zhengcong Fei, Mingyuan Fan, and Junshi Huang. A-JEPA : Joint-embedding predictive architecture can listen. arXiv preprint arXiv:2311.15830, 2024. doi:10.48550/arxiv.2311.15830

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.15830 2024

  27. [30]

    A theory of cortical responses

    Karl Friston. A theory of cortical responses. Philosophical Transactions of the Royal Society B: Biological Sciences, 360 0 (1456): 0 815--836, 2005. doi:10.1098/rstb.2005.1622

    arXiv 2005

  28. [31]

    Predictive coding under the free-energy principle

    Karl Friston and Stefan Kiebel. Predictive coding under the free-energy principle. Philos Trans R Soc Lond B Biol Sci, 364 0 (1521): 0 1211--1221, May 2009. doi:10.1098/rstb.2008.0300

    arXiv 2009

  29. [32]

    Efros, and Ken Goldberg

    Letian Fu, Long Lian, Renhao Wang, Baifeng Shi, XuDong Wang, Adam Yala, Trevor Darrell, Alexei A. Efros, and Ken Goldberg. Rethinking patch dependence for masked autoencoders. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=JT2KMuo2BV

    2025

  30. [33]

    Anthony Fuller, Koreen Millard, and James R. Green. SatViT : Pretraining transformers for earth observation. IEEE Geoscience and Remote Sensing Letters, 19: 0 1--5, 2022. doi:10.1109/LGRS.2022.3201489

    arXiv 2022

  31. [34]

    Bootstrap your own latent - a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altch\' e , Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent - a new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell...

    2020

  32. [35]

    Using a joint-embedding predictive architecture for symbolic music understanding

    Rafik Hachana and Bader Rasheed. Using a joint-embedding predictive architecture for symbolic music understanding. In AI for Music Workshop, 2025. URL https://openreview.net/forum?id=lieErtGZb6

    2025

  33. [37]

    ColorMAE : Exploring data-independent masking strategies in masked autoencoders

    Carlos Hinojosa, Shuming Liu, and Bernard Ghanem. ColorMAE : Exploring data-independent masking strategies in masked autoencoders. In Ale s Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and G \"u l Varol (eds.), Computer Vision -- ECCV 2024, pp.\ 432--449, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-72661-3. doi:10.10...

    doi:10.1007/978-3-031-72661-3 2024

  34. [38]

    Generic decoding of seen and imagined objects using hierarchical visual features

    Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, 8 0 (1): 0 15037, May 2017. ISSN 2041-1723. doi:10.1038/ncomms15037

    doi:10.1038/ncomms15037 2017

  35. [39]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  36. [40]

    Masked autoencoders that listen

    Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 28708--28720. Curran Associates, Inc., 2022. URL https:...

    2022

  37. [41]

    iNaturalist 2021 competition dataset

    iNaturalist 2021 competition dataset. iNaturalist 2021 competition dataset. https://github.com/visipedia/inat_comp/tree/master/2021, 2021

    2021

  38. [42]

    Keller and Thomas D

    Georg B. Keller and Thomas D. Mrsic-Flogel. Predictive processing: A canonical cortical computation. Neuron, 100 0 (2): 0 424--435, Oct 2018. ISSN 0896-6273. doi:10.1016/j.neuron.2018.10.003

    doi:10.1016/j.neuron.2018.10.003 2018

  39. [43]

    Similarity of neural network representations revisited

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning (ICML), pp.\ 3519--3529, 2019

    2019

  40. [44]

    M3- JEPA : Multimodal alignment via multi-gate M o E based on the joint-embedding predictive architecture

    Hongyang Lei, Xiaolong Cheng, Qi Qin, Dan Wang, Huazhen Huang, Qingqing Gu, Yetao Wu, and Luo Ji. M3- JEPA : Multimodal alignment via multi-gate M o E based on the joint-embedding predictive architecture. In Proceedings of the 42nd International Conference on Machine Learning (ICML). PMLR, 2025. URL https://proceedings.mlr.press/v267/lei25b.html

    2025

  41. [45]

    Lepori, Alexa R

    Michael A. Lepori, Alexa R. Tartaglini, Wai Keen Vong, Thomas Serre, Brenden M. Lake, and Ellie Pavlick. Beyond the doors of perception: Vision transformers represent relations between objects. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp.\ 131...

    work page doi:10.52202/079017-4180 2024

  42. [46]

    Ti-MAE : Self-supervised masked time series autoencoders

    Zhe Li, Zhongwen Rao, Lujia Pan, Pengyun Wang, and Zenglin Xu. Ti-MAE : Self-supervised masked time series autoencoders. arXiv preprint arXiv:2301.08871, 2023. doi:10.48550/arxiv.2301.08871

    Pith/arXiv arXiv doi:10.48550/arxiv.2301.08871 2023

  43. [47]

    Connecting joint-embedding predictive architecture with contrastive self-supervised learning

    Shentong Mo and Shengbang Tong. Connecting joint-embedding predictive architecture with contrastive self-supervised learning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp.\ 2348--2377. Curran Associates, Inc., 2024. doi:10.52202/079017-0077

    work page doi:10.52202/079017-0077 2024

  44. [48]

    Self-supervised predictive learning accounts for cortical layer-specificity

    Kevin Kermani Nejad, Paul Anastasiades, Loreen Hert \"a g, and Rui Ponte Costa. Self-supervised predictive learning accounts for cortical layer-specificity. Nature Communications, 16 0 (1): 0 6178, Jul 2025. ISSN 2041-1723. doi:10.1038/s41467-025-61399-5

    work page doi:10.1038/s41467-025-61399-5 2025

  45. [49]

    Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam

    Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=Jbdc0vTOcol

    2023

  46. [50]

    Maxime Oquab, Timoth \'e e Darcet, Th \'e o Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Pat...

    2024

  47. [51]

    What do self-supervised vision transformers learn? In The Eleventh International Conference on Learning Representations, 2023

    Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, and Sangdoo Yun. What do self-supervised vision transformers learn? In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=azCKuYyS74

    2023

  48. [52]

    Do vision transformers see like convolutional neural networks? In M

    Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 12116--12128. Curran Associates, Inc., 2021. URL h...

    2021

  49. [53]

    Rajesh P. N. Rao and Dana H. Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2 0 (1): 0 79--87, Jan 1999. ISSN 1546-1726. doi:10.1038/4580

    doi:10.1038/4580 1999

  50. [54]

    Vo, Camille Couprie, Patrick Labatut, Piotr Bojanowski, Valentin Wyart, and Jean-Rémi King

    Joséphine Raugel, Marc Szafraniec, Huy V. Vo, Camille Couprie, Patrick Labatut, Piotr Bojanowski, Valentin Wyart, and Jean-Rémi King. Disentangling the factors of convergence between brains and computer vision models. arXiv preprint arXiv:2508.18226, 2025. doi:10.48550/arxiv.2508.18226

    Pith/arXiv arXiv doi:10.48550/arxiv.2508.18226 2025

  51. [55]

    Stem-JEPA: A Joint-Embedding Predictive Architecture for Musical Stem Compatibility Estimation

    Alain Riou, Stefan Lattner, Ga \" e tan Hadjeres, Michael Anslow, and Geoffroy Peeters. Stem-JEPA: A Joint-Embedding Predictive Architecture for Musical Stem Compatibility Estimation . In Proceedings of the 25th International Society for Music Information Retrieval Conference, San Francisco, nov 2024. ISMIR

    2024

  52. [56]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115 0 (3): 0 211--252, Apr 2015. doi:10.1007/s11263-015-0816-y

    doi:10.1007/s11263-015-0816-y 2015

  53. [57]

    Enhancing DNA Foundation Models to Address Masking Inefficiencies

    Monireh Safari, Pablo Millan Arias, Scott C. Lowe, Lila Kari, Angel X. Chang, and Graham W. Taylor. Enhancing DNA foundation models to address masking inefficiencies. arXiv preprint arXiv:2502.18405, 2025. doi:10.48550/arxiv.2502.18405

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2502.18405 2025

  54. [58]

    Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2508.10104 2025

  55. [59]

    Segmenter: Transformer for semantic segmentation

    Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 7242--7252, 2021. doi:10.1109/ICCV48922.2021.00717

    arXiv 2021

  56. [60]

    RoFormer : Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer : Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024. ISSN 0925-2312. doi:10.1016/j.neucom.2023.127063

    arXiv 2024

  57. [61]

    Many-two-one: Diverse representations across visual pathways emerge from a single objective

    Yingtian Tang, Abdulkadir Gokce, Khaled Jedoui Al-Karkari, Daniel Yamins, and Martin Schrimpf. Many-two-one: Diverse representations across visual pathways emerge from a single objective. bioRxiv, 2025. doi:10.1101/2025.07.22.664908

    doi:10.1101/2025.07.22.664908 2025

  58. [62]

    T- JEPA : Augmentation-free self-supervised learning for tabular data

    Hugo Thimonier, Jos \'e Lucas De Melo Costa, Fabrice Popineau, Arpad Rimmel, and Bich-Li \^e n Doan. T- JEPA : Augmentation-free self-supervised learning for tabular data. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=gx3LMRB15C

    2025

  59. [63]

    Pereira, and William Bialek

    Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proceedings of the 37th Allerton Conference on Communication, Control, and Computing, pp.\ 368--377, 1999. doi:10.48550/arXiv.physics/0004057

    Pith/arXiv arXiv doi:10.48550/arxiv.physics/0004057 1999

  60. [64]

    Audio-JEPA : Joint-embedding predictive architecture for audio representation learning

    Ludovic Tuncay, Etienne Labbé, Emmanouil Benetos, and Thomas Pellegrini. Audio-JEPA : Joint-embedding predictive architecture for audio representation learning. arXiv preprint arXiv:2507.02915, 2025. doi:10.48550/arxiv.2507.02915

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.02915 2025

  61. [65]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. doi:10.48550/arxiv.1807.03748

    Pith/arXiv arXiv doi:10.48550/arxiv.1807.03748 2018

  62. [66]

    The iNaturalist species classification and detection dataset

    Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 8769--8778, Los Alamitos, CA, USA, June 2018. IEEE Computer Society. doi:10...

    arXiv 2018

  63. [67]

    Shashanka Venkataramanan, Valentinos Pariza, Mohammadreza Salehi, Lukas Knobel, Spyros Gidaris, Elias Ramzi, Andrei Bursuc, and Yuki M. Asano. Franca: Nested matryoshka clustering for scalable visual representation learning. arXiv preprint arXiv:2507.14137, 2025. doi:10.48550/arxiv.2507.14137

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.14137 2025

  64. [68]

    Vilas, Timothy Schauml\" o ffel, and Gemma Roig

    Martina G. Vilas, Timothy Schauml\" o ffel, and Gemma Roig. Analyzing vision transformers for image classification in class embedding space. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 40030--40041. Curran Associates, Inc., 2023. URL https://proceedings.n...

    2023

  65. [69]

    VideoMAE V2 : Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. VideoMAE V2 : Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 14549--14560, 2023. doi:10.1109/CVPR52729.2023.01398

    arXiv 2023

  66. [71]

    Delving into masked autoencoders for multi-label thorax disease classification

    Junfei Xiao, Yutong Bai, Alan Yuille, and Zongwei Zhou. Delving into masked autoencoders for multi-label thorax disease classification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp.\ 3577--3589, 2023. doi:10.1109/WACV56688.2023.00358

    arXiv 2023

  67. [72]

    SimMiM : A simple framework for masked image modeling

    Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. SimMiM : A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 9643--9653, 2022. doi:10.1109/CVPR52688.2022.00943

    arXiv 2022

  68. [73]

    Daniel L. K. Yamins and James J. DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19 0 (3): 0 356--365, Mar 2016. ISSN 1546-1726. doi:10.1038/nn.4244

    doi:10.1038/nn.4244 2016

  69. [74]

    Daniel L. K. Yamins, Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111 0 (23): 0 8619--8624, 2014. doi:10.1073/pnas.1403112111

    doi:10.1073/pnas.1403112111 2014

  70. [75]

    Learning efficient coding of natural images with maximum manifold capacity representations

    Thomas Yerxa, Yilun Kuang, Eero Simoncelli, and SueYeon Chung. Learning efficient coding of natural images with maximum manifold capacity representations. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 24103--24128. Curran Associates, Inc., 2023. URL https:/...

    2023

  71. [76]

    WavJEPA : Semantic learning unlocks robust audio foundation models for raw waveforms

    Goksenin Yuksel, Pierre Guetschel, Michael Tangermann, Marcel van Gerven, and Kiki van der Heijden. WavJEPA : Semantic learning unlocks robust audio foundation models for raw waveforms. arXiv preprint arXiv:2509.23238, 2025. doi:10.48550/arxiv.2509.23238

    doi:10.48550/arxiv.2509.23238 2025

  72. [77]

    Barlow twins: Self-supervised learning via redundancy reduction

    Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 12310--12320. PMLR, 18--24 Jul 2021. URL https://procee...

    2021

  73. [78]

    Zeiler and Rob Fergus

    Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), European Conference on Computer Vision, pp.\ 818--833, Cham, 2014. Springer International Publishing. doi:10.1007/978-3-319-10590-1\_53

    doi:10.1007/978-3-319-10590-1 2014

  74. [79]

    A large-scale study of representation learning with the visual task adaptation benchmark

    Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andr \'e Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task...

    Pith/arXiv arXiv doi:10.48550/arxiv.1910.04867 1910

  75. [80]

    Point- M2AE : Multi-scale masked autoencoders for hierarchical point cloud pre-training

    Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point- M2AE : Multi-scale masked autoencoders for hierarchical point cloud pre-training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 27061--27074. Curran Associa...

    2022

  76. [81]

    Object detectors emerge in deep scene CNNs

    Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene CNNs . In International Conference on Learning Representations, 2015. doi:10.48550/arxiv.1412.6856

    Pith/arXiv arXiv doi:10.48550/arxiv.1412.6856 2015

  77. [82]

    Scene parsing through ADE20K dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 5122--5130, 2017. doi:10.1109/CVPR.2017.544

    doi:10.1109/cvpr.2017.544 2017

  78. [83]

    Image BERT pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=ydopy-e6Dg

    2022

  79. [84]

    Self pre-training with masked autoencoders for medical image classification and segmentation

    Lei Zhou, Huidong Liu, Joseph Bae, Junjun He, Dimitris Samaras, and Prateek Prasanna. Self pre-training with masked autoencoders for medical image classification and segmentation. In Proceedings of the IEEE 20th International Symposium on Biomedical Imaging (ISBI), pp.\ 1--6. IEEE, 2023. doi:10.1109/ISBI53787.2023.10230477

    arXiv 2023