Pith. sign in

REVIEW 4 major objections 5 minor 74 references

Probing the Mid-level Vision Capabilities of Self-Supervised Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read High ImageNet accuracy doesn't guarantee mid-level vision

desk verdict Useful controlled benchmark with a real MAE/MaskFeat finding, but the headline pooled correlation is confounded by architecture and needs within-family analysis. read the letter →

arxiv 2411.17474 v2 pith:A3JKMX2A submitted 2024-11-25 cs.CV

Xuweiyi Chen , Markus Marks , Zezhou Cheng This is my paper · ORCID
classification cs.CV
keywords self-supervisedlearningmid-levelvisionbenchmarkdepthestimationsurfacenormalsgeometriccorrespondenceImageNetlinearprobingtransformers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a self-supervised visual model that excels at high-level recognition (ImageNet classification) also understands mid-level properties: where objects end, how far surfaces are, and which pixels match across views. The answer is "yes, but only loosely": across 22 ImageNet-1K SSL checkpoints and 8 mid-level tasks, mid-level performance does rise with ImageNet linear probing accuracy, yet the $R^2$ correlation is much weaker than the correlation among high-level tasks themselves, and for scene surface normals and scene geometric correspondence it is near zero. The paper also finds that generative SSL objectives, especially iBOT and MaskFeat, lead on mid-level tasks while MAE trails despite its strong ImageNet accuracy, and that ViT backbones and larger models help. If this holds, ImageNet accuracy is an incomplete yardstick for representation quality, and SSL benchmarks should include mid-level probes.

What carries the argument

The probing machinery is a fixed set of evaluation protocols applied to frozen representations: a DPT decoder trained on top of frozen features for dense tasks (generic object segmentation, depth, surface normals), and training-free feature matching for correspondence and similarity tasks using cosine distances between CLS or global tokens and dense feature matching. The organizing quantity is the $R^2$ coefficient of determination between each mid-level task's score and ImageNet linear probing accuracy across 22 models, which quantifies how much mid-level capability is explained by high-level recognition. The contrast that carries the argument is the paired comparison between these $R^2$ values and the $R^2$ among high-level tasks, plus the case studies of MAE and MaskFeat as outliers.

What would settle it

Retrain a representative set of SSL objectives (contrastive, self-distillation, masked reconstruction, HOG prediction) under one unified recipe with identical epochs, augmentations, batch size, and projection head on ImageNet-1K, then rerun the eight mid-level probes. If the $R^2$ between mid-level performance and ImageNet linear probing rises to the level seen among high-level tasks (for example above 0.6 for scene surface normals), or if MAE no longer underperforms on mid-level tasks, the paper's central claim about SSL objectives would be falsified.

Watch

Extended reading notes

Core claim

The paper's central empirical claim is that the mid-level vision capabilities of self-supervised models are positively correlated with their high-level vision performance, but this correlation is generally weaker than the correlation among high-level vision tasks. Concretely, regressing each mid-level task against ImageNet linear probing over 22 pretrained checkpoints yields $R^2$ values from 0.78 for mid-level image similarity and 0.70 for generic object segmentation down to 0.23 for scene depth, 0.08 for scene surface normals, and 0.03 for scene geometric correspondence, whereas high-level tasks correlate much more strongly with ImageNet probing. The paper further establishes that the pattern is not uniform: MaskFeat, a generative model trained to predict HOG features, is among the best mid-level models despite modest ImageNet accuracy, while MAE, strong on ImageNet, is among the worst mid-level models; iBOT and DINO lead in both; and within-method comparisons show ViT outperforms ResNet and larger backbones help.

Load-bearing premise

The controlled comparison assumes the 22 publicly released ImageNet-1K checkpoints differ only in SSL objective and architecture, when in practice they also differ in training recipe details such as epochs, augmentations, and projection heads.

Editorial extensions

If this is right

  • ImageNet linear probing is not a reliable proxy for mid-level vision; a model's rank can flip between high-level and mid-level benchmarks.
  • Generative objectives with mid-level-friendly targets, such as MaskFeat's HOG feature prediction, can produce strong mid-level representations without high recognition accuracy, suggesting the choice of training target matters for SSL design.
  • MAE's weak mid-level performance despite strong ImageNet accuracy shows that masked pixel reconstruction alone does not build geometric understanding.
  • Architecture decisions matter: ViT-B/16 beats ResNet-50 for the same SSL objective, and ViT-L/16 beats ViT-B/16, so architecture and capacity confound SSL comparisons.
  • Benchmarking SSL only on high-level tasks gives an incomplete picture of representation quality; mid-level probes should be part of standard evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension not in the paper: if the weak correlation is causal rather than an artifact of training-recipe differences, adding a HOG-prediction term to a contrastive objective should raise its scene surface-normal performance without hurting ImageNet accuracy.
  • Scene geometric correspondence has near-zero correlation with both recognition and other mid-level tasks, so it may be a better diagnostic for 3D-aware pretraining (such as cross-view completion) than for generic SSL, a possibility the paper leaves implicit.
  • The paper's controlled comparison is only as controlled as the publicly released checkpoints allow; retraining the same objectives under a single unified recipe could shrink or enlarge the observed gaps between objectives, and the ranking of methods might change.
  • The brief comparison with visual foundation models suggests scaling data and diversity, as in DINO-v2, may be a complementary route to mid-level competence; whether the mid-level/high-level correlation strengthens with pretraining scale is an open question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces a benchmark suite for probing the mid-level vision capabilities of self-supervised learning (SSL) models and evaluates 22 publicly available ImageNet-1K SSL checkpoints across eight mid-level vision tasks: generic object segmentation, scene/object depth estimation, scene/object surface normal estimation, scene/object geometric correspondence, and mid-level image similarity. The evaluation uses frozen features with DPT decoders for dense tasks and training-free correspondence/NIGHTS protocols for correspondence and similarity. The main empirical claim, stated in Sec. 5.1 and Fig. 2, is that mid-level vision performance is positively correlated with ImageNet linear probing accuracy, but generally more weakly than the correlation among high-level tasks. The paper also identifies imbalanced models (MAE low on mid-level despite strong high-level performance; MaskFeat strong on mid-level despite modest high-level results), analyzes factors such as pretraining objective, architecture, and model capacity, and compares the SSL models with several visual foundation models.

Significance. If the central empirical claim is established, the paper would be a useful and timely benchmark for the SSL community: it covers a broad range of SSL families, uses public checkpoints, provides task-specific protocols, and includes both training-free and frozen-feature probing, which is a practical and reproducible evaluation design. The paper also gives concrete per-model rankings and identifies MaskFeat and MAE as interesting outliers. However, the headline correlation result is not yet supported by the evidence as presented: the pooled regressions in Fig. 2 mix architectures and training recipes, no uncertainty or significance is reported, and Fig. 4 shows a negative within-family trend that contradicts the unqualified positive claim. These issues bear directly on the paper's central contribution, so the manuscript needs a major revision before the conclusions can be accepted.

major comments (4)
  1. [Sec. 5.1, Fig. 2] The central claim of a positive correlation between mid-level and high-level performance is computed by pooling 22 models that differ both in backbone (ResNet-50 vs. ViT-B/16) and in training recipe, since Table 5 lists checkpoints from VISSL, MMSelfSup, the SwAV repository, and an unofficial BYOL repository. Because ViT checkpoints tend to occupy the upper-right region of Fig. 2, a positive pooled regression can arise even if no within-architecture relationship exists. Fig. 4 in fact shows a negative trend among seven contrastive ResNet-50 models on depth (R²=0.13) and surface normals (R²=0.31). Please report within-architecture (ResNet-50-only and ViT-only) regressions for all eight tasks, and if the sign or strength differs, restate the claim with that qualification.
  2. [Sec. 5.1, Fig. 2] No measure of uncertainty accompanies the R² values, and with n=22 models the estimates have wide sampling variability. The reported values range from 0.03 (scene geometry correspondence) to 0.78 (mid-level image similarity), so the blanket statements in the abstract and Sec. 5.1 that the correlation is "weak" are an overstatement. Provide confidence intervals or permutation p-values for each regression and summarize the correlation per task rather than as a single global trend.
  3. [Sec. 5.2, Fig. 4] The negative-correlation result for contrastive SSLs is based on only seven ResNet-50 models, with no error bars or significance testing, and it directly undermines the unqualified positive claim in Sec. 5.1. Please add significance testing, include the "broader analysis across six additional tasks" promised in the text, and explicitly reconcile the negative within-family trend with the pooled positive correlation in Fig. 2.
  4. [Sec. 4, Table 5] The evaluation is described as a "controlled" comparison, but the selected checkpoints differ not only in SSL objective but also in pretraining epochs, augmentation strategies, projection heads, and implementation provenance (for instance, BYOL comes from an unofficial repository). These recipe differences are confounded with the SSL method, so the attributions in Sec. 5.2 about training objectives, architecture, and capacity should be framed as observational comparisons rather than controlled causal findings. At minimum, add a limitations paragraph that explicitly acknowledges this confound.
minor comments (5)
  1. [Table 3] In the MAE L/16 row, the entry "26.6 0" should presumably be "26.60".
  2. [Fig. 2, Fig. 4] The axis labels contain the corrupted string "Image/glyph1197et LP (%)", which should be fixed to "ImageNet LP (%)".
  3. [Sec. 5.2] The paragraphs titled "Network architecture" and "Network capability" are nearly identical and should be merged into one discussion.
  4. [References, Tables 7-8] MAE is cited inconsistently as reference [31] in Table 7 and as reference [32] in Table 8, and references [31] and [32] appear to be duplicate entries for the same work.
  5. [Sec. 5.4] The final sentence of the pre-text tasks paragraph, "it re-validate our claim that ImageNet (IN1k) probing results improve, we generally observe...", is grammatically broken and appears to restate Sec. 5.1; please rewrite.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports descriptive regressions on independently benchmarked checkpoints, and its self-citations are not load-bearing.

full rationale

The paper's central claim—that mid-level vision performance is positively correlated with high-level ImageNet linear probing, but more weakly than correlations among high-level tasks—is an empirical summary of measurements, not a derivation that reduces to its inputs. The mid-level task scores come from freezing 22 public ImageNet-1K checkpoints and evaluating them with DPT/AdaBins heads or training-free protocols on public datasets (VOC, NYUv2, NAVI, ScanNet, NIGHTS). The R2 values in Fig. 2 are descriptive linear-regression statistics computed from those independently obtained measurements; no fitted parameter is renamed as a prediction, and no quantity is defined in terms of the claimed conclusion. The only author self-citation is Marks et al. [41], used to motivate the selection of SSL methods and evaluation design, not as evidence for the correlation result; the probing protocols largely follow Probe3D [17], an independent prior work. The pooled-regression confound raised by the skeptic (architecture and training-recipe mixing, and the negative within-family trend in Fig. 4) is a statistical validity concern about the strength of the empirical claim, not a circularity in which the conclusion is equivalent to the input by construction. Under the specified review rules, that concern belongs in a correctness or robustness assessment rather than in the circularity score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new parameters fit to data and no new entities. It relies on domain assumptions about the validity of probing protocols, the representativeness of tasks, and the comparability of public checkpoints.

free parameters (2)
  • Dense feature extraction layers = ViT-B: layers 3, 6, 9, 12; ResNet: last 4 blocks
    Chosen by hand following Probe3D; determines what representations are probed and could affect rankings.
  • Correspondence recall thresholds = 5px, 25px, 50px (2D); 0.01m, 0.02m, 0.05m (3D)
    Standard thresholds from Probe3D; results depend on these choices.
assumptions (4)
  • domain assumption The frozen-backbone plus DPT decoder probing protocol is a valid measure of the representation's mid-level capabilities.
    Used throughout Sec. 4; if the decoder or layer selection masks or amplifies differences, the rankings and correlations would change.
  • domain assumption ImageNet-1K linear probing accuracy is an adequate proxy for high-level vision capability.
    Used in all correlation analyses (Fig. 2, 3); a different high-level measure could change the correlations.
  • domain assumption Public SSL checkpoints used are trained on ImageNet-1K with comparable training recipes, so performance differences reflect the SSL method rather than dataset size or training budget.
    The abstract and Sec. 4 claim a controlled study, but training epochs and other details of checkpoints vary and are not controlled.
  • domain assumption The chosen eight tasks adequately represent mid-level vision.
    Sec. 3 defines mid-level via Marr; the selection of tasks is a modeling choice that bounds the conclusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Probing the Mid-level Vision Capabilities of Self-Supervised Learning." pith.science (2026). https://pith.science/paper/A3JKMX2A

@misc{pith2026241117474,
  author       = {Pith},
  title        = {Pith review of: Probing the Mid-level Vision Capabilities of Self-Supervised Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A3JKMX2A}},
  note         = {Machine review of arXiv:2411.17474}
}
read the original abstract

Mid-level vision capabilities - such as generic object localization and 3D geometric understanding - are not only fundamental to human vision but are also crucial for many real-world applications of computer vision. These abilities emerge with minimal supervision during the early stages of human visual development. Despite their significance, current self-supervised learning (SSL) approaches are primarily designed and evaluated for high-level recognition tasks, leaving their mid-level vision capabilities largely unexamined. In this study, we introduce a suite of benchmark protocols to systematically assess mid-level vision capabilities and present a comprehensive, controlled evaluation of 22 prominent SSL models across 8 mid-level vision tasks. Our experiments reveal a weak correlation between mid-level and high-level task performance. We also identify several SSL methods with highly imbalanced performance across mid-level and high-level capabilities, as well as some that excel in both. Additionally, we investigate key factors contributing to mid-level vision performance, such as pretraining objectives and network architectures. Our study provides a holistic and timely view of what SSL models have learned, complementing existing research that primarily focuses on high-level vision tasks. We hope our findings guide future SSL research to benchmark models not only on high-level vision tasks but on mid-level as well.

Figures

Figures reproduced from arXiv: 2411.17474 by the authors.

Figure 1
Figure 1. Mid-Level Vision vs. High-Level Vision Tasks. We provide a comprehensive evaluation of prominent self-supervised learning methods (SSLs) across a wide range of mid-level vision tasks (a), complementing the standard evaluation in high-level vi￾sion tasks (b). Although SSL performance in mid-level vision tasks (e.g., depth estimation) is positively correlated with Ima￾geNet linear probing (c, left), this correlation i… view at source ↗
Figure 2
Figure 2. Mid-Level Vision v.s. ImageNet Linear Probing. We report the performance of self-supervised learning methods on mid-level vision tasks (y-axis) against their ImageNet 1K linear classification accuracy. Metrics are detailed in Tab. 2. Linear regression shows correlation with R 2 in each plot’s top-left corner, and the red shaded area marks the 95% confidence interval. 1 − cos fθ(x), fθ(˜x)  where fθ is the image rep… view at source ↗
Figure 3
Figure 3. Cross-task performance correlation. We present Pear￾son coefficients across mid-level and high-level vision tasks. We use the same set of metrics we used in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Ranking SSLs based on Overall Score. Comparative ranking of SSL models based on performance in mid-level vision tasks and ImageNet probing. Models are ranked per task, and average scores reflect overall performance across mid-level and high-level vision tasks, with hig…
Figure 6
Figure 6. Figure 6: Comparison of ImageNet1K-Pretrained SSLs with mod [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Qualitative Depth Estimation Results for Selected SSL Models. Depth estimation visualizations are shown for selected SSL models, with the δ1 score displayed below each visualization (higher is better). These results highlight the models’ effectiveness in capturing dept…
Figure 8
Figure 8. Figure 8: Qualitative Surface Normal Estimation Results for Selected SSL Models. Surface normal estimation visualizations are shown for selected SSL models, with the δ1 score displayed below each visualization (higher is better). These results highlight the models’ effectiveness…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 60 canonical work pages

  1. [1]

    Self-labelling via simultaneous clustering and rep- resentation learning, 2020

    Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and rep- resentation learning, 2020. 2

    work page 2020

  2. [2]

    Es- timating and exploiting the aleatoric uncertainty in surface normal estimation, 2021

    Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Es- timating and exploiting the aleatoric uncertainty in surface normal estimation, 2021. 4

    work page 2021

  3. [3]

    Deep clustering for unsupervised learning of visual features, 2019

    Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features, 2019. 3

    work page 2019

  4. [4]

    Unsupervised learning of visual features by contrasting cluster assignments, 2021

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Pi- otr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments, 2021. 4, 6, 2, 3, 8, 9, 10

    work page 2021

  5. [5]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, 2021. 1, 2, 3, 4, 8, 9, 10

    work page 2021

  6. [6]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020. 2, 4, 6, 1, 8, 9, 10

    work page 2020

  7. [7]

    Exploring simple siamese rep- resentation learning

    Xinlei Chen and Kaiming He. Exploring simple siamese rep- resentation learning. In CVPR, 2021. 2, 4, 8, 9, 10

    work page 2021

  8. [8]

    Improved baselines with momentum contrastive learning

    Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020. 1, 4, 2, 8, 9, 10

    arXiv 2003

Show all 74 references
  1. [9]

    An empirical study of training self-supervised vision transformers, 2021

    Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers, 2021. 2, 4, 6, 8, 3, 9, 10

    2021

  2. [10]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017. 4

    2017

  3. [11]

    Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner

    Angela Dai, Angel X. Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes, 2017. 5

    2017

  4. [12]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 2, 3, 4, 1

    2009

  5. [13]

    Maskclip: Masked self-distillation advances contrastive language-image pretraining, 2023

    Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Maskclip: Masked self-distillation advances contrastive language-image pretraining, 2023. 2

    2023

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 2, 3, 8

    2010 arXiv

  7. [15]

    Depth map prediction from a single image using a multi-scale deep net- work

    David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep net- work. In Neural Information Processing Systems, 2014. 4

    2014

  8. [16]

    Depth map prediction from a single image using a multi-scale deep net- work, 2014

    David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep net- work, 2014. 5, 4

    2014

  9. [17]

    Prob- ing the 3d awareness of visual foundation models

    Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Ab- hishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Prob- ing the 3d awareness of visual foundation models. In CVPR,

  10. [18]

    Hospedales

    Linus Ericsson, Henry Gouk, and Timothy M. Hospedales. How Well Do Self-Supervised Models Transfer? In CVPR,

  11. [19]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal- network.org/challenges/VOC/voc2007/workshop/index.html, . 4, 5, 8

    2007

  12. [20]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal- network.org/challenges/VOC/voc2012/workshop/index.html, . 4, 5, 8

    2012

  13. [21]

    Adabins: Depth estimation using adaptive bins

    Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021. 4

    2021

  14. [22]

    Fouhey, Wajahat Hussain, Abhinav Gupta, and Mar- tial Hebert

    David F. Fouhey, Wajahat Hussain, Abhinav Gupta, and Mar- tial Hebert. Single image 3d without a single 3d image. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1053–1061, 2015. 5, 4

    2015

  15. [23]

    Dream- sim: Learning new dimensions of human visual similarity using synthetic data, 2023

    Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dream- sim: Learning new dimensions of human visual similarity using synthetic data, 2023. 4

    2023

  16. [24]

    Dream- sim: Learning new dimensions of human visual similarity using synthetic data

    Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dream- sim: Learning new dimensions of human visual similarity using synthetic data. Advances in Neural Information Pro- cessing Systems, 36, 2024. 5

    2024

  17. [25]

    Un- supervised representation learning by predicting image rota- tions

    Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Un- supervised representation learning by predicting image rota- tions. In ICLR, 2018. 1, 2, 3, 4, 6, 8, 9, 10

    2018

  18. [26]

    Scaling and benchmarking self-supervised visual rep- resentation learning

    Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self-supervised visual rep- resentation learning. In Proceedings of the ieee/cvf Inter- national Conference on computer vision , pages 6391–6400,

  19. [27]

    Caltech-256 object category dataset

    Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007. 1

    2007

  20. [28]

    Richemond, Elena Buchatskaya, Carl Do- ersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Moham- mad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R´emi Munos, and Michal Valko

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Do- ersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Moham- mad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R´emi Munos, and Michal Valko. Bootstrap your own ...

    2020

  21. [29]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,

  22. [30]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In CVPR, 2020. 2

    2020

  23. [31]

    Masked autoencoders are scalable vision learners, 2021

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021. 2, 7, 3, 8, 9, 10

    2021

  24. [32]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 2, 4, 8, 9, 10

    2022

  25. [33]

    Visual intelligence: How we create what we see

    Donald D Hoffman. Visual intelligence: How we create what we see. WW Norton & Company, 2000. 1

    2000

  26. [34]

    Navi: Category- agnostic image collections with high-quality 3d shape and pose annotations

    Varun Jampani, Kevis-Kokitsi Maninis, Andreas Engel- hardt, Arjun Karpur, Karen Truong, Kyle Sargent, Stefan Popov, Andre Araujo, Ricardo Martin-Brualla, Kaushal Pa- tel, Daniel Vlasic, Vittorio Ferrari, Ameesh Makadia, Ce Liu, Yuanzhen Li, and Howard Zhou. Navi: Category- agn...

    2023

  27. [35]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In ICCV, 2023. 3, 8

    2023

  28. [36]

    Belongie, Lubomir D

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll’a r, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312,

    arXiv

  29. [37]

    Feature pyra- mid networks for object detection

    Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyra- mid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2117–2125, 2017. 3

    2017

  30. [38]

    Pixmim: Rethinking pixel reconstruction in masked image modeling, 2023

    Yuan Liu, Songyang Zhang, Jiacheng Chen, Kai Chen, and Dahua Lin. Pixmim: Rethinking pixel reconstruction in masked image modeling, 2023. 2

    2023

  31. [39]

    David G. Lowe. Distinctive image features from scale- invariant keypoints. Int. J. Comput. Vision , 60(2):91–110,

  32. [40]

    The three r’s of computer vision: Recognition, recon- struction and reorganization

    Jitendra Malik, Pablo Arbel ´aez, Joao Carreira, Katerina Fragkiadaki, Ross Girshick, Georgia Gkioxari, Saurabh Gupta, Bharath Hariharan, Abhishek Kar, and Shubham Tul- siani. The three r’s of computer vision: Recognition, recon- struction and reorganization. Pattern Recogniti...

    2016

  33. [41]

    A closer look at benchmarking self-supervised pre-training with image classification

    Markus Marks, Manuel Knott, Neehar Kondapaneni, Eli- jah Cole, Thijs Defraeye, Fernando Perez-Cruz, and Pietro Perona. A closer look at benchmarking self-supervised pre-training with image classification. arXiv preprint arXiv:2407.12210, 2024. 2, 3

    2024 arXiv

  34. [42]

    Vision: A computational investigation into the human representation and processing of visual information

    David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010. 1, 3

    2010

  35. [43]

    Self-supervised learning of pretext-invariant representations

    Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. InCVPR, 2020. 1, 4, 2, 8, 9, 10

    2020

  36. [44]

    Indoor segmentation and support inference from rgbd images

    Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012. 4, 5

    2012

  37. [45]

    How useful is self- supervised pretraining for visual tasks? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    Alejandro Newell and Jia Deng. How useful is self- supervised pretraining for visual tasks? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2

    2020

  38. [46]

    Unsupervised learning of visual representations by solving jigsaw puzzles

    Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV,

  39. [47]

    Maxime Oquab, Timoth ´ee Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Rus- sell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Ni...

    2023

  40. [48]

    Context encoders: Feature learning by inpainting

    Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016. 2

    2016

  41. [49]

    Beit v2: Masked image modeling with vector-quantized visual tokenizers, 2022

    Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers, 2022. 2, 3

    2022

  42. [50]

    idisc: Internal discretization for monocular depth estimation

    Luigi Piccinelli, Christos Sakaridis, and Fisher Yu. idisc: Internal discretization for monocular depth estimation. In CVPR, 2023. 4

    2023

  43. [51]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International conference on machine learning...

    2021

  44. [52]

    Vi- sion transformers for dense prediction

    Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. In ICCV, 2021. 4

    2021

  45. [53]

    Vi- sion transformers for dense prediction

    Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12179–12188, 2021. 3

    2021

  46. [54]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Chal- lenge. International Journal of Computer Vision ...

    2015

  47. [55]

    A Realistic Evaluation of Semi-supervised Learning for Fine- grained Classification

    Jong-Chyi Su, Zezhou Cheng, and Subhransu Maji. A Realistic Evaluation of Semi-supervised Learning for Fine- grained Classification. In CVPR, 2021. 2

    2021

  48. [56]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 2, 3, 7

    2024 arXiv

  49. [57]

    Discovering object masks with transformers for unsupervised semantic segmentation

    Wouter Van Gansbeke, Simon Vandenhende, and Luc Van Gool. Discovering object masks with transformers for unsupervised semantic segmentation. arXiv preprint arXiv:2206.06363, 2022. 4 10

    2022 arXiv

  50. [58]

    Benchmarking rep- resentation learning for natural world image collections

    Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking rep- resentation learning for natural world image collections. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 12884–12893, 2021. 2

    2021

  51. [59]

    Dense contrastive learning for self-supervised visual pre-training

    Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021. 4, 6, 2, 8, 9, 10

    2021

  52. [60]

    Freesolo: Learning to segment objects without annotations

    Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. Freesolo: Learning to segment objects without annotations. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 14176–14186, 2022. 4

    2022

  53. [61]

    Yu, and Ishan Misra

    Xudong Wang, Rohit Girdhar, Stella X. Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation, 2023. 4

    2023

  54. [62]

    Masked feature predic- tion for self-supervised visual pre-training

    Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature predic- tion for self-supervised visual pre-training. In CVPR, 2022. 2, 4, 6, 7, 8, 3, 9, 10

    2022

  55. [63]

    Croco: Self-supervised pre-training for 3d vision tasks by cross-view completion

    Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Ro- main Br´egier, Yohann Cabon, Vaibhav Arora, Leonid Ants- feld, Boris Chidlovskii, Gabriela Csurka, and J ´erˆome Re- vaud. Croco: Self-supervised pre-training for 3d vision tasks by cross-view completion. NeurIPS, 35, 2022. 2

    2022

  56. [64]

    CroCo v2: Improved Cross-view Completion Pre- training for Stereo Matching and Optical Flow

    Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Br ´egier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and J´erˆome Re- vaud. CroCo v2: Improved Cross-view Completion Pre- training for Stereo Matching and Optical Flow. In ICCV,

  57. [65]

    CroCo: Self- Supervised Pre-training for 3D Vision Tasks by Cross-View Completion

    Weinzaepfel, Philippe and Leroy, Vincent and Lucas, Thomas and Br ´egier, Romain and Cabon, Yohann and Arora, Vaibhav and Antsfeld, Leonid and Chidlovskii, Boris and Csurka, Gabriela and Revaud J ´erˆome. CroCo: Self- Supervised Pre-training for 3D Vision Tasks by Cross-View C...

    2022

  58. [66]

    Un- supervised feature learning via non-parametric instance dis- crimination

    Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Un- supervised feature learning via non-parametric instance dis- crimination. In CVPR, 2018. 1, 2, 4, 8, 9, 10

    2018

  59. [67]

    Clusterfit: Improving general- ization of visual representations, 2019

    Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadi- yaram, and Dhruv Mahajan. Clusterfit: Improving general- ization of visual representations, 2019. 4, 1, 2, 8, 9, 10

    2019

  60. [68]

    Zamir, Alexander Sax, William B

    Amir R. Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018. 3

    2018

  61. [69]

    Barlow twins: Self-supervised learning via redundancy reduction, 2021

    Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St´ephane Deny. Barlow twins: Self-supervised learning via redundancy reduction, 2021. 2, 4, 6, 3, 8, 9, 10

    2021

  62. [70]

    Colorful image colorization

    Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016. 2

    2016

  63. [71]

    ibot: Image bert pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832 ,

    arXiv

  64. [74]

    The 2D projection error is then defined as: Error2D = ∥p′ − q∥2 where ∥ · ∥2 represents the Euclidean distance in the image plane

    The 3D point P is transformed to the coordinate frame of 4 image 2 using the relative camera pose and projected back onto the image plane of image 2, yielding the pixel location p′. The 2D projection error is then defined as: Error2D = ∥p′ − q∥2 where ∥ · ∥2 represents the Euc...

  65. [2016]

    1, 2, 3, 4, 6, 8, 9, 10

  66. [2021]

    student” network and a “target

    1, 4, 6, 7, 8, 2, 9, 10 11 Probing the Mid-level Vision Capabilities of Self-Supervised Learning Supplementary Material Sec. A provides an overview the self-supervised learn- ing models (Tab. 5) included in our study. Sec. B details the evaluation metrics and presents the quan...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.