{"id":"72dd4cd1-8107-4fed-8433-90bd18a92edb","arxiv_id":"2411.17474","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A controlled benchmark of 22 ImageNet-1K self-supervised models on 8 mid-level vision tasks shows a generally weak and task-dependent correlation with high-level performance.","lead":"This paper benchmarks 22 self-supervised visual models on eight mid-level vision tasks, such as depth estimation and object segmentation. It finds that strong high-level performance does not reliably predict mid-level capability, with some models excelling in one but not the other.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The pooled positive correlation in Fig. 2 may be an artifact of mixing architectures; within the contrastive ResNet-50 family, Fig. 4 shows a negative trend, so the central claim needs within-group regression analysis.","rationale":"The reader's weakest_assumption flagged that the 22 checkpoints differ in training recipes, epochs, and augmentations, so the controlled comparison is imperfect. My concern is more specific: even if training recipes were held fixed, the pooled analysis mixes two very different architectures (ResNet-50 and ViT-B/16), and the data in Fig. 4 already suggest that within a single architecture family (contrastive ResNet-50) the correlation is negative for key mid-level tasks. This makes the pooled positive correlation in Fig. 2 a candidate Simpson's-paradox artifact. The paper does provide a valuable benchmark and detailed per-task tables, and its secondary observations (e.g., MAE underperforms, MaskFeat excels on 3D tasks) are well supported by the raw numbers. However, the central claim about a positive mid-level/high-level correlation is not statistically secure without within-architecture analysis. I do not think this requires rejection: the benchmark data are genuinely useful, and the claim could be repaired by adding the missing subgroup regressions and explicitly qualifying the statement in Sec. 5.1 to acknowledge family-dependent trends. Hence I keep the reader's CONDITIONAL verdict unchanged, but emphasize that the condition should include the within-group correlation test. I agree only partially with the reader's stated weakest assumption: the training-recipe confound is real but secondary; the architecture confound is the load-bearing issue because it can flip the sign of the conclusion.","tokens_in":27327,"tokens_out":4290,"duration_ms":39788,"concrete_test":"Recompute the Fig. 2 regressions separately for the ResNet-50 subset (model IDs 1–17), the ViT-B/16 subset (IDs 18–22), and within the contrastive-learning family, reporting Pearson and Spearman R² with 95% bootstrap confidence intervals for each group and for the pooled set. If the positive slope vanishes or reverses within the ResNet-50 or ViT subsets, the pooled 'positive correlation' claim is an artifact of architecture mixing and must be qualified or withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Sec. 5.1, Fig. 2) is that mid-level vision performance is positively correlated with ImageNet linear probing, but generally more weakly than among high-level tasks. This claim rests on pooled linear regressions across 22 models that differ not only in SSL objective but also in backbone architecture (ResNet-50 vs. ViT-B/16) and provenance (different repositories and training recipes). Since ViT-based models tend to achieve both higher ImageNet LP and higher mid-level performance, pooling can produce a positive correlation even if no positive relationship exists within each architecture family. In fact, Fig. 4 already shows that among seven contrastive ResNet-50 models, higher ImageNet LP is associated with lower depth-estimation and surface-normal accuracy (R² = 0.13 and 0.31), which directly contradicts the unqualified positive statement. The paper does not report within-architecture or within-family correlations, nor confidence intervals or significance tests for the Fig. 2 regressions, and the sample of 22 models is not a random or controlled draw. Therefore the headline claim 'mid-level vision capabilities are positively correlated with high-level vision' is not yet established; the observed pooled trend could be driven by the ResNet/ViT confound. This is the most load-bearing concern because the entire paper's primary conclusion depends on the validity of these pooled correlations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a benchmark suite for probing the mid-level vision capabilities of self-supervised learning (SSL) models and evaluates 22 publicly available ImageNet-1K SSL checkpoints across eight mid-level vision tasks: generic object segmentation, scene/object depth estimation, scene/object surface normal estimation, scene/object geometric correspondence, and mid-level image similarity. The evaluation uses frozen features with DPT decoders for dense tasks and training-free correspondence/NIGHTS protocols for correspondence and similarity. The main empirical claim, stated in Sec. 5.1 and Fig. 2, is that mid-level vision performance is positively correlated with ImageNet linear probing accuracy, but generally more weakly than the correlation among high-level tasks. The paper also identifies imbalanced models (MAE low on mid-level despite strong high-level performance; MaskFeat strong on mid-level despite modest high-level results), analyzes factors such as pretraining objective, architecture, and model capacity, and compares the SSL models with several visual foundation models.","tokens_in":27583,"tokens_out":4216,"duration_ms":40409,"significance":"If the central empirical claim is established, the paper would be a useful and timely benchmark for the SSL community: it covers a broad range of SSL families, uses public checkpoints, provides task-specific protocols, and includes both training-free and frozen-feature probing, which is a practical and reproducible evaluation design. The paper also gives concrete per-model rankings and identifies MaskFeat and MAE as interesting outliers. However, the headline correlation result is not yet supported by the evidence as presented: the pooled regressions in Fig. 2 mix architectures and training recipes, no uncertainty or significance is reported, and Fig. 4 shows a negative within-family trend that contradicts the unqualified positive claim. These issues bear directly on the paper's central contribution, so the manuscript needs a major revision before the conclusions can be accepted.","major_comments":[{"comment":"The central claim of a positive correlation between mid-level and high-level performance is computed by pooling 22 models that differ both in backbone (ResNet-50 vs. ViT-B/16) and in training recipe, since Table 5 lists checkpoints from VISSL, MMSelfSup, the SwAV repository, and an unofficial BYOL repository. Because ViT checkpoints tend to occupy the upper-right region of Fig. 2, a positive pooled regression can arise even if no within-architecture relationship exists. Fig. 4 in fact shows a negative trend among seven contrastive ResNet-50 models on depth (R²=0.13) and surface normals (R²=0.31). Please report within-architecture (ResNet-50-only and ViT-only) regressions for all eight tasks, and if the sign or strength differs, restate the claim with that qualification.","section":"Sec. 5.1, Fig. 2"},{"comment":"No measure of uncertainty accompanies the R² values, and with n=22 models the estimates have wide sampling variability. The reported values range from 0.03 (scene geometry correspondence) to 0.78 (mid-level image similarity), so the blanket statements in the abstract and Sec. 5.1 that the correlation is \"weak\" are an overstatement. Provide confidence intervals or permutation p-values for each regression and summarize the correlation per task rather than as a single global trend.","section":"Sec. 5.1, Fig. 2"},{"comment":"The negative-correlation result for contrastive SSLs is based on only seven ResNet-50 models, with no error bars or significance testing, and it directly undermines the unqualified positive claim in Sec. 5.1. Please add significance testing, include the \"broader analysis across six additional tasks\" promised in the text, and explicitly reconcile the negative within-family trend with the pooled positive correlation in Fig. 2.","section":"Sec. 5.2, Fig. 4"},{"comment":"The evaluation is described as a \"controlled\" comparison, but the selected checkpoints differ not only in SSL objective but also in pretraining epochs, augmentation strategies, projection heads, and implementation provenance (for instance, BYOL comes from an unofficial repository). These recipe differences are confounded with the SSL method, so the attributions in Sec. 5.2 about training objectives, architecture, and capacity should be framed as observational comparisons rather than controlled causal findings. At minimum, add a limitations paragraph that explicitly acknowledges this confound.","section":"Sec. 4, Table 5"}],"minor_comments":[{"comment":"In the MAE L/16 row, the entry \"26.6 0\" should presumably be \"26.60\".","section":"Table 3"},{"comment":"The axis labels contain the corrupted string \"Image/glyph1197et LP (%)\", which should be fixed to \"ImageNet LP (%)\".","section":"Fig. 2, Fig. 4"},{"comment":"The paragraphs titled \"Network architecture\" and \"Network capability\" are nearly identical and should be merged into one discussion.","section":"Sec. 5.2"},{"comment":"MAE is cited inconsistently as reference [31] in Table 7 and as reference [32] in Table 8, and references [31] and [32] appear to be duplicate entries for the same work.","section":"References, Tables 7-8"},{"comment":"The final sentence of the pre-text tasks paragraph, \"it re-validate our claim that ImageNet (IN1k) probing results improve, we generally observe...\", is grammatically broken and appears to restate Sec. 5.1; please rewrite.","section":"Sec. 5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of a computer vision venue. The main risk is that the headline correlation is an artifact of pooling architectures; the authors' own Fig. 4 shows a negative within-family trend. If only pooled correlations can be provided, the central claim should be substantially softened; if within-architecture regressions and significance tests are added, the paper could be a strong benchmark contribution. The \"controlled\" framing should also be tempered given the heterogeneity of the public checkpoints."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful benchmark paper with a real new result, but its headline correlation claim is not yet established. The pooled regressions in Fig. 2 mix ResNet-50 and ViT-B/16 models, so architecture is a confound; the paper's own Fig. 4 shows a negative trend within seven contrastive RN-50 models, which directly contradicts the unqualified positive statement in Sec. 5.1.\n\nWhat is actually new: a controlled comparison of 22 ImageNet-1K SSL checkpoints across 8 mid-level tasks, with freeze-backbone probing and a consistent decoder. The MAE-vs-MaskFeat imbalance is a genuine, non-obvious empirical finding, and the architecture/scale comparisons in Table 3 are useful. I appreciate the effort to restrict to IN-1K checkpoints to keep training data constant, even though training recipes still differ across repositories.\n\nThe soft spots are the ones the reader flagged, plus one more load-bearing one. The R² values range from 0.03 to 0.78, so the blanket \"weakly correlated\" summary is an overstatement; the paper itself says some tasks like object segmentation and mid-level similarity are highly correlated. The stress test is right: pooling across architectures can create a positive association even if no relationship exists within family. Fig. 4 shows exactly that for contrastive RN-50s on depth and normals. The paper needs within-family regressions and ideally significance tests or confidence intervals; n=22 is small and not a random draw. Also the \"negative correlation\" for contrastive methods in Fig. 4 rests on seven points with no error bars, so that is suggestive, not established. Minor: the duplicated architecture/scale paragraphs in Sec. 5.2, the BEiT-v2 description in the supplementary that does not match the evaluated model list, and the Table 3 typo (\"26.6 0\").\n\nThe data tables are there, the protocol is standard and reproducible in principle, but code and hyperparameters are not yet released, so I could not fully verify the probing setup. That is a revision requirement, not a fatal flaw.\n\nBottom line: worth serious review. The paper's practical value—showing that SSL evaluation should include mid-level tasks and that MAE/MaskFeat are outliers—stands even if the correlation claim needs tightening. Send it to referees, but the referee letter should require within-family correlations and a more careful wording of the central claim.","headline":"Useful controlled benchmark with a real MAE/MaskFeat finding, but the headline pooled correlation is confounded by architecture and needs within-family analysis.","tokens_in":28075,"tokens_out":1892,"would_cite":true,"duration_ms":16421,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"High ImageNet accuracy doesn't guarantee mid-level vision","keywords":["self-supervised learning","mid-level vision","benchmark","depth estimation","surface normals","geometric correspondence","ImageNet linear probing","vision transformers"],"falsifier":"Retrain a representative set of SSL objectives (contrastive, self-distillation, masked reconstruction, HOG prediction) under one unified recipe with identical epochs, augmentations, batch size, and projection head on ImageNet-1K, then rerun the eight mid-level probes. If the $R^2$ between mid-level performance and ImageNet linear probing rises to the level seen among high-level tasks (for example above 0.6 for scene surface normals), or if MAE no longer underperforms on mid-level tasks, the paper's central claim about SSL objectives would be falsified.","tokens_in":27117,"feed_emoji":"👁️","tokens_out":4648,"duration_ms":41850,"temperature":0.7,"pith_summary":"The paper asks whether a self-supervised visual model that excels at high-level recognition (ImageNet classification) also understands mid-level properties: where objects end, how far surfaces are, and which pixels match across views. The answer is \"yes, but only loosely\": across 22 ImageNet-1K SSL checkpoints and 8 mid-level tasks, mid-level performance does rise with ImageNet linear probing accuracy, yet the $R^2$ correlation is much weaker than the correlation among high-level tasks themselves, and for scene surface normals and scene geometric correspondence it is near zero. The paper also finds that generative SSL objectives, especially iBOT and MaskFeat, lead on mid-level tasks while MAE trails despite its strong ImageNet accuracy, and that ViT backbones and larger models help. If this holds, ImageNet accuracy is an incomplete yardstick for representation quality, and SSL benchmarks should include mid-level probes.","feed_headline":"High ImageNet accuracy doesn't guarantee mid-level vision","feed_subtitle":"A 22-model benchmark shows depth, surface and geometry skills track recognition only loosely","key_machinery":"The probing machinery is a fixed set of evaluation protocols applied to frozen representations: a DPT decoder trained on top of frozen features for dense tasks (generic object segmentation, depth, surface normals), and training-free feature matching for correspondence and similarity tasks using cosine distances between CLS or global tokens and dense feature matching. The organizing quantity is the $R^2$ coefficient of determination between each mid-level task's score and ImageNet linear probing accuracy across 22 models, which quantifies how much mid-level capability is explained by high-level recognition. The contrast that carries the argument is the paired comparison between these $R^2$ values and the $R^2$ among high-level tasks, plus the case studies of MAE and MaskFeat as outliers.","core_discovery":"The paper's central empirical claim is that the mid-level vision capabilities of self-supervised models are positively correlated with their high-level vision performance, but this correlation is generally weaker than the correlation among high-level vision tasks. Concretely, regressing each mid-level task against ImageNet linear probing over 22 pretrained checkpoints yields $R^2$ values from 0.78 for mid-level image similarity and 0.70 for generic object segmentation down to 0.23 for scene depth, 0.08 for scene surface normals, and 0.03 for scene geometric correspondence, whereas high-level tasks correlate much more strongly with ImageNet probing. The paper further establishes that the pattern is not uniform: MaskFeat, a generative model trained to predict HOG features, is among the best mid-level models despite modest ImageNet accuracy, while MAE, strong on ImageNet, is among the worst mid-level models; iBOT and DINO lead in both; and within-method comparisons show ViT outperforms ResNet and larger backbones help.","pith_inferences":["A testable extension not in the paper: if the weak correlation is causal rather than an artifact of training-recipe differences, adding a HOG-prediction term to a contrastive objective should raise its scene surface-normal performance without hurting ImageNet accuracy.","Scene geometric correspondence has near-zero correlation with both recognition and other mid-level tasks, so it may be a better diagnostic for 3D-aware pretraining (such as cross-view completion) than for generic SSL, a possibility the paper leaves implicit.","The paper's controlled comparison is only as controlled as the publicly released checkpoints allow; retraining the same objectives under a single unified recipe could shrink or enlarge the observed gaps between objectives, and the ranking of methods might change.","The brief comparison with visual foundation models suggests scaling data and diversity, as in DINO-v2, may be a complementary route to mid-level competence; whether the mid-level/high-level correlation strengthens with pretraining scale is an open question."],"forward_implications":["ImageNet linear probing is not a reliable proxy for mid-level vision; a model's rank can flip between high-level and mid-level benchmarks.","Generative objectives with mid-level-friendly targets, such as MaskFeat's HOG feature prediction, can produce strong mid-level representations without high recognition accuracy, suggesting the choice of training target matters for SSL design.","MAE's weak mid-level performance despite strong ImageNet accuracy shows that masked pixel reconstruction alone does not build geometric understanding.","Architecture decisions matter: ViT-B/16 beats ResNet-50 for the same SSL objective, and ViT-L/16 beats ViT-B/16, so architecture and capacity confound SSL comparisons.","Benchmarking SSL only on high-level tasks gives an incomplete picture of representation quality; mid-level probes should be part of standard evaluation."],"supporting_citations":[{"why":"Probe3D supplies the probing protocol, DPT-based dense evaluation, and correspondence metrics that this paper adapts to SSL checkpoints.","marker":"[17]"},{"why":"Ericsson et al. provide prior evidence of weak transfer correlation between ImageNet classification and dense/other downstream tasks, which this paper extends to mid-level vision.","marker":"[18]"},{"why":"Marks et al. guide the selection of SSL methods and evaluation protocols for controlled comparisons of self-supervised pretraining.","marker":"[41]"},{"why":"DPT is the dense decoder used to probe frozen features for segmentation, depth, and surface normal estimation.","marker":"[53]"},{"why":"AdaBins provides the binned depth-estimation formulation used in the depth probing protocol.","marker":"[21]"},{"why":"The NIGHTS dataset and DreamSim triplet protocol is the source of the mid-level image similarity benchmark.","marker":"[23]"},{"why":"MaskFeat anchors the claim that the pretraining target (HOG features) can yield strong mid-level performance despite modest high-level accuracy.","marker":"[62]"},{"why":"MAE anchors the opposite imbalance: strong ImageNet accuracy but weak mid-level performance, supporting the central decorrelation claim.","marker":"[32]"},{"why":"iBOT is the top all-round SSL model across mid-level and high-level tasks, used to show that some methods excel in both.","marker":"[71]"}],"fun_headline_variants":["SSL mid-level skills don't track ImageNet rank","ImageNet accuracy ≠ mid-level vision in SSL","Weak link between SSL recognition and geometry","Mid-level vision: a blind spot for SSL benchmarks","High-level SSL success doesn't guarantee mid-level"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The controlled comparison assumes the 22 publicly released ImageNet-1K checkpoints differ only in SSL objective and architecture, when in practice they also differ in training recipe details such as epochs, augmentations, and projection heads.","fun_headline_variants_meta":{"raw":{"variants":["SSL mid-level skills don't track ImageNet rank","ImageNet accuracy ≠ mid-level vision in SSL","Weak link between SSL recognition and geometry","Mid-level vision: a blind spot for SSL benchmarks","High-level SSL success doesn't guarantee mid-level"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000142,"raw_usage":{"total_tokens":1178,"prompt_tokens":966,"completion_tokens":212,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":141}},"tokens_in":582,"tokens_out":212,"duration_ms":2834,"temperature":1.0,"reasoning_tokens":141,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:49:02.640424+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain a representative set of SSL objectives (contrastive, self-distillation, masked reconstruction, HOG prediction) under one unified recipe with identical epochs, augmentations, batch size, and projection head on ImageNet-1K, then rerun the eight mid-level probes. If the $R^2$ between mid-level performance and ImageNet linear probing rises to the level seen among high-level tasks (for example above 0.6 for scene surface normals), or if MAE no longer underperforms on mid-level tasks, the paper's central claim about SSL objectives would be falsified.","supporting_citations":[{"cited_title":"Hospedales","cited_arxiv_id":null,"evidence_quote":"Ericsson et al. provide prior evidence of weak transfer correlation between ImageNet classification and dense/other downstream tasks, which this paper extends to mid-level vision."},{"cited_title":"A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classification","cited_arxiv_id":"2407.12210","evidence_quote":"Marks et al. guide the selection of SSL methods and evaluation protocols for controlled comparisons of self-supervised pretraining."},{"cited_title":"Adabins: Depth estimation using adaptive bins","cited_arxiv_id":null,"evidence_quote":"AdaBins provides the binned depth-estimation formulation used in the depth probing protocol."},{"cited_title":"Dream- sim: Learning new dimensions of human visual similarity using synthetic data, 2023","cited_arxiv_id":null,"evidence_quote":"The NIGHTS dataset and DreamSim triplet protocol is the source of the mid-level image similarity benchmark."},{"cited_title":"Masked feature predic- tion for self-supervised visual pre-training","cited_arxiv_id":null,"evidence_quote":"MaskFeat anchors the claim that the pretraining target (HOG features) can yield strong mid-level performance despite modest high-level accuracy."},{"cited_title":"Masked autoencoders are scalable vision learners","cited_arxiv_id":null,"evidence_quote":"MAE anchors the opposite imbalance: strong ImageNet accuracy but weak mid-level performance, supporting the central decorrelation claim."}],"review_version":1}