{"id":"4aecf4b7-e079-49e8-8f95-227e4c697504","arxiv_id":"2501.18463","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"New benchmarks based on ImageNet and WILDS show that no CLIP-based OOD detection method is consistently best once semantic and covariate shifts are combined.","lead":"This paper creates three harder test suites for detecting when vision-language models encounter unfamiliar images, using close ImageNet categories and real-world camera and satellite data. It shows that current CLIP-based methods change rank across the suites and none is the best everywhere.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ImageNet-X's WordNet split cannot deliver claimed small semantic shift for all 500 OOD classes: 1000 classes into 558 parents forces at least 116 singleton classes, and section 2.1 never specifies their assignment, so a large fraction of OOD labels may have no same-parent ID counterpart.","rationale":"The paper's central contribution is a set of benchmarks that are supposed to isolate small semantic shifts and covariate shifts; the headline empirical claim (no CLIP-based method wins across all benchmarks) is only meaningful if the ImageNet-X split actually delivers uniformly small semantic shifts. My stress-test focuses on the construction in section 2.1. The reader's weakest assumption identified the WordNet parent proxy without validation; I find a sharper, internal-consistency problem: with 1000 classes in 558 parent groups, at least 116 parent groups are singletons, so the described 'split labels within each parent' procedure cannot cover all classes. The paper does not state how singleton classes are handled, and any handling leaves some OOD classes without a sibling in ID. Because AUROC is averaged over all OOD classes, these singleton-derived classes can dominate the score and change method rankings for reasons unrelated to fine-grained semantic shift. This does not mean the empirical findings are wrong; the released code and splits could reveal that singleton classes are few or that rankings are robust. But the burden is on the benchmark to demonstrate this, and the current text does not. I therefore regard the conditional verdict as appropriate, pending the concrete check described above. I agree with the reader's identification of the WordNet assumption as the weakest point, and my objection is a sharpened version of that concern.","tokens_in":2,"tokens_out":10576,"duration_ms":171292,"concrete_test":"Download the released ImageNet-X split and the WordNet parent mapping from the official ImageNet site. For each of the 500 OOD classes, record whether its immediate parent contains at least one ID class, and report the number and fraction of OOD classes with no same-parent ID. Then recompute the ImageNet-X AUROC column of Table 2 restricted to the OOD classes that do share a parent with an ID class. If the fraction without a same-parent ID is above a small threshold (e.g., 5%) or if the restricted ranking differs from the full ranking for any method pair, the benchmark does not control semantic shift as claimed and the ranking conclusions are confounded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In section 2.1 the benchmark's core difficulty control rests on splitting labels within immediate WordNet parent classes so that ID and OOD are 'particularly closely related.' But the stated grouping of 1000 ImageNet classes into 558 parent classes makes this impossible for all classes. By pigeonhole, at least 116 parent classes contain exactly one ImageNet class (the maximum number of multi-child parents is 442, since 2x+y=1000 and x+y=558). A singleton class cannot be split into two halves, and the paper gives no rule for assigning these classes to ID or OOD. Any assignment places those classes in a partition without a same-parent counterpart, so for those classes the semantic shift is parent-level, not sibling-level, and may be much larger. If the 116 singletons are assigned to OOD, 23% of OOD classes violate the design premise; even a balanced assignment leaves about 58 OOD classes (12%) without an ID sibling. Since AUROC aggregates over all 500 OOD classes, the aggregate can be inflated by easy, large-shift classes, and the observed method rankings in Table 2 may reflect which methods are good at large shifts rather than at the intended fine-grained shifts. The paper neither reports the singleton assignment nor validates the semantic closeness of the resulting split, so the central comparison underpinning F1 is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes three OOD detection benchmarks built around small semantic shifts and covariate shifts: ImageNet-X (ImageNet-1k labels split by WordNet parent classes), ImageNet-FS-X (adding ImageNet-V2/R/C as covariate-shifted ID), and Wilds-FS-X (using iWildCam and FMoW from WILDS). It evaluates ten CLIP-based zero-shot and few-shot OOD detection methods and reports AUROC and classification accuracy. The central empirical claims are that method rankings change across benchmarks, no method consistently wins, covariate shifts degrade performance, few-shot methods are more sensitive to covariate shifts, and the WILDS-based benchmarks remain largely unsolved.","tokens_in":10874,"tokens_out":3143,"duration_ms":32294,"significance":"If the benchmarks are valid, the paper makes a useful contribution: it provides public code, evaluates a broad set of representative CLIP-based methods, and challenges the impression that existing ImageNet-based OOD benchmarks are saturated. The finding that rankings differ between conventional hard-OOD and the proposed ImageNet-X is potentially informative for the community. The strength is the systematic separation of semantic and covariate shifts and the inclusion of real-world WILDS data. However, the benchmark construction relies on an unvalidated WordNet split whose internal consistency is not demonstrated, and the reported numbers lack error bars despite the use of random splits; these issues currently limit the strength of the empirical conclusions.","major_comments":[{"comment":"The central ImageNet-X construction is not internally consistent. The paper states that the 1000 ImageNet classes are grouped into 558 WordNet parent classes and that the labels within each parent class are split into two halves. By pigeonhole, at least 116 parent classes contain exactly one ImageNet class (1000 - 558 = 442 multi-child parents at most, leaving at least 116 singletons). A singleton class cannot be split into ID and OOD counterparts, yet the paper never specifies how these classes are assigned. If they are assigned to OOD, then 116 of the 500 OOD classes (23%) have no same-parent ID sibling; even a balanced assignment leaves about 58 OOD classes (12%) without one. For those classes the semantic shift is parent-level, not sibling-level, and may be much larger. Since AUROC in Table 2 aggregates over all 500 OOD classes, the aggregate can be inflated by easy, large-shift classes, so the reported rankings may reflect performance on large shifts rather than on the intended fine-grained shifts. The authors must report the exact singleton assignment and verify that the resulting ID/OOD split provides comparable semantic difficulty across classes.","section":"Section 2.1, Table 2"},{"comment":"The paper assumes that the immediate WordNet parent class is a valid and uniform proxy for small semantic shift, but it provides no quantitative validation. Some parent classes contain very close species (e.g., dog breeds), while others may contain more heterogeneous members; the internal similarity of parent groups is not measured. Without evaluating inter-class distances, human similarity judgments, or at least per-parent AUROC breakdowns, the claim that ImageNet-X guarantees small and controlled semantic shifts is unsupported. This assumption is load-bearing because it is the basis for the main comparison in Table 2.","section":"Section 2.1"},{"comment":"The WILDS-based results lack error bars and the random split is not reproducible. iWildCam uses 91 randomly selected animal classes as ID and FMoW uses 31 randomly selected classes as ID, but no random seed is reported and the split appears to be a single draw. The few-shot methods are averaged over 3 seeds, but this does not capture the variance of the random class split. In Table 2, several FMoW AUROC values differ by only 1–2 points (e.g., 55.45 vs. 55.89), and the iWildCam ranking of zero-shot methods spans a range of about 16 points. The claim that 'no single winner emerges' and the reported Spearman correlation of 0.90 are not robust without confidence intervals or repeated benchmark splits.","section":"Section 3.5, Table 2"},{"comment":"ImageNet-FS-X does not apply the ImageNet-X label split uniformly. Table 1 shows that ImageNet-R has only 495 ID and 492 OOD classes, and ImageNet-C has only 315 ID and 313 OOD classes, rather than the nominal 500/500. The paper does not explain how the label splitting method from ImageNet-X is adapted when the variant dataset does not contain all 500 ID labels or all 500 OOD labels. This partial coverage changes the composition of the covariate-shifted ID set and the OOD set, and may affect the comparison between ImageNet-X and ImageNet-FS-X in findings F2 and F3. The authors should specify which classes are dropped and how the AUROC denominators are computed.","section":"Section 2.2, Table 1"}],"minor_comments":[{"comment":"There is a missing space in 'code ishttps://github.com/hoshi23/OOD-X-Benchmarks'; it should read 'code is https://github.com/hoshi23/OOD-X-Benchmarks'.","section":"Abstract"},{"comment":"The table is visually dense and does not report standard deviations or confidence intervals for the AUROC values, even though few-shot methods are averaged over 3 seeds. Adding error bars or variance notation would strengthen the interpretation of ranking changes.","section":"Table 2"},{"comment":"The Spearman rank-order correlation of 0.90 is reported without a confidence interval or significance test; with only 10 methods, the estimate is unstable and should be interpreted cautiously.","section":"Section 3.4"},{"comment":"Figure A is described as showing 'one representative sample' from ID and OOD, but the figure contains two images per method; the caption should be clarified to indicate that two samples are shown.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is a benchmark paper, so the main risk is not in the implementation of the evaluated methods but in the validity of the proposed benchmark splits. The WordNet singleton issue in Section 2.1 is a concrete, correctable technical flaw: the authors need to disclose the assignment of singleton classes and validate that the split actually delivers small, comparable semantic shifts. If the authors provide that analysis and add error bars or repeated splits for the WILDS benchmarks, the paper could be acceptable. There is no indication of circularity or deliberate misrepresentation; the issues are matters of missing validation and missing reproducibility details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful benchmark paper, and the central empirical finding - no CLIP-based OOD method wins across all three new benchmarks - is worth taking seriously. But the construction of ImageNet-X has a real gap that the paper never addresses: with 1000 classes grouped into 558 immediate WordNet parents, at least 116 parents contain a single class. A singleton can't be split into ID/OOD halves. The paper gives no rule for where those singletons go, and if they land in OOD, 23% of OOD classes have no same-parent ID sibling. Their semantic shift is parent-level, not sibling-level, which means the aggregate AUROC mixes intended fine-grained shifts with larger shifts. That could change the method ranking you'd measure if the split were properly controlled. This is fixable - report the assignment, or restrict the benchmark to parents with at least two children and validate closeness - but as written, the headline claim about ImageNet-X's semantic homogeneity is not supported.\n\nWhat's new: the three benchmarks assemble existing data with a WordNet-based 500/500 split, a full-spectrum variant, and two WILDS real-world settings. The evaluation is broad, covers representative zero-shot and few-shot CLIP methods, and ships code. The finding that rankings change across benchmarks and that few-shot methods are more covariate-sensitive is a useful data point for the community.\n\nOther soft spots, smaller: no error bars despite 3-seed averages, the WILDS random split seed isn't reported, and the name ImageNet-X collides with an existing dataset - that will cause confusion. Also, section 2.1's claim that parent classes are 'particularly closely related' isn't validated; not all sibling sets are equally confusable.\n\nWho's it for: people working on OOD detection evaluation with VLMs. It deserves a serious referee, but the reviewer should ask for the singleton handling, split seeds, error bars, and a naming disambiguation. If those are addressed, the benchmarks could be standard.","headline":"Useful benchmark suite for CLIP OOD evaluation, but the ImageNet-X split has an unaddressed singleton-class problem that undercuts the semantic-shift claim.","tokens_in":11430,"tokens_out":3085,"would_cite":true,"duration_ms":27009,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that CLIP-based OOD detection methods have no stable winner once benchmarks include small semantic shifts, covariate shifts, and real-world conditions, so conventional rankings are unreliable.","keywords":["out-of-distribution detection","vision-language models","CLIP","benchmark construction","semantic shift","covariate shift","full-spectrum OOD","WILDS"],"falsifier":"Compute the mean CLIP embedding distance between ID and OOD labels within each WordNet parent group in ImageNet-X; if these distances vary widely across groups, the benchmark's difficulty is not uniform, so the observed method rankings may reflect label-group composition rather than detector capability.","tokens_in":10418,"feed_emoji":"📊","tokens_out":6047,"duration_ms":47080,"temperature":0.7,"pith_summary":"This paper argues that conventional out-of-distribution (OOD) detection benchmarks for CLIP-based models are saturated, because the distribution shift between in-distribution and OOD data is too large to tell methods apart. To address this, the authors construct three benchmarks that progressively add controlled semantic shifts, covariate shifts, and real-world conditions. Across these benchmarks, ten representative CLIP-based methods change rank and none is best everywhere; covariate shifts alone can cut AUROC by more than twenty points. The paper concludes that method rankings from conventional benchmarks are unreliable and that evaluations must include multiple, harder shift conditions.","feed_headline":"No CLIP OOD detector wins across three real-world benchmarks","feed_subtitle":"Covariate shifts slash AUROC and method rankings flip, so current leaderboards may mislead.","key_machinery":"The core mechanism is the benchmark construction protocol rather than a single algorithm. ImageNet-X uses the WordNet hierarchy's immediate parent classes to list the 1000 ImageNet labels into 558 groups, then splits each group's labels into two halves and rebalances so that ID and OOD contain 500 labels each; this forces semantic shifts to be small and roughly comparable across classes. ImageNet-FS-X appends covariate-shifted ID data of the same labels from ImageNet-V2, ImageNet-R, and ImageNet-C, aligning the covariate distribution of OOD with ID. Wilds-FS-X applies the same split logic to WILDS' iWildCam and FMoW datasets, where covariate shifts arise naturally from camera locations and capture years. Together the three benchmarks separate semantic-shift difficulty from covariate-shift robustness and give a common protocol for measuring both.","core_discovery":"The central discovery is that CLIP-based OOD detection methods do not have a stable ranking once the evaluation moves from conventional large-shift benchmarks to smaller semantic gaps and added covariate shifts. On ImageNet-X, which splits ImageNet-1k's 1,000 classes into 500 ID and 500 OOD classes using WordNet parent classes to keep semantics close, the few-shot method NegPrompt overtakes the zero-shot CLIPN-A that wins on conventional hard-OOD sets. Adding covariate-shifted versions of the same classes (ImageNet-V2, -R, -C) in ImageNet-FS-X drops performance for every method, with the largest falls for few-shot methods, yet the overall ranking across methods stays similar (Spearman 0.90). On real-world Wilds-FS-X, built from iWildCam and FMoW, all methods perform poorly, and better classification accuracy from few-shot learning does not translate into better OOD detection.","pith_inferences":["The WordNet parent-class split may not equalize semantic difficulty across groups: some parent groups contain visually near-identical species while others contain more distinct objects, so ImageNet-X's difficulty could vary by label group and confound cross-method comparisons.","The high Spearman correlation (0.90) between ImageNet-X and ImageNet-FS-X rankings suggests semantic shift dominates method ordering when covariate shift is moderate; a stress test that increases covariate intensity (e.g., heavier corruptions) might break that stability.","Because few-shot methods overfit the training covariate distribution, a method's apparent advantage under one domain may invert under another; evaluating with per-source AUROC, as in the appendix, should become a standard diagnostic.","The result that higher classification accuracy does not imply better OOD detection suggests that OOD detection needs its own training objective or metric, not just improved feature extractors."],"forward_implications":["Method rankings from conventional large-shift OOD benchmarks should not be read as a guide to real-world performance, since rankings change on ImageNet-X and Wilds-FS-X.","Covariate shift degrades CLIP-based OOD detection: every method loses AUROC from ImageNet-X to ImageNet-FS-X, with the largest drops for few-shot prompt learners, so covariate robustness must be reported separately.","Few-shot prompt learning can bias a detector toward the covariate distribution of its training images, as seen in the differing behavior on ImageNet-V2 versus ImageNet-R and on WILDS Test versus Test-ID.","On real-world data (iWildCam, FMoW), current CLIP-based detectors are near baseline for OOD, and improving classification accuracy does not by itself improve OOD detection.","Benchmarking OOD detection requires paired evaluations under semantic and covariate shifts, as the FS-X design provides."],"supporting_citations":[{"why":"Provides ImageNet-1k as the source data; its 1,000 classes are split by WordNet parent to build ImageNet-X and ImageNet-FS-X ID/OOD sets.","marker":"[7]"},{"why":"Supplies the WordNet hierarchy whose immediate parent classes define semantically close label pairs for the ImageNet-X split.","marker":"[25]"},{"why":"Defines the full-spectrum OOD setting combining semantic and covariate shifts that ImageNet-FS-X and Wilds-FS-X instantiate.","marker":"[16]"},{"why":"Provides the WILDS datasets (iWildCam, FMoW) whose natural covariate shifts (camera location, year) back Wilds-FS-X.","marker":"[17]"},{"why":"Establishes that OOD detection is more sensitive to covariate shifts than semantic shifts, motivating the benchmark design.","marker":"[23]"},{"why":"SSBHard is a conventional hard-OOD benchmark whose performance is compared against ImageNet-X.","marker":"[20]"},{"why":"NINCO is a conventional hard-OOD benchmark used alongside SSBHard as a comparison point.","marker":"[21]"},{"why":"MCM is a representative zero-shot CLIP-based OOD method evaluated across all benchmarks.","marker":"[5]"},{"why":"LoCoOp is a representative few-shot CLIP-based method whose ranking shifts on ImageNet-X illustrate the benchmark's effect.","marker":"[6]"}],"fun_headline_variants":["No CLIP OOD detector tops all three new benchmark suites","Covariate shifts drop AUROC and flip method rankings in OOD","CLIP OOD performance varies wildly: no stable winner across tests","Real-world OOD benchmarks show leaderboards can mislead"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The split assumes that grouping labels by their immediate WordNet parent class makes semantic difficulty roughly equal for every group; if some parent groups contain much closer classes than others, benchmark difficulty is not controlled across classes and method comparisons become confounded.","fun_headline_variants_meta":{"raw":{"variants":["No CLIP OOD detector tops all three new benchmark suites","Covariate shifts drop AUROC and flip method rankings in OOD","CLIP OOD performance varies wildly: no stable winner across tests","Real-world OOD benchmarks show leaderboards can mislead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1331,"prompt_tokens":925,"completion_tokens":406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":333}},"tokens_in":541,"tokens_out":406,"duration_ms":4238,"temperature":1.0,"reasoning_tokens":333,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T23:24:13.689992+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the mean CLIP embedding distance between ID and OOD labels within each WordNet parent group in ImageNet-X; if these distances vary widely across groups, the benchmark's difficulty is not uniform, so the observed method rankings may reflect label-group composition rather than detector capability.","supporting_citations":[{"cited_title":"Imagenet large scale visual recognition challenge,","cited_arxiv_id":null,"evidence_quote":"Provides ImageNet-1k as the source data; its 1,000 classes are split by WordNet parent to build ImageNet-X and ImageNet-FS-X ID/OOD sets."},{"cited_title":"Wordnet: An electronic lexical database.,","cited_arxiv_id":null,"evidence_quote":"Supplies the WordNet hierarchy whose immediate parent classes define semantically close label pairs for the ImageNet-X split."},{"cited_title":"Full-spectrum out-of-distribution detection,","cited_arxiv_id":null,"evidence_quote":"Defines the full-spectrum OOD setting combining semantic and covariate shifts that ImageNet-FS-X and Wilds-FS-X instantiate."},{"cited_title":"WILDS: A benchmark of in-the-wild distribution shifts,","cited_arxiv_id":null,"evidence_quote":"Provides the WILDS datasets (iWildCam, FMoW) whose natural covariate shifts (camera location, year) back Wilds-FS-X."},{"cited_title":"Imagenet-ood: Deciphering modern out-of-distribution detec- tion algorithms,","cited_arxiv_id":null,"evidence_quote":"Establishes that OOD detection is more sensitive to covariate shifts than semantic shifts, motivating the benchmark design."},{"cited_title":"Open-set recog- nition: A good closed-set classifier is all you need?,","cited_arxiv_id":null,"evidence_quote":"SSBHard is a conventional hard-OOD benchmark whose performance is compared against ImageNet-X."},{"cited_title":"Delving into out-of-distribution detection with vision-language representations,","cited_arxiv_id":null,"evidence_quote":"MCM is a representative zero-shot CLIP-based OOD method evaluated across all benchmarks."},{"cited_title":"Lo- coop: Few-shot out-of-distribution detection via prompt learn- ing,","cited_arxiv_id":null,"evidence_quote":"LoCoOp is a representative few-shot CLIP-based method whose ranking shifts on ImageNet-X illustrate the benchmark's effect."}],"review_version":1}