REVIEW 4 major objections 4 minor 34 references
A Benchmark and Evaluation for Real-World Out-of-Distribution Detection Using Vision-Language Models
T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read The paper claims that CLIP-based OOD detection methods have no stable winner once benchmarks include small semantic shifts, covariate shifts, and real-world conditions, so conventional rankings are unreliable.
desk verdict Useful benchmark suite for CLIP OOD evaluation, but the ImageNet-X split has an unaddressed singleton-class problem that undercuts the semantic-shift claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The core mechanism is the benchmark construction protocol rather than a single algorithm. ImageNet-X uses the WordNet hierarchy's immediate parent classes to list the 1000 ImageNet labels into 558 groups, then splits each group's labels into two halves and rebalances so that ID and OOD contain 500 labels each; this forces semantic shifts to be small and roughly comparable across classes. ImageNet-FS-X appends covariate-shifted ID data of the same labels from ImageNet-V2, ImageNet-R, and ImageNet-C, aligning the covariate distribution of OOD with ID. Wilds-FS-X applies the same split logic to WILDS' iWildCam and FMoW datasets, where covariate shifts arise naturally from camera locations and capture years. Together the three benchmarks separate semantic-shift difficulty from covariate-shift robustness and give a common protocol for measuring both.
What would settle it
Compute the mean CLIP embedding distance between ID and OOD labels within each WordNet parent group in ImageNet-X; if these distances vary widely across groups, the benchmark's difficulty is not uniform, so the observed method rankings may reflect label-group composition rather than detector capability.
Extended reading notes
Core claim
The central discovery is that CLIP-based OOD detection methods do not have a stable ranking once the evaluation moves from conventional large-shift benchmarks to smaller semantic gaps and added covariate shifts. On ImageNet-X, which splits ImageNet-1k's 1,000 classes into 500 ID and 500 OOD classes using WordNet parent classes to keep semantics close, the few-shot method NegPrompt overtakes the zero-shot CLIPN-A that wins on conventional hard-OOD sets. Adding covariate-shifted versions of the same classes (ImageNet-V2, -R, -C) in ImageNet-FS-X drops performance for every method, with the largest falls for few-shot methods, yet the overall ranking across methods stays similar (Spearman 0.90). On real-world Wilds-FS-X, built from iWildCam and FMoW, all methods perform poorly, and better classification accuracy from few-shot learning does not translate into better OOD detection.
Load-bearing premise
The split assumes that grouping labels by their immediate WordNet parent class makes semantic difficulty roughly equal for every group; if some parent groups contain much closer classes than others, benchmark difficulty is not controlled across classes and method comparisons become confounded.
Editorial extensions
If this is right
- Method rankings from conventional large-shift OOD benchmarks should not be read as a guide to real-world performance, since rankings change on ImageNet-X and Wilds-FS-X.
- Covariate shift degrades CLIP-based OOD detection: every method loses AUROC from ImageNet-X to ImageNet-FS-X, with the largest drops for few-shot prompt learners, so covariate robustness must be reported separately.
- Few-shot prompt learning can bias a detector toward the covariate distribution of its training images, as seen in the differing behavior on ImageNet-V2 versus ImageNet-R and on WILDS Test versus Test-ID.
- On real-world data (iWildCam, FMoW), current CLIP-based detectors are near baseline for OOD, and improving classification accuracy does not by itself improve OOD detection.
- Benchmarking OOD detection requires paired evaluations under semantic and covariate shifts, as the FS-X design provides.
Reading between the lines
- The WordNet parent-class split may not equalize semantic difficulty across groups: some parent groups contain visually near-identical species while others contain more distinct objects, so ImageNet-X's difficulty could vary by label group and confound cross-method comparisons.
- The high Spearman correlation (0.90) between ImageNet-X and ImageNet-FS-X rankings suggests semantic shift dominates method ordering when covariate shift is moderate; a stress test that increases covariate intensity (e.g., heavier corruptions) might break that stability.
- Because few-shot methods overfit the training covariate distribution, a method's apparent advantage under one domain may invert under another; evaluating with per-source AUROC, as in the appendix, should become a standard diagnostic.
- The result that higher classification accuracy does not imply better OOD detection suggests that OOD detection needs its own training objective or metric, not just improved feature extractors.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes three OOD detection benchmarks built around small semantic shifts and covariate shifts: ImageNet-X (ImageNet-1k labels split by WordNet parent classes), ImageNet-FS-X (adding ImageNet-V2/R/C as covariate-shifted ID), and Wilds-FS-X (using iWildCam and FMoW from WILDS). It evaluates ten CLIP-based zero-shot and few-shot OOD detection methods and reports AUROC and classification accuracy. The central empirical claims are that method rankings change across benchmarks, no method consistently wins, covariate shifts degrade performance, few-shot methods are more sensitive to covariate shifts, and the WILDS-based benchmarks remain largely unsolved.
Significance. If the benchmarks are valid, the paper makes a useful contribution: it provides public code, evaluates a broad set of representative CLIP-based methods, and challenges the impression that existing ImageNet-based OOD benchmarks are saturated. The finding that rankings differ between conventional hard-OOD and the proposed ImageNet-X is potentially informative for the community. The strength is the systematic separation of semantic and covariate shifts and the inclusion of real-world WILDS data. However, the benchmark construction relies on an unvalidated WordNet split whose internal consistency is not demonstrated, and the reported numbers lack error bars despite the use of random splits; these issues currently limit the strength of the empirical conclusions.
major comments (4)
- [Section 2.1, Table 2] The central ImageNet-X construction is not internally consistent. The paper states that the 1000 ImageNet classes are grouped into 558 WordNet parent classes and that the labels within each parent class are split into two halves. By pigeonhole, at least 116 parent classes contain exactly one ImageNet class (1000 - 558 = 442 multi-child parents at most, leaving at least 116 singletons). A singleton class cannot be split into ID and OOD counterparts, yet the paper never specifies how these classes are assigned. If they are assigned to OOD, then 116 of the 500 OOD classes (23%) have no same-parent ID sibling; even a balanced assignment leaves about 58 OOD classes (12%) without one. For those classes the semantic shift is parent-level, not sibling-level, and may be much larger. Since AUROC in Table 2 aggregates over all 500 OOD classes, the aggregate can be inflated by easy, large-shift classes, so the reported rankings may reflect performance on large shifts rather than on the intended fine-grained shifts. The authors must report the exact singleton assignment and verify that the resulting ID/OOD split provides comparable semantic difficulty across classes.
- [Section 2.1] The paper assumes that the immediate WordNet parent class is a valid and uniform proxy for small semantic shift, but it provides no quantitative validation. Some parent classes contain very close species (e.g., dog breeds), while others may contain more heterogeneous members; the internal similarity of parent groups is not measured. Without evaluating inter-class distances, human similarity judgments, or at least per-parent AUROC breakdowns, the claim that ImageNet-X guarantees small and controlled semantic shifts is unsupported. This assumption is load-bearing because it is the basis for the main comparison in Table 2.
- [Section 3.5, Table 2] The WILDS-based results lack error bars and the random split is not reproducible. iWildCam uses 91 randomly selected animal classes as ID and FMoW uses 31 randomly selected classes as ID, but no random seed is reported and the split appears to be a single draw. The few-shot methods are averaged over 3 seeds, but this does not capture the variance of the random class split. In Table 2, several FMoW AUROC values differ by only 1–2 points (e.g., 55.45 vs. 55.89), and the iWildCam ranking of zero-shot methods spans a range of about 16 points. The claim that 'no single winner emerges' and the reported Spearman correlation of 0.90 are not robust without confidence intervals or repeated benchmark splits.
- [Section 2.2, Table 1] ImageNet-FS-X does not apply the ImageNet-X label split uniformly. Table 1 shows that ImageNet-R has only 495 ID and 492 OOD classes, and ImageNet-C has only 315 ID and 313 OOD classes, rather than the nominal 500/500. The paper does not explain how the label splitting method from ImageNet-X is adapted when the variant dataset does not contain all 500 ID labels or all 500 OOD labels. This partial coverage changes the composition of the covariate-shifted ID set and the OOD set, and may affect the comparison between ImageNet-X and ImageNet-FS-X in findings F2 and F3. The authors should specify which classes are dropped and how the AUROC denominators are computed.
minor comments (4)
- [Abstract] There is a missing space in 'code ishttps://github.com/hoshi23/OOD-X-Benchmarks'; it should read 'code is https://github.com/hoshi23/OOD-X-Benchmarks'.
- [Table 2] The table is visually dense and does not report standard deviations or confidence intervals for the AUROC values, even though few-shot methods are averaged over 3 seeds. Adding error bars or variance notation would strengthen the interpretation of ranking changes.
- [Section 3.4] The Spearman rank-order correlation of 0.90 is reported without a confidence interval or significance test; with only 10 methods, the estimate is unstable and should be interpreted cautiously.
- [Appendix B] Figure A is described as showing 'one representative sample' from ID and OOD, but the figure contains two images per method; the caption should be clarified to indicate that two samples are shown.
Circularity Check
No significant circularity; the paper is an empirical benchmark study whose measurements and constructions are self-contained.
full rationale
The paper's central claims are empirical measurements on newly constructed benchmarks, not derivations from fitted parameters. ImageNet-X is constructed by splitting ImageNet-1k labels according to WordNet parent classes; the resulting AUROC rankings in Table 2 are direct evaluations of published CLIP-based methods. No model parameter is fitted to a target outcome and then reported as a prediction, and no benchmark result is defined in terms of a method output it is later used to explain. The two baselines from the same author group (GL-MCM [10] and LoCoOp [6]) are included as competing methods with hyperparameters from their original publications; they are not used to tune or justify the benchmark, so this is ordinary self-citation rather than load-bearing circularity. The cited survey [31] is used only to define the zero-shot OOD detection setting, not to establish the paper's findings. The skeptical observation about the pigeonhole constraint from 1000 ImageNet classes into 558 WordNet parents is a legitimate benchmark-validity concern, but it is not circularity: even if some OOD classes lack a same-parent ID sibling, the benchmark construction is still an external design choice rather than an equation that reduces to its own output. The claim that 'none of them consistently outperforms the others' is an empirical result from Table 2 and is not assumed by the benchmark construction. Therefore the paper is self-contained against external benchmarks and receives a score of 0.
Assumptions & free parameters
free parameters (2)
- ID/OOD class split ratio =
500/500 for ImageNet-X; 91/91 for iWildCam; 31/31 for FMoW
- WILDS random split seed =
not reported
assumptions (4)
- domain assumption WordNet immediate-parent hierarchy is a valid and uniform proxy for semantic closeness between ImageNet labels.
- domain assumption ImageNet-V2, ImageNet-R, and ImageNet-C are covariate-shifted versions of ImageNet-1k with the same label semantics, and their partial label coverage can be treated as aligned.
- domain assumption WILDS domain shifts (camera location, year) are meaningful covariate shifts, and random label splits create balanced semantic difficulty.
- domain assumption Pooling AUROC over multiple OOD sources is a sufficient metric for comparing method characteristics.
Cite this review
Pith. "Pith review of A Benchmark and Evaluation for Real-World Out-of-Distribution Detection Using Vision-Language Models." pith.science (2026). https://pith.science/paper/Y5M55OOZ
@misc{pith2026250118463,
author = {Pith},
title = {Pith review of: A Benchmark and Evaluation for Real-World Out-of-Distribution Detection Using Vision-Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/Y5M55OOZ}},
note = {Machine review of arXiv:2501.18463}
}
read the original abstract
Out-of-distribution (OOD) detection is a task that detects OOD samples during inference to ensure the safety of deployed models. However, conventional benchmarks have reached performance saturation, making it difficult to compare recent OOD detection methods. To address this challenge, we introduce three novel OOD detection benchmarks that enable a deeper understanding of method characteristics and reflect real-world conditions. First, we present ImageNet-X, designed to evaluate performance under challenging semantic shifts. Second, we propose ImageNet-FS-X for full-spectrum OOD detection, assessing robustness to covariate shifts (feature distribution shifts). Finally, we propose Wilds-FS-X, which extends these evaluations to real-world datasets, offering a more comprehensive testbed. Our experiments reveal that recent CLIP-based OOD detection methods struggle to varying degrees across the three proposed benchmarks, and none of them consistently outperforms the others. We hope the community goes beyond specific benchmarks and includes more challenging conditions reflecting real-world scenarios. The code is https://github.com/hoshi23/OOD-X-Benchmarks.
Reference graph
Works this paper leans on
-
[1]
Concrete problems in ai safety,
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man ´e, “Concrete problems in ai safety,”arXiv preprint arXiv:1606.06565, 2016
arXiv 2016
-
[2]
Taxonomy of machine learning safety: A survey and primer,
Sina Mohseni, Haotao Wang, Chaowei Xiao, Zhiding Yu, Zhangyang Wang, and Jay Yadawa, “Taxonomy of machine learning safety: A survey and primer,”ACM Computing Sur- veys, vol. 55, no. 8, pp. 1–38, 2022
work page 2022
-
[3]
Generalized out-of-distribution detection: A survey,
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu, “Generalized out-of-distribution detection: A survey,”IJCV, vol. 132, no. 12, pp. 5635–5662, 2024
work page 2024
-
[4]
Learning transferable visual models from nat- ural language supervision,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever, “Learning transferable visual models from nat- ural language supervision,” inICML, 2021
2021
-
[5]
Delving into out-of-distribution detection with vision-language representations,
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li, “Delving into out-of-distribution detection with vision-language representations,” inNeurIPS, 2022
work page 2022
-
[6]
Lo- coop: Few-shot out-of-distribution detection via prompt learn- ing,
Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa, “Lo- coop: Few-shot out-of-distribution detection via prompt learn- ing,” inNeurIPS, 2023
work page 2023
-
[7]
Imagenet large scale visual recognition challenge,
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei, “Imagenet large scale visual recognition challenge,” IJCV, vol. 115, pp. 211–252, 2015
work page 2015
-
[8]
The inaturalist species classification and detection dataset,
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie, “The inaturalist species classification and detection dataset,” inCVPR, 2018
work page 2018
Show all 34 references
-
[9]
Describing textures in the wild,
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi, “Describing textures in the wild,” inCVPR, 2014
2014
-
[10]
Gl- mcm: Global and local maximum concept matching for zero- shot out-of-distribution detection,
Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa, “Gl- mcm: Global and local maximum concept matching for zero- shot out-of-distribution detection,”IJCV, 2025
2025
-
[11]
Negative label guided ood detection with pretrained vision-language models,
Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han, “Negative label guided ood detection with pretrained vision-language models,” inICLR, 2024
2024
-
[12]
Envisioning outlier exposure by large language models for out-of-distribution detection,
Chentao Cao, Zhun Zhong, Zhanke Zhou, Yang Liu, Tongliang Liu, and Bo Han, “Envisioning outlier exposure by large language models for out-of-distribution detection,” inICML, 2024
2024
-
[13]
Clipn for zero-shot ood detection: Teaching clip to say no,
Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li, “Clipn for zero-shot ood detection: Teaching clip to say no,” inICCV, 2023
2023
-
[14]
Learning transferable negative prompts for out-of- distribution detection,
Tianqi Li, Guansong Pang, Xiao Bai, Wenjun Miao, and Jin Zheng, “Learning transferable negative prompts for out-of- distribution detection,” inCVPR, 2024
2024
-
[15]
Id-like prompt learning for few- shot out-of-distribution detection,
Yichen Bai, Zongbo Han, Bing Cao, Xiaoheng Jiang, Qinghua Hu, and Changqing Zhang, “Id-like prompt learning for few- shot out-of-distribution detection,” inCVPR, 2024
2024
-
[16]
Full-spectrum out-of-distribution detection,
Jingkang Yang, Kaiyang Zhou, and Ziwei Liu, “Full-spectrum out-of-distribution detection,”IJCV, vol. 131, no. 10, pp. 2607–2622, 2023
2023
-
[17]
WILDS: A benchmark of in-the-wild distribution shifts,
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, A...
2021
-
[18]
A baseline for detecting misclassified and out-of-distribution examples in neural net- works,
Dan Hendrycks and Kevin Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural net- works,” inICLR, 2017
2017
-
[19]
Mos: Towards scaling out-of- distribution detection for large semantic space,
Rui Huang and Yixuan Li, “Mos: Towards scaling out-of- distribution detection for large semantic space,” inCVPR, 2021
2021
-
[20]
Open-set recog- nition: A good closed-set classifier is all you need?,
S Vaze, K Han, A Vedaldi, and A Zisserman, “Open-set recog- nition: A good closed-set classifier is all you need?,” inICLR, 2022
2022
-
[21]
In or out? fixing imagenet out-of-distribution detection evalua- tion,
Julian Bitterwolf, Maximilian Mueller, and Matthias Hein, “In or out? fixing imagenet out-of-distribution detection evalua- tion,”arXiv preprint arXiv:2306.00826, 2023
2023 arXiv
-
[22]
Openood v1. 5: Enhanced bench- mark for out-of-distribution detection,
Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, and Yueqian others Lin, “Openood v1. 5: Enhanced bench- mark for out-of-distribution detection,” inNeurIPS 2023 Work- shop on Distribution Shifts: New Frontiers with F oundation Models
2023
-
[23]
Imagenet-ood: Deciphering modern out-of-distribution detec- tion algorithms,
William Yang, Byron Zhang, and Olga Russakovsky, “Imagenet-ood: Deciphering modern out-of-distribution detec- tion algorithms,” inICLR, 2024
2024
-
[24]
Large-scale open-set classification protocols for imagenet,
Andres Palechor, Annesha Bhoumik, and Manuel G ¨unther, “Large-scale open-set classification protocols for imagenet,” in WACV, 2023
2023
-
[25]
Wordnet: An electronic lexical database.,
George A. Miller, “Wordnet: An electronic lexical database.,” MIT press, 1998
1998
-
[26]
Do imagenet classifiers generalize to ima- genet?,
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar, “Do imagenet classifiers generalize to ima- genet?,” inICML, 2019
2019
-
[27]
The many faces of robustness: A critical analysis of out-of-distribution generalization,
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer, “The many faces of robustness: A critical analysis of out-of-distribution generalization,” inI...
2021
-
[28]
Benchmarking neural network robustness to common corruptions and perturbations,
Dan Hendrycks and Thomas Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” inICLR, 2018
2018
-
[29]
Rethinking the evaluation of out-of-distribution detection: A sorites paradox,
Xingming Long, Jie Zhang, Shiguang Shan, and Xilin Chen, “Rethinking the evaluation of out-of-distribution detection: A sorites paradox,”arXiv preprint arXiv:2406.09867, 2024
2024 arXiv
-
[30]
Leveraging visual attention for out-of-distribution detection,
Luca Cultrera, Lorenzo Seidenari, and Alberto Del Bimbo, “Leveraging visual attention for out-of-distribution detection,” inICCVW, 2023
2023
-
[31]
Gen- eralized out-of-distribution detection and beyond in vision lan- guage model era: A survey,
Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Yueqian Lin, Qing Yu, Go Irie, Shafiq Joty, Yixuan Li, Hai Li, Ziwei Liu, Toshihiko Yamasaki, and Kiyoharu Aizawa, “Gen- eralized out-of-distribution detection and beyond in vision lan- guage model era: A survey,”arXiv...
2024 arXiv
-
[32]
Learning to prompt for vision-language models,
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu, “Learning to prompt for vision-language models,”IJCV, vol. 130, no. 9, pp. 2337–2348, 2022
2022
-
[33]
Vim: Out-of-distribution with virtual-logit matching,
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang, “Vim: Out-of-distribution with virtual-logit matching,” in CVPR, 2022
2022
-
[34]
Clipood: Generalizing clip to out-of-distributions,
Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long, “Clipood: Generalizing clip to out-of-distributions,” inICML, 2023. Appendix A. EXPERIMENTAL DETAIL In this section, we describe the details of each CLIP-based out-of-distribution (OOD) detection...
2023
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.