Pith. sign in

REVIEW 4 major objections 6 minor 64 references

The paper proposes DCV-ROOD, a dual cross-validation framework that splits in-distribution data by stratification and out-of-distribution data by class groups, and reports that it reproduces the statistically significant differences found b

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 04:58 UTC pith:KBGZCIRS

load-bearing objection A sensible dual-CV framework for OOD evaluation whose validation is anchored to a self-defined benchmark truth; needs a baseline comparison and convergence evidence before its fast-convergence claim can be trusted. the 4 major comments →

arxiv 2509.05778 v1 pith:KBGZCIRS submitted 2025-09-06 cs.LG cs.AIstat.ML

DCV-ROOD Evaluation Framework: Dual Cross-Validation for Robust Out-of-Distribution Detection

Arantxa Urrea-Casta\~no , Nicol\'as Segura-Kunsagi , Juan Luis Su\'arez-D\'iaz , Rosana Montes , Francisco Herrera This is my paper
classification cs.LG cs.AIstat.ML
keywords out-of-distribution detectioncross-validationevaluation frameworkoutlier exposureclass hierarchybenchmark truthstatistical significanceimage classification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Out-of-distribution (OOD) detection systems are only as trustworthy as the way they are tested, and a random train/test split can easily mislead. This paper proposes DCV-ROOD, a double cross-validation scheme that treats in-distribution and out-of-distribution data differently: stratified folds for ID classes, group folds for OOD classes, so outlier-exposed models never see their test outliers during training. The authors validate the scheme by comparing its ten runs against a "benchmark truth" built from 100 random splits, and report that it reproduces most statistically significant differences between eight detection methods — up to 9.8571 out of 10 for the TPR5 metric. If correct, DCV-ROOD gives OOD researchers a cheaper, stable way to reach conclusions that previously required an order of magnitude more computation.

Core claim

The central claim is that an evaluation framework can be adapted to OOD detection without leaking OOD classes into training and still match exhaustive evaluation. DCV-ROOD does this by running a stratified k-fold on the ID set and a group k-fold at the class level on the OOD set, pairing a fresh OOD class group with the held-out ID fold in every iteration. For hierarchical datasets, the split respects the whole class hierarchy so that OOD tests are neither artificially easy nor prone to false-OOD flags from unseen subclasses. Against a benchmark truth defined as the mean of 100 random train/test splits, DCV-ROOD reproduces significant differences between eight OOD detectors with hit rates ab

What carries the argument

The dual cross-validation mechanism: stratified k-fold for ID data (preserving class proportions) plus group k-fold for OOD data (keeping whole classes together so test classes never appear in training), with folds joined by index in each iteration. In hierarchical settings, Algorithm 2 selects ID/OOD splits by strata above the classification level, and Algorithm 3 builds stratified ID folds at the deepest level and group OOD folds per stratum, preventing semantic leakage across the hierarchy.

Load-bearing premise

The 'benchmark truth'—the mean of 100 random train/test splits over different OOD class subsets—is a converged estimate of true OOD performance, and DCV-ROOD's deterministic class-group splits estimate exactly that same quantity.

What would settle it

Build a synthetic dataset with known ground-truth anomaly scores (for example, Gaussian classes with specified separations), compute the exact OOD error directly, then compare DCV-ROOD's 10-run estimates against that exact truth; if the agreement that appears against the 100-split benchmark collapses, the benchmark was not a valid target.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • OOD evaluation can be run with one set of k-fold runs (here, 10) instead of 100 random repetitions and still preserve most statistically significant method rankings.
  • Methods with and without outlier exposure can be compared on identical partitions, because group k-fold protects the train/test separation for exposed outliers.
  • For hierarchical datasets, the hierarchy-aware split avoids artificially easy OOD tasks and reduces false-OOD flags caused by unseen subclasses inside ID classes.
  • All five reported metrics—TPR5, AUROC, AUPR, F1, and accuracy@90—stay above a hit rate of 8, so the speedup does not sacrifice metric coverage.
  • The low error rates reported (at most about 2.1 among the studied thresholds) indicate that DCV-ROOD rarely invents significant differences that the benchmark does not show.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same dual-split logic could apply to other tasks where test data have hidden group structure—for example, anomaly detection in medical or industrial settings—not just image classification.
  • A natural testable extension is to validate DCV-ROOD against an external ground truth, such as synthetic data with known anomaly rates, which would remove the self-referential aspect of validating against an internally defined benchmark truth.
  • Because group k-fold is deterministic, repeated DCV-ROOD runs vary mainly through ID stratification and model initialization; isolating these factors could show how much of the residual disagreement with the benchmark comes from each.
  • If DCV-ROOD is widely adopted, sample-size guidance for OOD evaluation may shift: these results suggest that grouped k-fold runs can carry more statistical information per run than the same number of random splits when OOD classes are the main source of variability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes DCV-ROOD, an evaluation framework that applies stratified k-fold CV to ID data and group k-fold CV to OOD data, with a hierarchical-extension variant (Algorithms 1–3). To validate the framework, the authors construct a 'benchmark truth' as the mean of 100 random train/test splits (Section 5.1), then compare pairwise statistically significant differences among eight OOD detection methods found in the benchmark with those found in 10 DCV-ROOD runs (Section 5.2), plus a method-wise comparison (Section 5.3). They report high hit rates (up to 9.8571/10 for TPR5) and low error rates, concluding that DCV-ROOD converges quickly to true performance.

Significance. The paper addresses a real and under-served problem: standard CV is not directly applicable to OOD detection because of class leakage in outlier exposure. The algorithmic description is clear, reproducible, and the hierarchical splitting (Algorithms 2–3) is a thoughtful contribution. The experimental breadth—8 methods, 11 ID-OOD pairs, with/without outlier exposure—is substantial. However, the central validation claim depends entirely on a self-defined benchmark truth whose convergence and sampling protocol are not demonstrated, and no alternative evaluation scheme is compared. The contribution is potentially valuable, but the current evidence does not yet establish the advertised 'very fast convergence to the true performance.'

major comments (4)
  1. [§5.1] Convergence is asserted, not demonstrated. The text states 'A setup based on 100 repetitions was found to be sufficient to conclude that the results had converged' but gives no convergence curves, stopping criterion, or variance/confidence intervals for the benchmark means. Since the benchmark truth is the sole reference for all hit-rate computations in Tables 7–8, an unconverged or noisy benchmark would directly propagate into the reported agreement scores. Please report running means vs. number of repetitions, Monte Carlo standard errors, and stability of the pairwise significance decisions (e.g., via bootstrap over the 100 seeds).
  2. [§5.1 vs §3.1] The benchmark sampling protocol is not matched to DCV-ROOD's fold structure. The benchmark 'reserved a subset of OOD classes' for outlier-exposure training and 'exclusive OOD classes' for testing, but the paper does not state the proportion of OOD classes used for training, nor whether the test OOD set is the complement. In DCV-ROOD with group k-fold, each OOD class appears in training in exactly K-1 folds and in test in exactly one fold. If the benchmark used a different split ratio (e.g., 50/50), the OE training distribution differs; the benchmark then estimates a different target quantity. Specify the benchmark's OOD split ratio and justify equivalence, or adjust the benchmark protocol.
  3. [§5.2, Tables 7–8] No baseline evaluation scheme is compared. The claim of 'very fast convergence to the true performance' requires a comparator such as single holdout, repeated holdout with a small number of repeats, or a standard k-fold without group-OOD partitioning. Without such a baseline, the hit rates show agreement with the authors' own benchmark but not that DCV-ROOD is faster or more accurate than existing validation practice. Also, the 'error rate' is acknowledged as relative; the benchmark's own false-positive/false-negative rate in significance detection is not bounded, so the error-rate denominators are not anchored.
  4. [§4.1, Algorithm 2] The value of p (proportion of subclasses assigned to OOD) for the SUPERCIFAR100 experiments is not reported. Figure 4 illustrates p=0.4, but Section 5 and Table 2 do not state the p used in the benchmark or in DCV-ROOD. This parameter controls the ID/OOD class balance and the difficulty of the hierarchical task; without reporting it and the random seeds, the experiments are not reproducible and the comparison between benchmark and DCV-ROOD is incomplete.
minor comments (6)
  1. [Tables 6, 26] Values in the hit-rate tables are run together (e.g., 'ebo -10010010-16') and are very hard to read. Use explicit cell separators. Also 'KLN' should be 'KLM', and the appendix note 'p-value≤0.5' should be 'p≤0.1' or 'p≤0.05'.
  2. [Figure 4 caption] 'DC-ROOD' should be 'DCV-ROOD'.
  3. [§4.4] State whether early stopping (patience 3) also applies during the 5 epochs of outlier-exposure fine-tuning, or whether the OE step uses a fixed 5 epochs without early stopping.
  4. [§5.3, Table 9] The first column labels ('TPR5 pval≤0.05', 'auroc pval≤0.05', etc.) are confusing because they mix metric names with significance thresholds. Clarify what is being tested.
  5. [§2.1] Minor typo: 'changes on the clases' should be 'changes on the classes'.
  6. [§5.1] The statement that the benchmark setup is 'similar to that used in [35]' is not self-explanatory, as [35] concerns partition-induced dataset shift in k-fold CV. Explain the connection or cite a more directly relevant benchmark construction.

Circularity Check

0 steps flagged

Mild self-reference in benchmark truth; no circular reduction by construction.

full rationale

The paper's central validation is an empirical agreement study, not a derivation that reduces to its own inputs. A benchmark truth is constructed as the mean over 100 random class-disjoint splits (Sec 5.1), and the paper then measures how often statistically significant pairwise differences found in that benchmark are also found in 10 independent DCV-ROOD runs (Tables 7, 8). The hit rates (e.g., 9.8571 for TPR5 at p<=0.1) are computed from fresh evaluations; no parameter is fitted to the benchmark, and no equation defines DCV-ROOD's output in terms of the benchmark. The only self-citation is [35] (co-authored by F. Herrera), used as methodological inspiration for the benchmark-truth strategy; the benchmark itself is constructed in this paper, so the citation is not load-bearing. Two limitations are worth flagging but are not circularity: (1) the benchmark truth is a self-defined reference rather than an external ground truth, and its convergence is asserted ('A setup based on 100 repetitions was found to be sufficient to conclude that the results had converged') without showing the convergence analysis or confidence intervals; (2) the benchmark's OOD-class sampling proportion is not reported, so its target distribution may not exactly match DCV-ROOD's group-k-fold exposure ratio. These affect how strongly the 'true performance' claim can be interpreted, but they do not make the agreement scores equivalent by construction. Score 2 reflects the minor self-citation and the self-referential benchmark, with no reducible circular step.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The framework rests entirely on domain assumptions about how to avoid leakage in OOD evaluation and on the self-defined benchmark truth. No new physical or mathematical entities are introduced. The main free parameters are the fold count K, the hierarchical OOD proportion p, the number of benchmark repetitions, and training hyperparameters, none of which are justified by ablations or sensitivity analysis.

free parameters (6)
  • Number of folds K for DCV-ROOD
    Not stated for the experiments; Figure 1 shows a 5-fold example and Figure 4 a 4-fold example, so K is a hand-chosen parameter that affects the size of training and test OOD sets.
  • Proportion p of subclasses assigned to OOD (Algorithm 2)
    Figure 4 uses 40% OOD (60% ID), but the paper does not report p for the SUPERCIFAR100 experiments.
  • Random repetitions for benchmark truth = 100
    Chosen because "mean convergence" was found sufficient, but no convergence plot or criterion is shown (Section 5.1).
  • OE fine-tuning epochs = 5
    Additional 5 epochs with outlier exposure loss; no ablation.
  • Early stopping patience = 3
    Patience of 3 epochs for the base classifier; standard but arbitrary.
  • Significance thresholds for hit rates = 0.1 and 0.05
    Hit rates are reported at p<=0.1 as the main result; no multiple-testing correction is applied across the 28 method pairs.
axioms (6)
  • domain assumption Conventional k-fold CV on OOD detection data exposes the model to OOD classes during training, undermining the validity of the evaluation.
    Section 3, first paragraph: "Conventional CV risks exposing the model to OOD data during training..." This motivates the entire dual-split design.
  • domain assumption Grouping OOD data by class for folds and stratifying ID data by class prevents leakage and yields a fair evaluation.
    Section 3.1 defines the framework on this premise; no formal proof is given, only the heuristic that same-class instances must not be split.
  • domain assumption The mean of 100 random repetitions is a converged approximation of the true performance of each OOD detector.
    Section 5.1 states "A setup based on 100 repetitions was found to be sufficient to conclude that the results had converged" without showing the convergence analysis.
  • standard math Mann-Whitney U tests on per-run performance samples are a valid way to detect significant differences between OOD detection methods.
    Used in Section 5.2 on the benchmark truth and DCV-ROOD outputs; valid under the usual i.i.d. assumption, but with n=10 the test has low power and this is not discussed.
  • domain assumption In hierarchical datasets, using the level immediately above the classification level as strata and sampling a proportion of subclasses per stratum for OOD yields fair ID/OOD partitions.
    Section 3.2 states this ensures higher levels are represented in every fold; no empirical or formal validation of this fairness claim.
  • domain assumption No class at the classification level may appear in both ID and OOD splits.
    Section 3.2 states this as a necessity to avoid the same data being both ID and OOD.

pith-pipeline@v1.3.0-alltime-deepseek · 27264 in / 17661 out tokens · 175455 ms · 2026-08-05T04:58:20.961139+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of DCV-ROOD Evaluation Framework: Dual Cross-Validation for Robust Out-of-Distribution Detection." pith.science (2026). https://pith.science/paper/KBGZCIRS

@misc{pith2026250905778,
  author       = {Pith},
  title        = {Pith review of: DCV-ROOD Evaluation Framework: Dual Cross-Validation for Robust Out-of-Distribution Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KBGZCIRS}},
  note         = {Machine review of arXiv:2509.05778}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Out-of-distribution (OOD) detection plays a key role in enhancing the robustness of artificial intelligence systems by identifying inputs that differ significantly from the training distribution, thereby preventing unreliable predictions and enabling appropriate fallback mechanisms. Developing reliable OOD detection methods is a significant challenge, and rigorous evaluation of these techniques is essential for ensuring their effectiveness, as it allows researchers to assess their performance under diverse conditions and to identify potential limitations or failure modes. Cross-validation (CV) has proven to be a highly effective tool for providing a reasonable estimate of the performance of a learning algorithm. Although OOD scenarios exhibit particular characteristics, an appropriate adaptation of CV can lead to a suitable evaluation framework for this setting. This work proposes a dual CV framework for robust evaluation of OOD detection models, aimed at improving the reliability of their assessment. The proposed evaluation framework aims to effectively integrate in-distribution (ID) and OOD data while accounting for their differing characteristics. To achieve this, ID data are partitioned using a conventional approach, whereas OOD data are divided by grouping samples based on their classes. Furthermore, we analyze the context of data with class hierarchy to propose a data splitting that considers the entire class hierarchy to obtain fair ID-OOD partitions to apply the proposed evaluation framework. This framework is called Dual Cross-Validation for Robust Out-of-Distribution Detection (DCV-ROOD). To test the validity of the evaluation framework, we selected a set of state-of-the-art OOD detection methods, both with and without outlier exposure. The results show that the method achieves very fast convergence to the true performance.

Figures

Figures reproduced from arXiv: 2509.05778 by Arantxa Urrea-Casta\~no, Francisco Herrera, Juan Luis Su\'arez-D\'iaz, Nicol\'as Segura-Kunsagi, Rosana Montes.

Figure 1
Figure 1. Figure 1: Illustration of DCV-ROOD used in an outlier exposure scenario. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: A simplified view of the category hierarchy is shown, featuring two superclasses, four classes, and four [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Visualization of a False OOD due to missing subclass representation in training. Although the model learns [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Balanced hierarchical classes with 2 superclasses, each containing 5 classes and 5 subclasses per class. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 53 canonical work pages · 1 internal anchor

  1. [1]

    Pattern Recognition40(8), 2154–2162 (2007)

    An, S., Liu, W., Venkatesh, S.: Fast cross-validation algorithms for least squares support vector machine and kernel ridge regression. Pattern Recognition40(8), 2154–2162 (2007)

    work page 2007

  2. [2]

    Information Sciences191, 192–213 (2012)

    Bergmeir, C., Ben ´ıtez, J.M.: On the use of cross-validation for time series predictor evaluation. Information Sciences191, 192–213 (2012)

    work page 2012

  3. [3]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Boult, T.E., Cruz, S., Dhamija, A.R., Gunther, M., Henrydoss, J., Scheirer, W.J.: Learning and the unknown: Sur- veying steps toward open world recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 9801–9807 (2019)

    work page 2019

  4. [4]

    Pattern Recognition36(11), 2585–2592 (2003)

    Cawley, G.C., Talbot, N.L.: Efficient leave-one-out cross-validation of kernel fisher discriminant classifiers. Pattern Recognition36(11), 2585–2592 (2003)

    work page 2003

  5. [5]

    The Journal of Machine Learning Research11, 2079–2107 (2010)

    Cawley, G.C., Talbot, N.L.: On over-fitting in model selection and subsequent selection bias in performance evaluation. The Journal of Machine Learning Research11, 2079–2107 (2010)

    work page 2079

  6. [6]

    ACM Computing Surveys56(3), 1–24 (2023)

    Christen, P., Hand, D.J., Kirielle, N.: A review of the f-measure: its history, properties, criticism, and alternatives. ACM Computing Surveys56(3), 1–24 (2023)

    2023

  7. [7]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., Vedaldi, A.: Describing textures in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3606–3613 (2014)

    work page 2014

  8. [8]

    In: 2017 International Joint Conference on Neural Networks (IJCNN)

    Cohen, G., Afshar, S., Tapson, J., Van Schaik, A.: Emnist: Extending mnist to handwritten letters. In: 2017 International Joint Conference on Neural Networks (IJCNN). pp. 2921–2926. IEEE (2017)

    work page 2017

  9. [9]

    In: Proceedings of the 23rd International Conference on Machine Learning

    Davis, J., Goadrich, M.: The relationship between precision-recall and roc curves. In: Proceedings of the 23rd International Conference on Machine Learning. pp. 233–240 (2006)

    work page 2006

  10. [10]

    Hastie, T.: The elements of statistical learning: data mining, inference, and prediction (2009)

    work page 2009

  11. [11]

    Taylor & Francis (2025)

    Hendrycks, D.: Introduction to AI safety, ethics, and society. Taylor & Francis (2025)

    work page 2025

  12. [12]

    In: International Conference on Machine Learning

    Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., Song, D.: Scaling out- of-distribution detection for real-world settings. In: International Conference on Machine Learning. pp. 8759–

    work page

  13. [13]

    arXiv preprint arXiv:2109.13916 (2021)

    Hendrycks, D., Carlini, N., Schulman, J., Steinhardt, J.: Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916 (2021)

    Pith/arXiv arXiv 2021

  14. [14]

    In: International Conference on Learning Representations (2017)

    Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: International Conference on Learning Representations (2017)

    2017

  15. [15]

    arXiv preprint arXiv:1812.04606 (2018)

    Hendrycks, D., Mazeika, M., Dietterich, T.: Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606 (2018)

    Pith/arXiv arXiv 2018

  16. [16]

    In: 2023 IEEE International Conference On Artificial Intelligence Testing (AITest)

    Henriksson, J., Berger, C., Ursing, S., Borg, M.: Evaluation of out-of-distribution detection performance on autonomous driving datasets. In: 2023 IEEE International Conference On Artificial Intelligence Testing (AITest). pp. 74–81. IEEE (2023)

    work page 2023

  17. [17]

    arXiv preprint arXiv:2503.04739 (2025)

    Herrera-Poyatos, A., Del Ser, J., de Prado, M.L., Wang, F.Y ., Herrera-Viedma, E., Herrera, F.: Responsible artificial intelligence systems: A roadmap to society’s trust through trustworthy ai, auditability, accountability, and governance. arXiv preprint arXiv:2503.04739 (2025)

    arXiv 2025

  18. [18]

    Pattern Recognition161, 111214 (2025)

    Jia, Y ., Li, J., Zhao, G., Liu, S., Sun, W., Lin, L., Li, G.: Enhancing out-of-distribution detection via diversified multi-prototype contrastive learning. Pattern Recognition161, 111214 (2025)

    work page 2025

  19. [19]

    Pattern Recognition 69, 94–106 (2017) 18

    Jiang, G., Wang, W.: Error estimation based on variance analysis of k-fold cross-validation. Pattern Recognition 69, 94–106 (2017) 18

    work page 2017

  20. [20]

    Scientific Data3(1), 1–9 (2016)

    Johnson, A.E., Pollard, T.J., Shen, L., Lehman, L.w.H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., Mark, R.G.: Mimic-iii, a freely accessible critical care database. Scientific Data3(1), 1–9 (2016)

    work page 2016

  21. [21]

    In: Pro- ceedings of the 10th Annual Conference on Computational Learning Theory

    Kearns, M., Ron, D.: Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. In: Pro- ceedings of the 10th Annual Conference on Computational Learning Theory. pp. 152–162 (1997)

    work page 1997

  22. [22]

    Advances in Neural Information Processing Systems36, 43754–43779 (2023)

    Kim, J.H., Yun, S., Song, H.O.: Neural relation graph: A unified framework for identifying label noise and outlier data. Advances in Neural Information Processing Systems36, 43754–43779 (2023)

    work page 2023

  23. [23]

    In: Proceedings of the 14th International Joint Conference on Artificial Intelligence - V olume 2

    Kohavi, R., et al.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the 14th International Joint Conference on Artificial Intelligence - V olume 2. pp. 1137–1143. Montreal, Canada (1995)

    work page 1995

  24. [24]

    Technometrics10(1), 1–11 (1968)

    Lachenbruch, P.A., Mickey, M.R.: Estimation of error rates in discriminant analysis. Technometrics10(1), 1–11 (1968)

    work page 1968

  25. [25]

    Pattern Recognition37(9), 1835–1840 (2004)

    Lau, K., Wu, Q.: Leave one support vector out cross validation for fast estimation of generalization errors. Pattern Recognition37(9), 1835–1840 (2004)

    work page 2004

  26. [26]

    CS 231N7(7), 3 (2015)

    Le, Y ., Yang, X.: Tiny imagenet visual recognition challenge. CS 231N7(7), 3 (2015)

    2015

  27. [27]

    Advances in Neural Information Processing Systems31(2018)

    Lee, K., Lee, K., Lee, H., Shin, J.: A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in Neural Information Processing Systems31(2018)

    work page 2018

  28. [28]

    arXiv preprint arXiv:2312.11536 (2023)

    Liu, L., Qin, Y .: Fast decision boundary based out-of-distribution detector. arXiv preprint arXiv:2312.11536 (2023)

    Pith/arXiv arXiv 2023

  29. [29]

    Pattern Recognition161, 111308 (2025)

    Liu, W., Yu, H., Wang, Y ., Liu, B., Tao, D., Chen, H.: Iw-vit: Independence-driven weighting vision transformer for out-of-distribution generalization. Pattern Recognition161, 111308 (2025)

    work page 2025

  30. [30]

    Advances in Neural Information Processing Systems33, 21464–21475 (2020)

    Liu, W., Wang, X., Owens, J., Li, Y .: Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems33, 21464–21475 (2020)

    work page 2020

  31. [31]

    In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Liu, X., Lochman, Y ., Zach, C.: Gen: Pushing the limits of softmax-based out-of-distribution detection. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23946–23955 (2023)

    work page 2023

  32. [32]

    In: 2023 3rd International Conference on Robotics, Automation and Intelligent Control (ICRAIC)

    Liu, Z., Zhang, W., Wang, R.: Ood-ens: Cross-validated out-of-distribution ensemble detector. In: 2023 3rd International Conference on Robotics, Automation and Intelligent Control (ICRAIC). pp. 95–101. IEEE (2023)

    work page 2023

  33. [33]

    Advances in Neural Information Processing Systems37, 89806–89833 (2024)

    Long, X., Zhang, J., Shan, S., Chen, X.: Rethinking the evaluation of out-of-distribution detection: A sorites paradox. Advances in Neural Information Processing Systems37, 89806–89833 (2024)

    work page 2024

  34. [34]

    Pattern Recognition p

    Miao, W., Pang, G., Nguyen, T.T., Fang, R., Zheng, J., Bai, X.: Opencil: Benchmarking out-of-distribution detection in class incremental learning. Pattern Recognition p. 112163 (2025)

    work page 2025

  35. [35]

    IEEE Transactions on Neural Networks and Learning Systems23(8), 1304–1312 (2012)

    Moreno-Torres, J.G., S ´aez, J.A., Herrera, F.: Study on the impact of partition-induced dataset shift onk-fold cross-validation. IEEE Transactions on Neural Networks and Learning Systems23(8), 1304–1312 (2012)

    work page 2012

  36. [36]

    Handbook of Social Psychology2, 80–203 (1968)

    Mosteller, F., Tukey, J.W., et al.: Data analysis, including statistics. Handbook of Social Psychology2, 80–203 (1968)

    work page 1968

  37. [37]

    Journal of the American Statistical Association 58(302), 275–309 (1963)

    Mosteller, F., Wallace, D.L.: Inference in an authorship problem: A comparative study of discrimination meth- ods applied to the authorship of the disputed federalist papers. Journal of the American Statistical Association 58(302), 275–309 (1963)

    work page 1963

  38. [38]

    In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning

    Netzer, Y ., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y ., et al.: Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning. vol. 2011, p. 7. Granada (2011)

    work page 2011

  39. [39]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Park, J., Jung, Y .G., Teoh, A.B.J.: Nearest neighbor guidance for out-of-distribution detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1686–1695 (2023)

    work page 2023

  40. [40]

    the Journal of Machine Learning Research12, 2825–2830 (2011)

    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V ., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V ., et al.: Scikit-learn: Machine learning in python. the Journal of Machine Learning Research12, 2825–2830 (2011)

    work page 2011

  41. [41]

    In: 2010 IEEE International Conference on Data Mining

    Raeder, T., Hoens, T.R., Chawla, N.V .: Consequences of variability in classifier performance estimates. In: 2010 IEEE International Conference on Data Mining. pp. 421–430. IEEE (2010)

    work page 2010

  42. [42]

    arXiv preprint arXiv:2106.09022 (2021) 19

    Ren, J., Fort, S., Liu, J., Roy, A.G., Padhy, S., Lakshminarayanan, B.: A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022 (2021) 19

    Pith/arXiv arXiv 2021

  43. [43]

    IEEE Transactions on Pattern Analysis and Machine Intelligence32(3), 569–575 (2009)

    Rodriguez, J.D., Perez, A., Lozano, J.A.: Sensitivity analysis of k-fold cross validation in prediction error esti- mation. IEEE Transactions on Pattern Analysis and Machine Intelligence32(3), 569–575 (2009)

    work page 2009

  44. [44]

    The Annals of Statistics pp

    Rogers, W.H., Wagner, T.J.: A finite sample distribution-free performance bound for local discrimination rules. The Annals of Statistics pp. 506–514 (1978)

    work page 1978

  45. [45]

    International Journal of Computer Vision 115(3), 211–252 (2015)

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115(3), 211–252 (2015)

    work page 2015

  46. [46]

    arXiv preprint arXiv:2110.14051 (2021)

    Salehi, M., Mirzaei, H., Hendrycks, D., Li, Y ., Rohban, M.H., Sabokrou, M.: A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges. arXiv preprint arXiv:2110.14051 (2021)

    Pith/arXiv arXiv 2021

  47. [47]

    STOOD-X methodology: using statistical nonparametric test for OOD Detection Large-Scale datasets enhanced with explainability

    Sevillano-Garc ´ıa, I., Luengo, J., Herrera, F.: Stood-x methodology: using statistical nonparametric test for ood detection large-scale datasets enhanced with explainability. arXiv preprint arXiv:2504.02685 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  48. [48]

    Cambridge University Press (2014)

    Shalev-Shwartz, S., Ben-David, S.: Understanding machine learning: From theory to algorithms. Cambridge University Press (2014)

    work page 2014

  49. [49]

    Journal of the Royal Statistical Society: Series B (Methodological)36(2), 111–133 (1974)

    Stone, M.: Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society: Series B (Methodological)36(2), 111–133 (1974)

    work page 1974

  50. [50]

    In: International Conference on Machine Learning

    Sun, Y ., Ming, Y ., Zhu, X., Li, Y .: Out-of-distribution detection with deep nearest neighbors. In: International Conference on Machine Learning. pp. 20827–20840. PMLR (2022)

    work page 2022

  51. [51]

    The American Atatistician61(3), 198–200 (2007)

    Taleb, N.N.: Black swans and the domains of statistics. The American Atatistician61(3), 198–200 (2007)

    work page 2007

  52. [52]

    In: 2023 IEEE International Conference on Bioinfor- matics and Biomedicine (BIBM)

    Tamo, J.B., Nnamdi, M.C., Lesbats, L., Shi, W., Zhong, Y ., Wang, M.D.: Uncertainty-aware ensemble learning models for out-of-distribution medical imaging analysis. In: 2023 IEEE International Conference on Bioinfor- matics and Biomedicine (BIBM). pp. 4243–4250. IEEE (2023)

    work page 2023

  53. [53]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    Van Horn, G., Mac Aodha, O., Song, Y ., Cui, Y ., Sun, C., Shepard, A., Perona, P., Belongie, S.: The inaturalist species classification and detection dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8769–8778 (2018)

    work page 2018

  54. [54]

    Pattern Recognition162, 111401 (2025)

    Wei, J., Wang, G., Zhang, S.: Fine-grained medical image out-of-distribution detection through multi-view feature uncertainty and adversarial sample generation. Pattern Recognition162, 111401 (2025)

    work page 2025

  55. [55]

    arXiv preprint arXiv:2007.05566 (2020)

    Winkens, J., Bunel, R., Roy, A.G., Stanforth, R., Natarajan, V ., Ledsam, J.R., MacWilliams, P., Kohli, P., Karthikesalingam, A., Kohl, S., et al.: Contrastive training for improved out-of-distribution detection. arXiv preprint arXiv:2007.05566 (2020)

    Pith/arXiv arXiv 2007

  56. [56]

    Chemometrics and Intelligent Laboratory Systems56(1), 1–11 (2001)

    Xu, Q.S., Liang, Y .Z.: Monte carlo cross validation. Chemometrics and Intelligent Laboratory Systems56(1), 1–11 (2001)

    work page 2001

  57. [57]

    In: 2016 IEEE 6th International Conference on Advanced Computing (IACC)

    Yadav, S., Shukla, S.: Analysis of k-fold cross-validation over hold-out validation on colossal datasets for quality classification. In: 2016 IEEE 6th International Conference on Advanced Computing (IACC). pp. 78–83. IEEE (2016)

    work page 2016

  58. [58]

    International Journal of Computer Vision132(12), 5635–5662 (2024)

    Yang, J., Zhou, K., Li, Y ., Liu, Z.: Generalized out-of-distribution detection: A survey. International Journal of Computer Vision132(12), 5635–5662 (2024)

    work page 2024

  59. [59]

    Journal of Experimental & Theoretical Artificial Intelligence12(1), 1–12 (2000)

    Zeng, X., Martinez, T.R.: Distribution-balanced stratified cross-validation for accuracy estimation. Journal of Experimental & Theoretical Artificial Intelligence12(1), 1–12 (2000)

    work page 2000

  60. [60]

    5: Enhanced benchmark for out-of-distribution detection

    Zhang, J., Yang, J., Wang, P., Wang, H., Lin, Y ., Zhang, H., Sun, Y ., Du, X., Li, Y ., Liu, Z., et al.: Openood v1. 5: Enhanced benchmark for out-of-distribution detection. arXiv preprint arXiv:2306.09301 (2023)

    Pith/arXiv arXiv 2023

  61. [61]

    In: International Conference on Machine Learning

    Zhang, L., Goldstein, M., Ranganath, R.: Understanding failures in out-of-distribution detection with deep gen- erative models. In: International Conference on Machine Learning. pp. 12427–12436. PMLR (2021)

    work page 2021

  62. [62]

    Pattern Recog- nition160, 111212 (2025)

    Zhang, Y ., Hu, J., Wen, D., Deng, W.: Unsupervised evaluation for out-of-distribution detection. Pattern Recog- nition160, 111212 (2025)

    work page 2025

  63. [63]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing28, 1198–1209 (2020)

    Zheng, Y ., Chen, G., Huang, M.: Out-of-domain detection for natural language understanding in dialog systems. IEEE/ACM Transactions on Audio, Speech, and Language Processing28, 1198–1209 (2020)

    work page 2020

  64. [64]

    Zhou, Z.H.: Open-environment machine learning. National Science Review9(8), 1–11 (2022) 20 A Appendix of results The following presents the results obtained according to the metrics employed in this study: AUROC (Section A.1), AUPR (Section A.2), F1-score (Section A.3), Accuracy@90 (Section A.4) and TPR@5 (Section A.5). Each subsection of this appendix pr...

    work page arXiv 2022