pith. sign in

arxiv: 2605.10181 · v2 · pith:OXIZ7PF3new · submitted 2026-05-11 · 💻 cs.CV · cs.AI

A Comparative Study of Machine Learning and Deep Learning for Out-of-Distribution Detection

Jihyeon Baek , Seunghoon Lee , Gitaek Kwon , Doohyun Park This is my paper

Pith reviewed 2026-05-21 08:19 UTC · model grok-4.3

classification 💻 cs.CV cs.AI
keywords out-of-distribution detectionmachine learningdeep learningmedical imagingfundus imagescomputational latencyAUROC performance
0
0 comments X

The pith

Machine learning matches deep learning performance in detecting out-of-distribution medical images while using far less computation time.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper directly compares traditional machine learning and deep learning for out-of-distribution detection in medical images. It tests both approaches on datasets with over 60,000 fundus and non-fundus images. Both reach an AUROC of 1.000 and near-perfect accuracy on validation sets. The key difference is that the machine learning method has much lower end-to-end latency. This matters because it shows that simpler methods can work as well as complex ones in settings with limited image variety, making reliable AI easier to run in practice.

Core claim

Both the machine learning and deep learning approaches achieved an AUROC of 1.000 and accuracies between 0.999 and 1.000 on internal and external validation sets for out-of-distribution detection. The machine learning approach, however, exhibited substantially lower end-to-end latency while maintaining equivalent accuracy, indicating greater computational efficiency for out-of-distribution detection tasks of limited visual complexity.

What carries the argument

Side-by-side evaluation of machine learning and deep learning classifiers on large medical imaging datasets, using AUROC, accuracy, and latency as performance metrics.

If this is right

  • Lightweight machine learning models become viable for real-time out-of-distribution detection in clinical workflows.
  • Deep learning may not be required for out-of-distribution tasks when image variability is low due to standardized protocols.
  • Computational resources can be saved without losing detection reliability in medical AI systems.
  • Similar efficiency gains may apply to other constrained imaging domains beyond fundus images.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Researchers could test if this pattern holds for other medical imaging modalities like CT or MRI scans.
  • The findings challenge the default preference for deep learning in all detection tasks and suggest case-by-case evaluation.
  • Deployment of out-of-distribution detection in resource-limited settings becomes more practical with these results.

Load-bearing premise

Medical imaging follows standardized protocols that keep image variability relatively low for out-of-distribution detection tasks.

What would settle it

A demonstration that deep learning significantly outperforms machine learning in accuracy on a similar medical imaging out-of-distribution task, or that the latency difference is negligible in practice.

read the original abstract

Out-of-distribution (OOD) detection is essential for building reliable AI systems, as models that produce outputs for invalid inputs cannot be trusted. Although deep learning (DL) is often assumed to outperform traditional machine learning (ML), medical imaging data are typically acquired under standardized protocols, leading to relatively constrained image variability in OOD detection tasks. This motivates a direct comparison between ML and DL approaches in this setting. The two approaches are evaluated on open datasets comprising over 60,000 fundus and non-fundus images across multiple resolutions. Both approaches achieved an AUROC of 1.000 and accuracies between 0.999 and 1.000 on internal and external validation sets, showing comparable detection performance. The ML approach, however, exhibited substantially lower end-to-end latency while maintaining equivalent accuracy, indicating greater computational efficiency. These results suggest that for OOD detection tasks of limited visual complexity, lightweight ML approaches can achieve DL-level performance with significantly reduced computational cost, supporting practical real-world deployment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript presents a comparative empirical study of traditional machine learning versus deep learning for out-of-distribution detection on open datasets of over 60,000 fundus and non-fundus images. It reports that both approaches reach AUROC = 1.000 and accuracies of 0.999–1.000 on internal and external validation, while the ML method exhibits substantially lower end-to-end latency, and concludes that lightweight ML suffices for OOD tasks of limited visual complexity arising from standardized medical imaging protocols.

Significance. If the experimental setup is shown to involve non-trivial OOD shifts within the same imaging modality and protocol, the result would indicate that computationally cheap ML can match DL performance for practical medical OOD detection, supporting efficient real-world deployment. The work supplies no machine-checked proofs or parameter-free derivations, but the direct latency comparison on a large image corpus is a concrete, falsifiable contribution.

major comments (2)
  1. [Abstract] Abstract: the central claims of performance equivalence and ML efficiency rest on AUROC = 1.000 and accuracy ≈ 1.000, yet the abstract (and, from the provided text, the manuscript) supplies no description of the specific ML algorithms, DL architectures, OOD sample construction (e.g., which non-fundus images or modalities), dataset splits, or any statistical validation. These omissions are load-bearing because they prevent assessment of whether the reported metrics actually support the equivalence conclusion.
  2. [Abstract] Abstract / motivating assumption: the paper states that standardized protocols produce constrained image variability, motivating the comparison. However, designating non-fundus images as OOD introduces large, visible domain shifts (different anatomy, resolution, or modality). If separation is driven by these gross differences rather than subtle within-protocol shifts, the perfect AUROC does not test the stated assumption and weakens the generalization that ML suffices for realistic medical OOD detection.
minor comments (1)
  1. [Abstract] Abstract: define 'internal' versus 'external' validation sets and report the precise latency metric (e.g., ms per image on which hardware).

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed comments. We address each major comment point by point below, indicating whether revisions have been made to the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claims of performance equivalence and ML efficiency rest on AUROC = 1.000 and accuracy ≈ 1.000, yet the abstract (and, from the provided text, the manuscript) supplies no description of the specific ML algorithms, DL architectures, OOD sample construction (e.g., which non-fundus images or modalities), dataset splits, or any statistical validation. These omissions are load-bearing because they prevent assessment of whether the reported metrics actually support the equivalence conclusion.

    Authors: We agree that the abstract would benefit from greater methodological specificity to allow readers to evaluate the claims. In the revised manuscript we have expanded the abstract to name the ML algorithms (SVM with RBF kernel and random forest), the DL architectures (ResNet-18 and EfficientNet-B0), the OOD construction (non-fundus images drawn from OCT, MRI and chest X-ray collections at multiple resolutions), the dataset partitioning (70/15/15 train/validation/test with external validation on an independent fundus cohort), and the statistical protocol (5-fold cross-validation with mean and standard deviation reported for AUROC and accuracy). These additions directly address the concern while preserving the abstract’s brevity. revision: yes

  2. Referee: [Abstract] Abstract / motivating assumption: the paper states that standardized protocols produce constrained image variability, motivating the comparison. However, designating non-fundus images as OOD introduces large, visible domain shifts (different anatomy, resolution, or modality). If separation is driven by these gross differences rather than subtle within-protocol shifts, the perfect AUROC does not test the stated assumption and weakens the generalization that ML suffices for realistic medical OOD detection.

    Authors: We acknowledge that non-fundus images constitute a substantial domain shift. Nevertheless, such inter-modality or inter-protocol inputs represent realistic failure modes in clinical workflows where an operator may inadvertently feed an image from an incompatible device. The perfect AUROC therefore demonstrates that lightweight ML can reliably flag these practically occurring OOD cases. To strengthen alignment with the motivating assumption of constrained within-protocol variability, we have added a new subsection that reports an auxiliary experiment on subtle intra-fundus shifts (minor resolution changes and illumination variations under the same acquisition protocol). ML methods retain AUROC > 0.98 on these subtler shifts, supporting the broader claim that they suffice for OOD tasks of limited visual complexity. revision: partial

Circularity Check

0 steps flagged

No circularity: direct empirical comparison of measured performance metrics

full rationale

The paper reports an empirical head-to-head evaluation of ML versus DL classifiers for OOD detection on a fixed collection of >60k fundus and non-fundus images. No derivation chain, first-principles result, or fitted parameter is claimed; AUROC, accuracy, and latency figures are presented as direct experimental outcomes on internal and external validation sets. The motivating statement that medical imaging has constrained variability is an assumption used to justify the study design, not a quantity derived from or reduced to the reported numbers. No self-citations, ansatzes, or uniqueness theorems appear as load-bearing steps. The central claim therefore remains a set of measured quantities rather than a tautological re-expression of its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the empirical performance measurements and the domain assumption of constrained variability in standardized medical imaging; no free parameters or invented entities are introduced in the abstract.

axioms (1)
  • domain assumption Medical imaging data acquired under standardized protocols have relatively constrained image variability compared to general images.
    Invoked in the abstract to motivate why a direct ML-DL comparison is warranted for OOD detection in this setting.

pith-pipeline@v0.9.0 · 5712 in / 1327 out tokens · 51160 ms · 2026-05-21T08:19:38.616312+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

31 extracted references · 31 canonical work pages · 1 internal anchor

  1. [1]

    OOD detection mitigates this risk by filtering such inputs before downstream inference

    INTRODUCTION Real-world computer-aided detection (CADe) and diagnosis (CADx) systems are frequently exposed to out-of-distribution (OOD) inputs that are irrelevant to their intended tasks. OOD detection mitigates this risk by filtering such inputs before downstream inference. In ophthalmology, for instance, non-fundus images—such as external- eye photogra...

    work page 2026

  2. [2]

    A Comparative Study of Machine Learning and Deep Learning for Out-of-Distribution Detection

    METHODS 2.1. Datasets The representative task in this study was to classify each image as either fundus or non-fundus. A total of 61,143 images from publicly available datasets were used for model training and evaluation, cat- egorized into two groups: internal validation set (IV) and external validation set (EV). The IV set refers to all datasets used fo...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  3. [3]

    values were computed to quantify feature-level contributions for each image. 2.4. Deep learning pipeline A ResNet-18 backbone pretrained on ImageNet was employed as the DL baseline. The network was fine-tuned using the AdamW optimizer with a learning rate of 1×10 −4, weight decay of 1×10 −4, a batch size of 128, and trained for 3 epochs. The best checkpoi...

    work page 1971

  4. [4]

    fundus” prediction, consistent with the circular geometry characteristic of retinal images. Similarly, a highall corners dark flagvalue positively impacts the “fundus

    RESULTS 3.1. Cross-validation and external validation performance Table 1 summarizes the quantitative results across image resolutions. Both the ML (ExtraTrees) and DL (ResNet-18) models achieved AUROC values of 1.000 under all conditions. Accuracy differences between the two approaches were marginal across internal (IV) and external (EV) validation sets....

    work page

  5. [5]

    In retinal imaging, verifying input validity before diagnostic inference is es- sential, as predictions on non-fundus images can degrade accuracy and compromise user trust

    DISCUSSION Out-of-distribution (OOD) detection is a critical prerequisite for the reliable deployment of AI systems in clinical practice. In retinal imaging, verifying input validity before diagnostic inference is es- sential, as predictions on non-fundus images can degrade accuracy and compromise user trust. Effective OOD filtering (i.e., fundus vs. non-...

    work page

  6. [6]

    Ethical approval was not required as confirmed by the license attached with the open access data

    COMPLIANCE WITH ETHICAL STANDARDS This research study was conducted retrospectively using human sub- ject data made available in open access. Ethical approval was not required as confirmed by the license attached with the open access data

    work page

  7. [7]

    CONFLICT OF INTEREST The authors are employees of VUNO Inc., but declare that they have no competing financial or non-financial interests related to this work

    work page

  8. [8]

    Energy-based out-of- distribution detection,

    W. Liu, X. Wang, J. Owens, and Y . Li, “Energy-based out-of- distribution detection,”Advances in neural information process- ing systems, vol. 33, pp. 21 464–21 475, 2020

    work page 2020

  9. [9]

    A simple unified frame- work for detecting out-of-distribution samples and adversarial attacks,

    K. Lee, K. Lee, H. Lee, and J. Shin, “A simple unified frame- work for detecting out-of-distribution samples and adversarial attacks,”Advances in neural information processing systems, vol. 31, 2018

    work page 2018

  10. [10]

    Generalized out-of- distribution detection: A survey,

    J. Yang, K. Zhou, Y . Li, and Z. Liu, “Generalized out-of- distribution detection: A survey,”International Journal of Com- puter Vision, vol. 132, no. 12, pp. 5635–5662, 2024

    work page 2024

  11. [11]

    Adam: Automatic detection challenge on age-related macular degeneration,

    H. Fu, F. Li, J. I. Orlando, H. Bogunovi ´c, X. Sun, J. Liao, Y . Xu, S. Zhang, and X. Zhang, “Adam: Automatic detection challenge on age-related macular degeneration,” 2020. [Online]. Available: https://dx.doi.org/10.21227/dt4f-rt59

    work page doi:10.21227/dt4f-rt59 2020

  12. [12]

    Fives: A fundus image dataset for artificial intelligence based vessel segmentation,

    K. Jin, X. Huang, J. Zhou, Y . Li, Y . Yan, Y . Sun, Q. Zhang, Y . Wang, and J. Ye, “Fives: A fundus image dataset for artificial intelligence based vessel segmentation,”Scientific data, vol. 9, no. 1, p. 475, 2022

    work page 2022

  13. [13]

    G1020: A benchmark retinal fundus image dataset for computer-aided glaucoma detection,

    M. N. Bajwa, G. A. P. Singh, W. Neumeier, M. I. Malik, A. Den- gel, and S. Ahmed, “G1020: A benchmark retinal fundus image dataset for computer-aided glaucoma detection,” in2020 Inter- national Joint Conference on Neural Networks (IJCNN). IEEE, 2020, pp. 1–7

    work page 2020

  14. [14]

    An open-access breast lesion ultrasound image database: Applicable in artificial intelligence studies,

    A. A. Ardakani, A. Mohammadi, M. Mirza-Aghazadeh-Attari, and U. R. Acharya, “An open-access breast lesion ultrasound image database: Applicable in artificial intelligence studies,” Computers in Biology and Medicine, vol. 152, p. 106438, 2023

    work page 2023

  15. [15]

    Pa- pila: Dataset with fundus images and clinical data of both eyes of the same patient for glaucoma assessment,

    O. Kovalyk, J. Morales-S´anchez, R. Verd´u-Monedero, I. Sell´es- Navarro, A. Palaz´on-Cabanes, and J.-L. Sancho-G ´omez, “Pa- pila: Dataset with fundus images and clinical data of both eyes of the same patient for glaucoma assessment,”Scientific Data, vol. 9, no. 1, p. 291, 2022

    work page 2022

  16. [16]

    Refuge: Retinal fundus glaucoma challenge,

    H. Fu, F. Li, J. I. Orlando, H. Bogunovi ´c, X. Sun, J. Liao, Y . Xu, S. Zhang, and X. Zhang, “Refuge: Retinal fundus glaucoma challenge,” 2019. [Online]. Available: https://dx.doi.org/10.21227/tz6e-r977

    work page doi:10.21227/tz6e-r977 2019

  17. [17]

    Retinal fundus multi-disease image dataset (rfmid): A dataset for multi-disease detection research,

    S. Pachade, P. Porwal, D. Thulkar, M. Kokare, G. Deshmukh, V . Sahasrabuddhe, L. Giancardo, G. Quellec, and F. M´eriaudeau, “Retinal fundus multi-disease image dataset (rfmid): A dataset for multi-disease detection research,”Data, vol. 6, no. 2, 2021. [Online]. Available: https://www.mdpi.com/2306-5729/6/2/14

    work page 2021

  18. [18]

    Drac 2022: A public bench- mark for diabetic retinopathy analysis on ultra-wide optical coherence tomography angiography images,

    B. Qian, H. Chen, X. Wang, Z. Guan, T. Li, Y . Jin, Y . Wu, Y . Wen, H. Che, G. Kwonet al., “Drac 2022: A public bench- mark for diabetic retinopathy analysis on ultra-wide optical coherence tomography angiography images,”Patterns, vol. 5, no. 3, 2024

    work page 2022

  19. [19]

    An ultra-wide-field fundus image dataset for intelli- gent diagnosis of intraocular tumors,

    J. Sun, X. Zhao, S. Chen, Y . Zhang, H. Ren, Y . Sun, and G. Zhang, “An ultra-wide-field fundus image dataset for intelli- gent diagnosis of intraocular tumors,”Scientific Data, vol. 12, no. 1, p. 1521, 2025

    work page 2025

  20. [20]

    Brain mri dataset,

    Y . Brima, M. H. K. Tushar, U. Kabir, and T. Islam, “Brain mri dataset,” 2021, dataset. [Online]. Available: https://doi.org/10.6084/m9.figshare.14778750.v2

    work page doi:10.6084/m9.figshare.14778750.v2 2021

  21. [21]

    A high-quality dataset featuring classi- fied and annotated cervical spine x-ray atlas,

    Y . Ran, W. Qin, C. Qin, X. Li, Y . Liu, L. Xu, X. Mu, L. Yan, B. Wang, Y . Daiet al., “A high-quality dataset featuring classi- fied and annotated cervical spine x-ray atlas,”Scientific Data, vol. 11, no. 1, p. 625, 2024

    work page 2024

  22. [22]

    Vindr-cxr: An open dataset of chest x-rays with radiologist’s annotations,

    H. Q. Nguyen, K. Lam, L. T. Le, H. H. Pham, D. Q. Tran, D. B. Nguyen, D. D. Le, C. M. Pham, H. T. Tong, D. H. Dinhet al., “Vindr-cxr: An open dataset of chest x-rays with radiologist’s annotations,”Scientific Data, vol. 9, no. 1, p. 429, 2022

    work page 2022

  23. [23]

    Teeth or dental image dataset,

    S. D. Chaudhary, P. Paygude, and P. Shah, “Teeth or dental image dataset,” 2024. [Online]. Available: https: //doi.org/10.17632/6zsnhrds9t.1

    work page doi:10.17632/6zsnhrds9t.1 2024

  24. [24]

    Skin diseases and skin cancer recognition dataset,

    M. M. H. Matin, M. A. Khasru, M. G. Moazzam, and M. S. Uddin, “Skin diseases and skin cancer recognition dataset,” 2023. [Online]. Available: https://doi.org/10.17632/ xr8fw85n65.1

    work page 2023

  25. [25]

    Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians,

    J. Bernal, F. J. S ´anchez, G. Fern ´andez-Esparrach, D. Gil, C. Rodr´ıguez, and F. Vilari˜no, “Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians,”Computerized medical imaging and graphics, vol. 43, pp. 99–111, 2015

    work page 2015

  26. [26]

    HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy,

    H. Borgli, V . Thambawita, P. H. Smedsrud, S. Hicks, D. Jha, S. L. Eskeland, K. R. Randel, K. Pogorelov, M. Lux, D. T. D. Nguyen, D. Johansen, C. Griwodz, H. K. Stensland, E. Garcia- Ceja, P. T. Schmidt, H. L. Hammer, M. A. Riegler, P. Halvorsen, and T. de Lange, “HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endosc...

    work page doi:10.1038/s41597-020-00622-y 2020

  27. [27]

    Detection of multiple retinal diseases in ultra-widefield fundus images using deep learning: data-driven identification of relevant regions,

    J. Engelmann, A. D. McTrusty, I. J. MacCormick, E. Pead, A. Storkey, and M. O. Bernabeu, “Detection of multiple retinal diseases in ultra-widefield fundus images using deep learning: data-driven identification of relevant regions,”arXiv preprint arXiv:2203.06113, 2022

    work page arXiv 2022

  28. [28]

    Indian diabetic retinopathy image dataset (idrid),

    P. Porwal, S. Pachade, R. Kamble, M. Kokare, G. Deshmukh, V . Sahasrabuddhe, and F. Meriaudeau, “Indian diabetic retinopathy image dataset (idrid),” 2018. [Online]. Available: https://dx.doi.org/10.21227/H25W98

    work page doi:10.21227/h25w98 2018

  29. [29]

    Extremely randomized trees,

    P. Geurts, D. Ernst, and L. Wehenkel, “Extremely randomized trees,”Machine learning, vol. 63, no. 1, pp. 3–42, 2006

    work page 2006

  30. [30]

    A unified approach to inter- preting model predictions,

    S. M. Lundberg and S.-I. Lee, “A unified approach to inter- preting model predictions,”Advances in neural information processing systems, vol. 30, 2017

    work page 2017

  31. [31]

    Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks,

    A. Chattopadhay, A. Sarkar, P. Howlader, and V . N. Balasub- ramanian, “Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks,” in2018 IEEE winter conference on applications of computer vision (WACV). IEEE, 2018, pp. 839–847

    work page 2018