REVIEW 5 major objections 5 minor 26 references
Unsupervised Out-of-Distribution Detection in Medical Imaging Using Multi-Exit Class Activation Maps and Feature Masking
T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read An image is in-distribution if erasing its class-activation map shifts the model's features a lot; the paper reports near-perfect OOD detection from this test alone.
desk verdict A clean, easily stated OOD idea with striking numbers, but the central assumption that OOD CAMs are diffuse is never measured and the evaluation has enough gaps that I'd want a revision before believing the near-perfect results. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is the multi-exit CAM combined with inverted-CAM masking and the MSE feature-shift score. A multi-exit network produces a CAM at every exit; each CAM is softmax-normalized, the predicted class's map is selected, and the maps are weighted by the normalized classification logits so that high-confidence exits dominate. The weighted map is subtracted from 1 and multiplied with the input, producing $x'$. The load-bearing idea is that masking the regions a model actually uses constitutes an intervention, and the size of the intervention's effect on the final feature vector is the OOD score. The multi-exit aspect lets the map capture both global and local activations at different resolutions.
What would settle it
An experiment would directly measure CAM focus on the OOD sets: for each OOD dataset, compute the entropy or top-1% mass of the predicted-class CAM, and check whether a substantial fraction of OOD images have CAMs as concentrated as the ID images. If they do, the inverted-CAM mask would also remove their evidence and the feature-shift score would fail to separate the classes.
Extended reading notes
Core claim
The paper's central discovery is a criterion: define $M(x)$ as the confidence-weighted aggregate of per-exit CAMs for the predicted class, build the masked image $x' = x \otimes (1 - M(x))$, and score the input by the mean squared error between the feature vectors $v = f(x)$ and $v' = f(x')$ taken at the layer immediately before the final convolutional layer, $\mathrm{Score}_{\mathrm{OOD}} = \frac{1}{d}\sum_{i=1}^{d}(v_i - v'_i)^2$. The paper states the principle directly: a larger feature shift means the original features relied heavily on the masked regions, so the input is in-distribution; a smaller shift means OOD. On the tested medical benchmarks this score gives, for example, 0.30% FPR95 and 99.69 AUC on PathMNIST with ResNet-50 against RSNA, and the method reports consistent gains over the seven compared baselines across all four OOD sets.
Load-bearing premise
The load-bearing premise is that in-distribution images activate a tight, meaningful region while out-of-distribution images do not, so erasing that region damages in-distribution features much more than out-of-distribution features; the paper asserts this but never measures it on the OOD sets.
Editorial extensions
If this is right
- On the reported benchmarks, MECAM lowers FPR95 by factors of roughly 3 to 27 compared with MSP, ODIN, Energy, DICE, FeatureNorm, CORES, and MOOD, depending on ID/OOD pair and backbone.
- The ablation shows each exit contributes: single-exit CAMs give worse separation, and combining all four exits gives the best numbers, so the multi-exit aggregation itself is doing load-bearing work.
- MECAM beats single-layer CAM methods (Grad-CAM and LayerCAM), indicating that hierarchical aggregation matters more than the choice of CAM formula.
- Because the detector needs only the trained multi-exit classifier and no OOD samples, it can be attached to an existing medical classifier at deployment time; the only tuning is the threshold at 95% true positive rate on ID data.
- On the natural-image OOD set iSUN, MECAM also reduces FPR95 relative to baselines, suggesting the spatial-focus signal transfers beyond medical domain gaps.
Reading between the lines
- The method could be interpreted as a counterfactual sensitivity test: ID inputs are the ones whose predictions are causally anchored in a localized region; an editor's inference is that the same score might also rank OOD inputs by difficulty or by distance from the training domain.
- The paper fixes the feature space at the layer before the final convolutional layer. A testable extension would replace that layer with each exit's embedding and measure whether the score becomes more calibrated or less sensitive to network depth.
- The core assumption—OOD images lack focused CAMs—is never directly measured. A natural experiment is to compute CAM concentration (e.g., entropy or top-1% mass) on the OOD sets and correlate it with score separation; if some OOD classes produce focused maps, the criterion should fail on them.
- Because CAMs can be computed for any convolutional architecture, the same masking score could be applied to segmentation models or 3D medical volumes, where the mask becomes a volumetric map; the paper does not test this.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MECAM, an unsupervised out-of-distribution (OOD) detection framework for medical imaging that leverages multi-exit class activation maps (CAMs) and input feature masking. The method computes CAMs at multiple network exits, aggregates them with a confidence-weighted scheme, masks the input image with the inverted aggregated CAM, and scores OOD-ness by the mean squared feature shift between the original and masked images. The authors claim that in-distribution (ID) images exhibit larger feature shifts under this masking, and they report strong detection performance on ISIC19 and PathMNIST as ID datasets against RSNA Pneumonia, COVID-19, HeadCT, and iSUN as OOD datasets, consistently outperforming several existing OOD detection baselines. The paper includes ablation studies showing the benefit of using multiple exits and comparisons with Grad-CAM and LayerCAM.
Significance. If the central assumption is validated, the method is potentially significant: it offers a CAM-based, fully unsupervised OOD score that requires no OOD labels and no fitted constants, and it reports very strong FPR95/AUC numbers on several medical benchmarks (e.g., 0.30% FPR95 and 99.69 AUC on PathMNIST ResNet-50 versus RSNA). The manuscript is clearly written, the method is described with explicit equations, and the authors provide a public code repository, which supports reproducibility. The main scientific risk is that the decision rule's direction depends on an unmeasured property of OOD data, and the empirical evaluation is presented without uncertainty estimates, making the reported gains difficult to assess.
major comments (5)
- [Section 2, Eq. (1); Abstract] The decision rule in Eq. (1) identifies an input as ID when the feature shift Score_OOD is large, justified by the claim that ID data have focused CAMs while OOD data 'often lacks such focused activations.' This premise is never measured on any OOD dataset in the paper. If OOD inputs (e.g., natural images like iSUN) produce confident, concentrated CAMs, the masked image would contain a large hole, the feature shift would be large, and the input would be pushed toward the ID side, inverting the intended score direction. The elevated FPR95 values on iSUN (17.11% and 31.99% for ISIC19 ResNet-18/50 in Table 1) are consistent with this failure mode. The authors should provide a quantitative check of CAM concentration (e.g., entropy or top-k mass) on ID versus each OOD dataset for each exit, and show that the distribution of Score_OOD separates in the assumed direction. Without this, the central mechanism of the method remains unsupported.
- [Tables 1-3] All metrics are reported as point estimates from a single run, with no error bars, standard deviations, or statistical significance tests. This is especially problematic for the HeadCT OOD set, which contains only 200 images: the FPR95 values are quantized (e.g., 6.00%, 24.50%) and a few percent difference corresponds to only a handful of images. The claimed superiority of MECAM (e.g., FPR95 0.30% vs. 6.33% on PathMNIST ResNet-50 against RSNA) cannot be distinguished from noise without multiple runs. The authors should report means and standard deviations over at least three random seeds, or provide per-sample score distributions and confidence intervals.
- [Section 3 and Table 1 caption] Table 1 is captioned 'under a small-scale setting,' but the text never defines what this setting is. The evaluation section says a mixed test set is created by combining the full test sets of the ID and OOD datasets, with no mention of subsampling. If a subsample was used, its size, composition, and sampling procedure must be specified, since FPR95 estimates are highly sensitive to the size of the OOD set. Please clarify whether the HeadCT set contains only 200 images in total or 200 images after subsampling.
- [Section 4.1 and Table 1] The comparison protocol is under-specified. It is not stated whether the single-head baselines (MSP, ODIN, Energy, DICE, FeatureNorm, CORES) are evaluated on the same multi-exit network (e.g., using the final exit) or on standard ResNets; MOOD, a multi-exit baseline, also lacks configuration details. Several baseline numbers appear implausibly poor, such as FeatureNorm achieving an AUC of 23.12% on ISIC19 ResNet-50 against RSNA, which is below chance; this suggests possible implementation or hyperparameter mismatches that could disadvantage the baselines. The authors should describe the exact backbone, exit structure, and feature layer used for each baseline, and ideally release the evaluation code so the comparisons can be reproduced.
- [Section 4.1] The headline improvement claims, such as 'AUC improvement of 32.93% for ResNet-18 and 54.95% for ResNet-50,' are not reproducible from Table 1 because no comparison baseline is named for these percentages. Whether these are relative improvements, absolute differences, or averages over OOD datasets is unclear. For example, the absolute AUC gap between MECAM and the best baseline on ISIC19 ResNet-18 RSNA is 6.84 points (99.62 vs. 92.78), not 32.93 points. Please state the exact computation and the reference baseline for each claimed improvement.
minor comments (5)
- [Section 2] The CAM extraction step is described only as 'applying the softmax function along the class dimension,' which is not the standard CAM formulation used in Grad-CAM or LayerCAM. Please provide the exact CAM equation or cite the specific formula from reference [1] so that the method is reproducible.
- [Section 2] The weighting scheme for combining CAMs across exits is stated as 'based on the normalized classification logit across all exits,' but the exact formula is not given. Please provide the equation or a precise description of how the exit weights are computed.
- [Abstract] There are several typos in the abstract, including 'mult-exit networks' and 'Inthispaper,weintroducea novelunsupervisedOODde- tection framework.' Please proofread the text.
- [Section 4.1] Phrases such as '6× and 3× lower' for FPR95 reductions are ambiguous; please report the actual FPR95 ratios or the percentage reductions, and clarify whether the comparison is against the best baseline or the average of baselines.
- [References] Reference [11] lists the first author as 'Lemar Abdi, M.'; please verify this spelling against the original publication.
Circularity Check
No significant circularity: MECAM's score is a stated empirical assumption, not a fitted or self-referential prediction.
full rationale
The central mechanism is an explicitly stated empirical assumption, not a derived result: the paper says 'A larger feature shift indicates that the original features relied heavily on the masked regions, suggesting that the input is in-distribution. Conversely, a smaller shift indicates OOD characteristics.' This direction of the inequality in Eq. (1) is the premise of the method, not a conclusion derived from the inputs. No OOD labels, OOD statistics, or OOD-tuned constants enter the score or threshold; the threshold is set on ID data ('the threshold τ is typically chosen such that 95% of ID data is correctly classified as ID'). The CAMs and embeddings come from the same network, but this is inherent to unsupervised OOD detection and does not make the prediction forced by construction. The only self-citation is [1] for the multi-exit training procedure ('Following [1], we train the multi-exit classifier using a multi-exit cross-entropy loss'), which is an independent published training method and is not the target OOD-detection claim. The unmeasured assumption that OOD inputs lack focused CAMs is a genuine correctness risk—focused OOD CAMs could invert the score—but that is an empirical vulnerability, not circularity. No equation in the paper reduces to another by construction, and no fitted parameter is renamed as a prediction.
Assumptions & free parameters
assumptions (3)
- domain assumption In-distribution images produce focused CAMs; OOD images produce unfocused CAMs.
- domain assumption Multi-exit networks trained with the loss from [1] yield meaningful CAMs at intermediate exits.
- domain assumption The feature embedding immediately before the final convolutional layer is a suitable space for measuring the masking shift.
Cite this review
Pith. "Pith review of Unsupervised Out-of-Distribution Detection in Medical Imaging Using Multi-Exit Class Activation Maps and Feature Masking." pith.science (2026). https://pith.science/paper/ZR2CD2VE
@misc{pith2026250508604,
author = {Pith},
title = {Pith review of: Unsupervised Out-of-Distribution Detection in Medical Imaging Using Multi-Exit Class Activation Maps and Feature Masking},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZR2CD2VE}},
note = {Machine review of arXiv:2505.08604}
}
read the original abstract
Out-of-distribution (OOD) detection is essential for ensuring the reliability of deep learning models in medical imaging applications. This work is motivated by the observation that class activation maps (CAMs) for in-distribution (ID) data typically emphasize regions that are highly relevant to the model's predictions, whereas OOD data often lacks such focused activations. By masking input images with inverted CAMs, the feature representations of ID data undergo more substantial changes compared to those of OOD data, offering a robust criterion for differentiation. In this paper, we introduce a novel unsupervised OOD detection framework, Multi-Exit Class Activation Map (MECAM), which leverages multi-exit CAMs and feature masking. By utilizing mult-exit networks that combine CAMs from varying resolutions and depths, our method captures both global and local feature representations, thereby enhancing the robustness of OOD detection. We evaluate MECAM on multiple ID datasets, including ISIC19 and PathMNIST, and test its performance against three medical OOD datasets, RSNA Pneumonia, COVID-19, and HeadCT, and one natural image OOD dataset, iSUN. Comprehensive comparisons with state-of-the-art OOD detection methods validate the effectiveness of our approach. Our findings emphasize the potential of multi-exit networks and feature masking for advancing unsupervised OOD detection in medical imaging, paving the way for more reliable and interpretable models in clinical practice.
Figures
Reference graph
Works this paper leans on
-
[1]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Chen, Y.J., Hu, X., Shi, Y., Ho, T.Y.: Ame-cam: Attentive multiple-exit cam for weakly supervised segmentation on mri brain tumor. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 173–182. Springer (2023)
2023
-
[2]
Medical Image Analysis95, 103188 (2024)
Chiu, C.H., Chen, Y.J., Wu, Y., Shi, Y., Ho, T.Y.: Achieve fairness without demo- graphics for dermatological disease diagnosis. Medical Image Analysis95, 103188 (2024)
work page 2024
-
[3]
Chowdhury, M.E., Rahman, T., Khandakar, A., Mazhar, R., Kadir, M.A., Mahbub, Z.B., Islam, K.R., Khan, M.S., Iqbal, A., Al Emadi, N., et al.: Can ai help in screening viral and covid-19 pneumonia? Ieee Access8, 132665–132676 (2020)
2020
-
[4]
In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018)
Codella, N.C., Gutman, D., Celebi, M.E., Helba, B., Marchetti, M.A., Dusza, S.W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al.: Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomed- ical imaging (isbi), hosted by the international skin imaging collaboration (isic). In: 2018 IEEE 15th intern...
2018
-
[5]
arXiv preprint arXiv:1908.02288 (2019)
Combalia, M., Codella, N.C., Rotemberg, V., Helba, B., Vilaplana, V., Reiter, O., Carrera, C., Barreiro, A., Halpern, A.C., Puig, S., et al.: Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288 (2019)
arXiv 2019
-
[6]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Graham, M.S., Pinaya, W.H.L., Wright, P., Tudosiu, P.D., Mah, Y.H., Teo, J.T., Jäger, H.R., Werring, D., Nachev, P., Ourselin, S., et al.: Unsupervised 3d out-of- distribution detection with latent diffusion models. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 446–456. Springer (2023)
work page 2023
-
[7]
arXiv preprint arXiv:1610.02136 (2016)
Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of- distribution examples in neural networks. arXiv preprint arXiv:1610.02136 (2016)
arXiv 2016
-
[8]
IEEE Transactions on Image Processing 30, 5875–5888 (2021)
Jiang, P.T., Zhang, C.B., Hou, Q., Cheng, M.M., Wei, Y.: Layercam: Exploring hierarchical class activation maps for localization. IEEE Transactions on Image Processing 30, 5875–5888 (2021)
2021
Show all 26 references
-
[9]
PLoS medicine16(1), e1002730 (2019)
Kather, J.N., Krisam, J., Charoentong, P., Luedde, T., Herpel, E., Weis, C.A., Gaiser, T., Marx, A., Valous, N.A., Ferber, D., et al.: Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS medicine16(1), e10027...
2019
-
[10]
https://doi.org/10.34740/ KAGGLE/DSV/152137, https://www.kaggle.com/dsv/152137
Kitamura, F.C.: Head ct - hemorrhage (2018). https://doi.org/10.34740/ KAGGLE/DSV/152137, https://www.kaggle.com/dsv/152137
2018
-
[11]
Lemar Abdi, M., Viviers, C.G., Peter, H.: Typicality excels likelihood for unsu- pervised out-of-distribution detection in medical imaging. In: Uncertainty for Safe Utilization of Machine Learning in Medical Imaging: 6th International Workshop, UNSURE 2024, Held in Conjunction...
2025
-
[12]
arXiv preprint arXiv:1706.02690 (2017)
Liang, S., Li, Y., Srikant, R.: Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690 (2017)
2017 arXiv
-
[13]
In: Pro- ceedings of the IEEE/CVF conference on Computer Vision and Pattern Recogni- tion
Lin, Z., Roy, S.D., Li, Y.: Mood: Multi-level out-of-distribution detection. In: Pro- ceedings of the IEEE/CVF conference on Computer Vision and Pattern Recogni- tion. pp. 15313–15323 (2021) 10 Chen et al
2021
-
[14]
Medical Image Analysis93, 103088 (2024)
Linmans, J., Raya, G., van der Laak, J., Litjens, G.: Diffusion models for out- of-distribution detection in digital pathology. Medical Image Analysis93, 103088 (2024)
2024
-
[15]
Advances in neural information processing systems33, 21464–21475 (2020)
Liu, W., Wang, X., Owens, J., Li, Y.: Energy-based out-of-distribution detection. Advances in neural information processing systems33, 21464–21475 (2020)
2020
-
[16]
Liu, Z., Zhou, J.P., Wang, Y., Weinberger, K.Q.: Unsupervised out-of-distribution detection withdiffusion inpainting.In: International Conferenceon Machine Learn- ing. pp. 22528–22538. PMLR (2023)
2023
-
[17]
In: International Conference on Medical Image Computing and Computer- Assisted Intervention
Mishra, D., Zhao, H., Saha, P., Papageorghiou, A.T., Noble, J.A.: Dual conditioned diffusion models for out-of-distribution detection: Application to fetal ultrasound videos. In: International Conference on Medical Image Computing and Computer- Assisted Intervention. pp. 216–2...
2023
-
[18]
Computers in biology and medicine132, 104319 (2021)
Rahman, T., Khandakar, A., Qiblawey, Y., Tahir, A., Kiranyaz, S., Kashem, S.B.A., Islam, M.T., Al Maadeed, S., Zughaier, S.M., Khan, M.S., et al.: Exploring the effect of image enhancement techniques on covid-19 detection using chest x-ray images. Computers in biology and medi...
2021
-
[19]
In: Proceedings of the IEEE international conference on computer vision
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad- cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
2017
-
[20]
Radiology: Artificial Intelligence1(1), e180041 (2019)
Shih, G., Wu, C.C., Halabi, S.S., Kohli, M.D., Prevedello, L.M., Cook, T.S., Sharma, A., Amorosa, J.K., Arteaga, V., Galperin-Aizenberg, M., et al.: Aug- menting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiology...
2019
-
[21]
In: European Conference on Computer Vision
Sun, Y., Li, Y.: Dice: Leveraging sparsification for out-of-distribution detection. In: European Conference on Computer Vision. pp. 691–708. Springer (2022)
2022
-
[22]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Tang, K., Hou, C., Peng, W., Chen, R., Zhu, P., Wang, W., Tian, Z.: Cores: Con- volutional response-based score for out-of-distribution detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10916–10925 (2024)
2024
-
[23]
Scientific data 5(1), 1–9 (2018)
Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5(1), 1–9 (2018)
2018
-
[24]
arXiv preprint arXiv:1504.06755 (2015)
Xu, P., Ehinger, K.A., Zhang, Y., Finkelstein, A., Kulkarni, S.R., Xiao, J.: Turk- ergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755 (2015)
2015 arXiv
-
[25]
Scientific Data 10(1), 41 (2023)
Yang,J.,Shi,R.,Wei,D.,Liu,Z.,Zhao,L.,Ke,B.,Pfister,H.,Ni,B.:Medmnistv2- a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data 10(1), 41 (2023)
2023
-
[26]
In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition
Yu, Y., Shin, S., Lee, S., Jun, C., Lee, K.: Block selection method for using feature norm in out-of-distribution detection. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition. pp. 15701–15711 (2023)
2023
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.