{"id":"7dbc872e-2923-4603-b664-d87741aafcf5","arxiv_id":"2505.08604","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MECAM detects out-of-distribution medical images by masking class-activation regions and measuring the resulting feature shift in a multi-exit network.","lead":"Medical AI models often cannot tell when a new image is completely different from their training data. This paper proposes covering the image regions a model relies on and measuring how much its internal picture changes, a method that reports near-perfect detection on skin and pathology images tested against X-rays, CTs, and natural images.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central assumption that OOD CAMs are unfocused is unmeasured; focused OOD CAMs would invert the score in Eq. (1).","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the method assumes OOD data does not produce the focused CAMs that ID data does, and this is never measured. I agree this is the central risk. The paper's score is a monotone function of feature shift; the direction 'large shift => ID' is only valid if masking with the inverted CAM removes a substantial portion of informative features for ID and not for OOD. That is exactly the premise that remains unverified. Other issues raised by the reader (single-run metrics, small HeadCT test set, unexplained 'small-scale setting', poor baseline numbers) affect the strength of the empirical comparison but are secondary; even if the numbers are reproducible, the claim would still rest on the unmeasured CAM-focus premise. A concrete measurement of CAM concentration on OOD inputs would directly settle whether the mechanism works, and would also explain the weaker iSUN results if they are caused by focused OOD attention. Since the reader already conditionally accepts the paper pending replication, and this concern does not change the conditional status, the verdict remains unchanged.","tokens_in":8697,"tokens_out":3678,"duration_ms":39430,"concrete_test":"Using the released code and the trained ResNet-50 models, compute for every test image in ISIC19, PathMNIST, RSNA, COVID-19, HeadCT, and iSUN a CAM focus statistic, e.g., the inverse entropy of the normalized multi-exit CAM or the fraction of its mass in the top 10% of spatial locations. Compare the distribution of this statistic between ID and OOD, and plot it against the per-sample Score_OOD. If OOD focus scores overlap with ID, or if any OOD sample with above-median focus is nonetheless scored ID, the assumption in Section 2 fails. A sharper version: rank all OOD samples by focus score and compute the MECAM FPR95 on the top-decile focus group; if it is substantially higher than the overall FPR95, the focused-CAM failure mode is real.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The decision rule in Eq. (1) hinges on the direction of the feature-shift comparison: an input is ID when Score_OOD >= tau. The paper justifies this direction by claiming that masking with the inverted CAM removes regions highly relevant for ID data, while OOD data 'often lacks such focused activations' (Abstract; Section 2). This premise is never measured on any OOD dataset. The CAM is computed for the predicted class P at each exit; for OOD inputs P is arbitrary, and nothing guarantees the resulting CAM is diffuse. Natural images (iSUN) contain salient objects and strong edges that typically produce focused attention maps even in networks trained on medical images. If such an OOD input produces a concentrated CAM, x' inherits a large hole and the feature shift (v_i - v'_i)^2 will be large, pushing the input toward the ID side. The reported degradation on iSUN (FPR95 17.11 and 31.99 for ISIC19 ResNet-18/50 in Table 1) is consistent with this failure mode. The entire claimed advantage rests on the sign of this shift being aligned with ID/OOD status; without a quantitative check of CAM concentration on OOD data, the central mechanism is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MECAM, an unsupervised out-of-distribution (OOD) detection framework for medical imaging that leverages multi-exit class activation maps (CAMs) and input feature masking. The method computes CAMs at multiple network exits, aggregates them with a confidence-weighted scheme, masks the input image with the inverted aggregated CAM, and scores OOD-ness by the mean squared feature shift between the original and masked images. The authors claim that in-distribution (ID) images exhibit larger feature shifts under this masking, and they report strong detection performance on ISIC19 and PathMNIST as ID datasets against RSNA Pneumonia, COVID-19, HeadCT, and iSUN as OOD datasets, consistently outperforming several existing OOD detection baselines. The paper includes ablation studies showing the benefit of using multiple exits and comparisons with Grad-CAM and LayerCAM.","tokens_in":8885,"tokens_out":5717,"duration_ms":55912,"significance":"If the central assumption is validated, the method is potentially significant: it offers a CAM-based, fully unsupervised OOD score that requires no OOD labels and no fitted constants, and it reports very strong FPR95/AUC numbers on several medical benchmarks (e.g., 0.30% FPR95 and 99.69 AUC on PathMNIST ResNet-50 versus RSNA). The manuscript is clearly written, the method is described with explicit equations, and the authors provide a public code repository, which supports reproducibility. The main scientific risk is that the decision rule's direction depends on an unmeasured property of OOD data, and the empirical evaluation is presented without uncertainty estimates, making the reported gains difficult to assess.","major_comments":[{"comment":"The decision rule in Eq. (1) identifies an input as ID when the feature shift Score_OOD is large, justified by the claim that ID data have focused CAMs while OOD data 'often lacks such focused activations.' This premise is never measured on any OOD dataset in the paper. If OOD inputs (e.g., natural images like iSUN) produce confident, concentrated CAMs, the masked image would contain a large hole, the feature shift would be large, and the input would be pushed toward the ID side, inverting the intended score direction. The elevated FPR95 values on iSUN (17.11% and 31.99% for ISIC19 ResNet-18/50 in Table 1) are consistent with this failure mode. The authors should provide a quantitative check of CAM concentration (e.g., entropy or top-k mass) on ID versus each OOD dataset for each exit, and show that the distribution of Score_OOD separates in the assumed direction. Without this, the central mechanism of the method remains unsupported.","section":"Section 2, Eq. (1); Abstract"},{"comment":"All metrics are reported as point estimates from a single run, with no error bars, standard deviations, or statistical significance tests. This is especially problematic for the HeadCT OOD set, which contains only 200 images: the FPR95 values are quantized (e.g., 6.00%, 24.50%) and a few percent difference corresponds to only a handful of images. The claimed superiority of MECAM (e.g., FPR95 0.30% vs. 6.33% on PathMNIST ResNet-50 against RSNA) cannot be distinguished from noise without multiple runs. The authors should report means and standard deviations over at least three random seeds, or provide per-sample score distributions and confidence intervals.","section":"Tables 1-3"},{"comment":"Table 1 is captioned 'under a small-scale setting,' but the text never defines what this setting is. The evaluation section says a mixed test set is created by combining the full test sets of the ID and OOD datasets, with no mention of subsampling. If a subsample was used, its size, composition, and sampling procedure must be specified, since FPR95 estimates are highly sensitive to the size of the OOD set. Please clarify whether the HeadCT set contains only 200 images in total or 200 images after subsampling.","section":"Section 3 and Table 1 caption"},{"comment":"The comparison protocol is under-specified. It is not stated whether the single-head baselines (MSP, ODIN, Energy, DICE, FeatureNorm, CORES) are evaluated on the same multi-exit network (e.g., using the final exit) or on standard ResNets; MOOD, a multi-exit baseline, also lacks configuration details. Several baseline numbers appear implausibly poor, such as FeatureNorm achieving an AUC of 23.12% on ISIC19 ResNet-50 against RSNA, which is below chance; this suggests possible implementation or hyperparameter mismatches that could disadvantage the baselines. The authors should describe the exact backbone, exit structure, and feature layer used for each baseline, and ideally release the evaluation code so the comparisons can be reproduced.","section":"Section 4.1 and Table 1"},{"comment":"The headline improvement claims, such as 'AUC improvement of 32.93% for ResNet-18 and 54.95% for ResNet-50,' are not reproducible from Table 1 because no comparison baseline is named for these percentages. Whether these are relative improvements, absolute differences, or averages over OOD datasets is unclear. For example, the absolute AUC gap between MECAM and the best baseline on ISIC19 ResNet-18 RSNA is 6.84 points (99.62 vs. 92.78), not 32.93 points. Please state the exact computation and the reference baseline for each claimed improvement.","section":"Section 4.1"}],"minor_comments":[{"comment":"The CAM extraction step is described only as 'applying the softmax function along the class dimension,' which is not the standard CAM formulation used in Grad-CAM or LayerCAM. Please provide the exact CAM equation or cite the specific formula from reference [1] so that the method is reproducible.","section":"Section 2"},{"comment":"The weighting scheme for combining CAMs across exits is stated as 'based on the normalized classification logit across all exits,' but the exact formula is not given. Please provide the equation or a precise description of how the exit weights are computed.","section":"Section 2"},{"comment":"There are several typos in the abstract, including 'mult-exit networks' and 'Inthispaper,weintroducea novelunsupervisedOODde- tection framework.' Please proofread the text.","section":"Abstract"},{"comment":"Phrases such as '6× and 3× lower' for FPR95 reductions are ambiguous; please report the actual FPR95 ratios or the percentage reductions, and clarify whether the comparison is against the best baseline or the average of baselines.","section":"Section 4.1"},{"comment":"Reference [11] lists the first author as 'Lemar Abdi, M.'; please verify this spelling against the original publication.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and presents a novel and potentially useful idea. The main technical risk is that the core mechanism (OOD CAMs are less focused) is never validated, and the evaluation lacks uncertainty quantification. These issues are fixable with additional experiments and clarifications, so I recommend major revision rather than rejection. The unusually poor performance of some baselines also warrants careful checking of the evaluation protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a clean, easily stated idea — mask an input with the inverted multi-exit CAM and score OOD by how little the features shift — and the reported numbers are striking, but the central assumption that OOD inputs produce diffuse CAMs is asserted, not measured, and the evaluation has enough gaps that the near-perfect results should be read with caution.\n\nWhat's actually new: the specific combination of multi-exit CAM aggregation and inverted-CAM feature masking as an unsupervised OOD score. The components are established, but I don't know of prior work that uses exactly this score. The ablation in Table 3 is a genuine strength: it shows that combining exits matters and that later exits dominate, which makes the mechanism more credible. The paper also gives explicit equations and a code link, so replication is feasible.\n\nWhere it gets soft. The load-bearing claim is the sign of the score: ID if the masked image causes a large feature shift, OOD if the shift is small. The paper justifies this by saying OOD data 'lacks focused activations.' That premise is never tested on any OOD set. Natural images like iSUN have salient structure and can easily produce concentrated CAMs even in medical-trained networks; when that happens, the score in Eq. (1) pushes the input to the ID side. The relatively weak iSUN results (FPR95 17–32 on ISIC19) are consistent with exactly this failure mode. So the core mechanism is plausible but unsupported until the authors measure CAM concentration on OOD data.\n\nAlso worth flagging: Table 1 reports a 'small-scale setting' that is never defined; all metrics appear to be single runs with no error bars; some baseline numbers look implausibly bad (FeatureNorm AUC 23.12 on ISIC19/ResNet-50) and need checking; HeadCT has only 200 OOD images, which makes those FPR95 numbers noisy; and the 'first CAM-based OOD detection' claim is asserted without a systematic search of the literature. None of these are fatal on their own, but together they mean the headline claims of 'consistent' superiority are not yet established.\n\nBottom line: the idea is worth taking seriously and replicating. The main missing experiment — measuring whether OOD CAMs are actually diffuse — is easy to run, and the eval needs standard robustness reporting. This deserves a serious referee, but I'd want the revision to close the gap between the claims and the evidence.","headline":"A clean, easily stated OOD idea with striking numbers, but the central assumption that OOD CAMs are diffuse is never measured and the evaluation has enough gaps that I'd want a revision before believing the near-perfect results.","tokens_in":9476,"tokens_out":1544,"would_cite":true,"duration_ms":13966,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An image is in-distribution if erasing its class-activation map shifts the model's features a lot; the paper reports near-perfect OOD detection from this test alone.","keywords":["out-of-distribution detection","class activation maps","multi-exit networks","feature masking","medical imaging","unsupervised detection","skin lesion classification","histopathology"],"falsifier":"An experiment would directly measure CAM focus on the OOD sets: for each OOD dataset, compute the entropy or top-1% mass of the predicted-class CAM, and check whether a substantial fraction of OOD images have CAMs as concentrated as the ID images. If they do, the inverted-CAM mask would also remove their evidence and the feature-shift score would fail to separate the classes.","tokens_in":8462,"feed_emoji":"🩻","tokens_out":7182,"duration_ms":64539,"temperature":0.7,"pith_summary":"Medical classifiers need to know when an input is outside what they were trained on, but current OOD detectors often rely on confidence scores that are overconfident and spatially blind. This paper proposes MECAM, an unsupervised detector built from a multi-exit network: at each exit it computes a class activation map (CAM), aggregates the maps with confidence weights, masks the input with the inverted CAM, and measures the mean-squared shift in the penultimate-layer features. The paper's claim is that in-distribution inputs lose important visual evidence when masked, so their features move a lot, while OOD inputs are comparatively unaffected, so their features barely move. In the reported experiments on ISIC19 and PathMNIST as in-distribution sets and RSNA, COVID-19, HeadCT, and iSUN as OOD sets, this single scalar separates the two classes with FPR95 as low as 0.30% and AUC up to 99.69%, beating the compared baselines.","feed_headline":"Masking the model's focus region flags OOD images at 0.3% FPR","feed_subtitle":"MECAM erases an image's class-activation map and measures the feature shift: big shift means in-distribution, small shift means OOD.","key_machinery":"The mechanism is the multi-exit CAM combined with inverted-CAM masking and the MSE feature-shift score. A multi-exit network produces a CAM at every exit; each CAM is softmax-normalized, the predicted class's map is selected, and the maps are weighted by the normalized classification logits so that high-confidence exits dominate. The weighted map is subtracted from 1 and multiplied with the input, producing $x'$. The load-bearing idea is that masking the regions a model actually uses constitutes an intervention, and the size of the intervention's effect on the final feature vector is the OOD score. The multi-exit aspect lets the map capture both global and local activations at different resolutions.","core_discovery":"The paper's central discovery is a criterion: define $M(x)$ as the confidence-weighted aggregate of per-exit CAMs for the predicted class, build the masked image $x' = x \\otimes (1 - M(x))$, and score the input by the mean squared error between the feature vectors $v = f(x)$ and $v' = f(x')$ taken at the layer immediately before the final convolutional layer, $\\mathrm{Score}_{\\mathrm{OOD}} = \\frac{1}{d}\\sum_{i=1}^{d}(v_i - v'_i)^2$. The paper states the principle directly: a larger feature shift means the original features relied heavily on the masked regions, so the input is in-distribution; a smaller shift means OOD. On the tested medical benchmarks this score gives, for example, 0.30% FPR95 and 99.69 AUC on PathMNIST with ResNet-50 against RSNA, and the method reports consistent gains over the seven compared baselines across all four OOD sets.","pith_inferences":["The method could be interpreted as a counterfactual sensitivity test: ID inputs are the ones whose predictions are causally anchored in a localized region; an editor's inference is that the same score might also rank OOD inputs by difficulty or by distance from the training domain.","The paper fixes the feature space at the layer before the final convolutional layer. A testable extension would replace that layer with each exit's embedding and measure whether the score becomes more calibrated or less sensitive to network depth.","The core assumption—OOD images lack focused CAMs—is never directly measured. A natural experiment is to compute CAM concentration (e.g., entropy or top-1% mass) on the OOD sets and correlate it with score separation; if some OOD classes produce focused maps, the criterion should fail on them.","Because CAMs can be computed for any convolutional architecture, the same masking score could be applied to segmentation models or 3D medical volumes, where the mask becomes a volumetric map; the paper does not test this."],"forward_implications":["On the reported benchmarks, MECAM lowers FPR95 by factors of roughly 3 to 27 compared with MSP, ODIN, Energy, DICE, FeatureNorm, CORES, and MOOD, depending on ID/OOD pair and backbone.","The ablation shows each exit contributes: single-exit CAMs give worse separation, and combining all four exits gives the best numbers, so the multi-exit aggregation itself is doing load-bearing work.","MECAM beats single-layer CAM methods (Grad-CAM and LayerCAM), indicating that hierarchical aggregation matters more than the choice of CAM formula.","Because the detector needs only the trained multi-exit classifier and no OOD samples, it can be attached to an existing medical classifier at deployment time; the only tuning is the threshold at 95% true positive rate on ID data.","On the natural-image OOD set iSUN, MECAM also reduces FPR95 relative to baselines, suggesting the spatial-focus signal transfers beyond medical domain gaps."],"supporting_citations":[{"why":"Supplies the multi-exit network training and CAM extraction scheme that MECAM builds on.","marker":"[1]"},{"why":"Softmax-confidence baseline that MECAM is compared against.","marker":"[7]"},{"why":"Hierarchical CAM method compared as a single-layer CAM alternative.","marker":"[8]"},{"why":"Temperature-scaled softmax baseline for OOD detection.","marker":"[12]"},{"why":"Multi-exit energy-based OOD baseline whose architecture MECAM also uses.","marker":"[13]"},{"why":"Energy-based score baseline for OOD detection.","marker":"[15]"},{"why":"Grad-CAM baseline CAM method used in the comparison.","marker":"[19]"},{"why":"Sparsification-based baseline for OOD detection.","marker":"[21]"},{"why":"Convolutional response-based score baseline for OOD detection.","marker":"[22]"},{"why":"Feature-norm baseline for OOD detection.","marker":"[26]"}],"fun_headline_variants":["Erase CAM, measure shift: OOD detection via feature masking","Multi-exit CAMs plus masking spot OOD at 0.3% FPR","Feature shift after CAM erasure flags OOD in medical scans","Mask class-activation maps to spot out-of-distribution images","Unsupervised OOD detection with multi-exit CAM masking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that in-distribution images activate a tight, meaningful region while out-of-distribution images do not, so erasing that region damages in-distribution features much more than out-of-distribution features; the paper asserts this but never measures it on the OOD sets.","fun_headline_variants_meta":{"raw":{"variants":["Erase CAM, measure shift: OOD detection via feature masking","Multi-exit CAMs plus masking spot OOD at 0.3% FPR","Feature shift after CAM erasure flags OOD in medical scans","Mask class-activation maps to spot out-of-distribution images","Unsupervised OOD detection with multi-exit CAM masking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000813,"raw_usage":{"total_tokens":3612,"prompt_tokens":1038,"completion_tokens":2574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":2481}},"tokens_in":654,"tokens_out":2574,"duration_ms":17565,"temperature":1.0,"reasoning_tokens":2481,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:51:03.094546+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An experiment would directly measure CAM focus on the OOD sets: for each OOD dataset, compute the entropy or top-1% mass of the predicted-class CAM, and check whether a substantial fraction of OOD images have CAMs as concentrated as the ID images. If they do, the inverted-CAM mask would also remove their evidence and the feature-shift score would fail to separate the classes.","supporting_citations":[{"cited_title":"In: Pro- ceedings of the IEEE/CVF conference on Computer Vision and Pattern Recogni- tion","cited_arxiv_id":null,"evidence_quote":"Multi-exit energy-based OOD baseline whose architecture MECAM also uses."},{"cited_title":"In: European Conference on Computer Vision","cited_arxiv_id":null,"evidence_quote":"Sparsification-based baseline for OOD detection."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"Convolutional response-based score baseline for OOD detection."},{"cited_title":"In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"Feature-norm baseline for OOD detection."}],"review_version":1}