REVIEW 4 major objections 5 minor 30 references
MIAS-SAM: Medical Image Anomaly Segmentation without thresholding
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read MIAS-SAM removes the threshold from unsupervised medical anomaly segmentation by prompting SAM's decoder with the weighted center of gravity of a nearest-neighbor anomaly map.
desk verdict A clean, incremental trick for prompting SAM with anomaly maps, but the 'no thresholding' claim is undercut by a test-set-selected mask index and an unfair baseline comparison. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the pairing of a memory bank of normal patches with SAM's promptable decoder. The memory bank stores $5 \times 5$ patches, extracted with stride 2 from the last layer of SAM's ViT encoder; at inference, each test patch receives the maximum distance to its nearest stored neighbours, producing an anomaly map. That map is then reduced to a single point, the intensity-weighted center of gravity with exponent $\gamma=5$, which serves as the point prompt for SAM, and the decoder's third mask, the finest of the three it produces, is taken as the final segmentation. The idea that carries the argument is that the center of gravity of the anomaly map lies inside the anomalous region and therefore acts as a reliable prompt, removing the need to binarize the map with a threshold.
What would settle it
Run MIAS-SAM on a new medical imaging dataset while committing to the third SAM mask before seeing any test labels; if the third mask is not the best one for every modality, or if fixing it beforehand drops DICE below the threshold-based baselines, the central claim fails.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that a patch-based memory bank built from SAM's ViT encoder, combined with a point prompt computed as the intensity-weighted center of gravity of the anomaly map, lets SAM's decoder produce the anomaly segmentation without thresholding. The anomaly map is generated by assigning each test patch the maximum distance to its nearest neighbours in a memory bank of normal patches, then interpolating back to image space. The center of gravity, computed with an exponent $\gamma=5$ that suppresses background pixels, is converted to positional embeddings by SAM's prompt encoder, and the decoder is run on the sum of dense and image embeddings. The authors report that of SAM's three hierarchical masks, the third and finest one consistently gives the best DICE on all three datasets, and they adopt it as the method's fixed output.
Load-bearing premise
The load-bearing premise is that SAM's third decoder mask is always the right one for the anomaly, a choice the authors made after observing test-set results; if the correct mask changes across datasets or images, MIAS-SAM needs a selection procedure just as threshold-based methods do.
Editorial extensions
If this is right
- On Brain MRI, MIAS-SAM's DICE of 37.04 is about 13% above the second-best method, and on Liver CT its 42.85 is about 185% above the second-best, both obtained without thresholding the anomaly map.
- The center-of-gravity point prompt beats both the maximum-anomaly point and a thresholded bounding-box prompt on Brain and Liver datasets, so prompt placement, not just decoder capability, drives the gain.
- Across all three datasets, the SAM decoder's third and finest mask gives the best DICE, so the pipeline can output that mask directly instead of tuning a threshold.
- The P-AUROC differences among methods are small while DICE differences are large, indicating the method's contribution is in segmentation quality rather than in ranking anomalous pixels.
- Since neither a threshold nor a text prompt is needed, the method can be applied to a new imaging modality without dataset-specific prompt engineering or validation-set threshold selection.
Reading between the lines
- Beyond the paper: the same center-of-gravity prompting could be applied to other promptable segmentation models that output multiple hierarchical masks, potentially extending threshold-free anomaly segmentation beyond SAM.
- Beyond the paper: the method's reliance on the third mask could be stated as a fixed rule, always take the finest mask, and tested on new datasets; if the best mask index varies by modality, the no-threshold claim would need to be scoped to mask selection.
- Beyond the paper: on Retina OCT the gap to the best baseline appears tied to annotation convention and SAM's tendency to segment edema fluid; aligning the target mask with clinically meaningful substructures might close the remaining gap.
- Beyond the paper: using a robust quantile of nearest-neighbor distances instead of the maximum, or adding negative prompts at background locations, could make the prompt more reliable when the anomaly map is noisy; these are natural next experiments not reported in the paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MIAS-SAM, an unsupervised anomaly segmentation method for medical images. It stores patch embeddings extracted from a SAM ViT encoder on normal training images in a memory bank. At test time, it computes patch-wise nearest-neighbor distances to produce an anomaly map, then uses the weighted center of gravity of that map (with weighting exponent gamma set to 5) as a point prompt for the SAM decoder, and finally takes the third of the three candidate masks produced by the decoder as the segmentation. The method is evaluated on Brain MRI, Liver CT, and Retina OCT datasets against six embedding-based anomaly detection baselines. The authors report DICE improvements of about 13% on Brain MRI and 185% on Liver CT, with second-best performance on Retina OCT, and they claim the method does not require thresholding the anomaly map.
Significance. If the reported results are reproducible, MIAS-SAM would be a useful contribution to unsupervised medical anomaly segmentation: it replaces threshold-based binarization with a prompt-based SAM decoding step, potentially removing a difficult hyperparameter, and it demonstrates cross-modality applicability with public code. The paper's strengths include evaluation on three public datasets, comparison with six baselines, ablations on prompt type and mask index, and a public code repository. However, the central 'threshold-free' claim is weakened by the fact that the third SAM mask is selected after inspecting test-set results, and the gamma weighting factor is fixed without validation. These issues are load-bearing because the method's main novelty is the removal of post-hoc selection choices, and they must be addressed before the central claim can be accepted.
major comments (4)
- [§4.4, Table 2] The third SAM decoder mask is selected after inspecting test-set results ('we empirically found that the third mask aligns with the finest granularity of anomalies'). Because the reported DICE scores in Table 1 use this mask, the method still contains a test-set-tuned discrete choice that plays the same role as a threshold. The paper should either select the mask on a validation split, provide a rule for choosing it without test labels, or report a sensitivity analysis showing that the choice is stable across datasets and runs.
- [§3.3, Eq. (2)] The center-of-gravity prompt depends on the weighting factor gamma, which is set to 5 in the experiments without any validation or ablation. Since the prompt entirely determines the SAM decoder input, the reported DICE values are conditional on this arbitrary hyperparameter. A sensitivity analysis over gamma, and over whether the anomaly map is normalized before computing the weighted center, is needed to establish that the method is robust rather than tuned to the test set.
- [§4.1] All six baselines are evaluated with a fixed 0.5 threshold on the anomaly map, including ReContrast, which the authors ran themselves. This is a consistent protocol, but it conflates segmentation-head performance with threshold choice. To support the claim that avoiding thresholding is beneficial, the paper should compare MIAS-SAM against baselines with oracle thresholds (e.g., the best threshold on the test set) or with validation-based threshold selection, and report both fixed-threshold and oracle-threshold DICE values.
- [§4.3, Fig. 2] The Retina OCT analysis shows that MIAS-SAM often segments the gray retinal-edema region while the ground truth is defined as the white region only. This is an important domain-specific confound. The paper should quantify how frequently the center of gravity falls in the gray region or outside the white ground-truth region, since that would directly explain the 11% DICE deficit and is relevant to assessing the method's generalizability.
minor comments (5)
- [Section 2] 'text-vison models' should be 'text-vision models'.
- [Section 4.2] 'th methodol ogy' appears to be a typo; the sentence introducing the methodology should be cleaned up.
- [Figure 2] The text describing Figure 2 lists five elements (original image, ground-truth mask, anomaly map, first mask, third mask), but the caption describes four columns; this inconsistency should be resolved.
- [Section 4.4] 'lastmaks' should be 'last mask'.
- [Table 1] The text reports standard deviations for P-AUROC but no variance or significance tests for DICE; please add standard deviations or confidence intervals for the DICE values.
Circularity Check
The 'threshold-free' result is partly undermined by test-set selection of SAM decoder mask 3, but the core anomaly-map pipeline remains independent.
-
fitted input called prediction
[Section 4.4 / Table 2; method fixed in Section 3.3; final results in Table 1]
"Among the three hierarchical outputs from the SAM decoder, we empirically found that the third mask aligns with the finest granularity of anomalies (see Section 4.4). ... Table 2 shows that our proposed technique always performs better with the last mask."
The reported DICE in Table 1 is computed using mask 3, but mask 3 was selected only after computing DICE for masks 1-3 on the test set (Table 2). Thus the evaluation metric is also the selection criterion: the headline DICE is the best-mask value on the test set, not an independent prediction of a fixed, pre-specified method. The paper explicitly criticizes this same pattern for thresholds, writing that selecting the threshold that maximizes DICE on test data 'represent[s] an upper bound for the results but possibly not the actual performance that could be obtained in real-world applications.' No validation split is described for the mask-index choice, and gamma=5 in Eq. (2) is likewise fixed without validation.
full rationale
There is no true derivation to be circular: MIAS-SAM's anomaly map is built from a memory bank of normal patch embeddings via nearest-neighbor distances, and the SAM prompt is the weighted center of gravity of that map; neither component is defined in terms of the segmentation masks or the final DICE. The central pipeline is therefore self-contained and empirically testable. The main circularity-adjacent issue is the choice of the third SAM decoder mask, which the authors 'empirically found' after evaluating all three masks on the test set (Table 2) and then used for all reported DICE scores. This is functionally similar to test-set threshold optimization, which the paper itself criticizes in the introduction, and it makes the reported segmentation scores optimistic. However, mask 3 is a fixed, discrete architectural choice rather than a per-image fitted quantity, and the anomaly-map and prompt components retain independent content. The only self-citation (reference [8]) is used as an example of threshold-optimization upper bounds and is not load-bearing. Overall, the paper is not circular in its derivation, but the central 'threshold-free' claim is materially weakened by test-set-tuned mask selection, so a moderate-low score of 3 is appropriate.
Assumptions & free parameters
free parameters (3)
- gamma (weighting factor in Eq. 2) =
5
- SAM decoder mask index =
3 (third mask)
- patch size and stride =
5x5, stride 2
assumptions (5)
- domain assumption SAM encoder features pre-trained on natural images are informative for medical anomaly detection.
- domain assumption Patch-wise nearest-neighbor distance to a memory bank of normal patches is a valid anomaly score.
- domain assumption The weighted center of gravity of the anomaly map is located inside the anomalous region and is a reliable SAM point prompt.
- ad hoc to paper The third mask returned by the SAM decoder is the right one to use for anomalies.
- domain assumption A fixed 0.5 threshold is a representative way to evaluate the baseline methods.
Cite this review
Pith. "Pith review of MIAS-SAM: Medical Image Anomaly Segmentation without thresholding." pith.science (2026). https://pith.science/paper/I5AXHSXK
@misc{pith2026250522762,
author = {Pith},
title = {Pith review of: MIAS-SAM: Medical Image Anomaly Segmentation without thresholding},
year = {2026},
howpublished = {\url{https://pith.science/paper/I5AXHSXK}},
note = {Machine review of arXiv:2505.22762}
}
read the original abstract
This paper presents MIAS-SAM, a novel approach for the segmentation of anomalous regions in medical images. MIAS-SAM uses a patch-based memory bank to store relevant image features, which are extracted from normal data using the SAM encoder. At inference time, the embedding patches extracted from the SAM encoder are compared with those in the memory bank to obtain the anomaly map. Finally, MIAS-SAM computes the center of gravity of the anomaly map to prompt the SAM decoder, obtaining an accurate segmentation from the previously extracted features. Differently from prior works, MIAS-SAM does not require to define a threshold value to obtain the segmentation from the anomaly map. Experimental results conducted on three publicly available datasets, each with a different imaging modality (Brain MRI, Liver CT, and Retina OCT) show accurate anomaly segmentation capabilities measured using DICE score. The code is available at: https://github.com/warpcut/MIAS-SAM
Figures
Reference graph
Works this paper leans on
-
[1]
Yunkang Cao, Xiaohao Xu, Chen Sun, Yuqi Cheng, Zongwei Du, Liang Gao, and Weiming Shen. Segment any anomaly without training via hybrid prompt regularization.arXiv preprint arXiv:2305.10724, 2023
arXiv 2023
-
[2]
Towards accurate unified anomaly segmentation
Wenxin Ma, Qingsong Yao, Xiang Zhang, Zhelong Huang, Zihang Jiang, and S.Kevin Zhou. Towards accurate unified anomaly segmentation. InProceedings of the Winter Conference on Applications of Computer Vision (WACV), pages 1342–1352, February 2025
work page 2025
-
[3]
Christoph Baur, Stefan Denner, Benedikt Wiestler, Nassir Navab, and Shadi Albarqouni. Autoencoders for unsupervised anomaly segmentation in brain mr images: a comparative study.Medical image analysis, 69:101952, 2021. 7 MIAS-SAM: Medical Image Anomaly Segmentation without thresholding. A PREPRINT
work page 2021
-
[4]
Diffusion models with implicit guidanceformedicalanomalydetection
Cosmin I Bercea, Benedikt Wiestler, Daniel Rueckert, and Julia A Schnabel. Diffusion models with implicit guidanceformedicalanomalydetection. InInternationalConferenceonMedicalImageComputingandComputer- Assisted Intervention, pages 211–220. Springer, 2024
work page 2024
-
[5]
Cosmin I Bercea, Philippe C Cattin, Julia A Schnabel, and Julia Wolleb. Denoising diffusion models for anomaly localization in medical images.arXiv preprint arXiv:2410.23834, 2024
-
[6]
Bmad: Benchmarks for medical anomaly detection
Jinan Bao, Hanshi Sun, Hanqiu Deng, Yinsheng He, Zhaoxiang Zhang, and Xingyu Li. Bmad: Benchmarks for medical anomaly detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4042–4053, 2024
2024
-
[7]
Leveraging the mahalanobis distance to enhance unsupervised brain mri anomaly detection
Finn Behrendt, Debayan Bhattacharya, Robin Mieling, Lennart Maack, Julia Krüger, Roland Opfer, and Alexander Schlaefer. Leveraging the mahalanobis distance to enhance unsupervised brain mri anomaly detection. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 394–404. Springer, 2024
work page 2024
-
[8]
Loris-weakly-supervised anomaly detection for ultrasound images
Marco Colussi, Dragan Ahmetovic, Gabriele Civitarese, Claudio Bettini, Aiman Solyman, Roberta Gualtierotti, Flora Peyvandi, and Sergio Mascetti. Loris-weakly-supervised anomaly detection for ultrasound images. In International Workshop on Advances in Simplifying Medical Ultrasound, pages 198–208. Springer, 2024
work page 2024
Show all 30 references
-
[9]
Towards total recall in industrial anomaly detection
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14318–14328, 2022
2022
-
[10]
An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:...
2010 arXiv
-
[11]
Clipsam: Clipandsamcollaboration for zero-shot anomaly segmentation.Neurocomputing, 618:129122, 2025
ShengzeLi,JianjianCao,PengYe,YuhanDing,ChongjunTu,andTaoChen. Clipsam: Clipandsamcollaboration for zero-shot anomaly segmentation.Neurocomputing, 618:129122, 2025
2025
-
[12]
Fade: Few-shot/zero-shot anomaly detection engine using large vision-language model
Yuanwei Li, Elizaveta Ivanova, and Martins Bruveris. Fade: Few-shot/zero-shot anomaly detection engine using large vision-language model. In35th British Machine Vision Conference 2024, BMVC 2024, Glasgow, UK, November 25-28, 2024. BMVA, 2024
2024
-
[13]
Cutpaste: Self-supervised learning for anomaly detectionandlocalization
Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detectionandlocalization. InProceedingsoftheIEEE/CVFconferenceoncomputervisionandpatternrecognition, pages 9664–9674, 2021
2021
-
[14]
Simplenet: A simple network for image anomaly de- tectionandlocalization
Zhikang Liu, Yiming Zhou, Yuansheng Xu, and Zilei Wang. Simplenet: A simple network for image anomaly de- tectionandlocalization. InProceedingsoftheIEEE/CVFConferenceonComputerVisionandPatternRecognition, pages 20402–20411, 2023
2023
-
[15]
Recontrast: Domain-specific anomaly detection via contrastive reconstruction.Advances in Neural Information Processing Systems, 36, 2024
Jia Guo, Lize Jia, Weihang Zhang, Huiqi Li, et al. Recontrast: Domain-specific anomaly detection via contrastive reconstruction.Advances in Neural Information Processing Systems, 36, 2024
2024
-
[16]
Cfa: Coupled-hypersphere-based feature adaptation for target-oriented anomaly localization.IEEE Access, 10:78446–78454, 2022
Sungwook Lee, Seunghyun Lee, and Byung Cheol Song. Cfa: Coupled-hypersphere-based feature adaptation for target-oriented anomaly localization.IEEE Access, 10:78446–78454, 2022
2022
-
[17]
Anomalydetectionviareversedistillationfromone-classembedding
HanqiuDengandXingyuLi. Anomalydetectionviareversedistillationfromone-classembedding. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9737–9746, 2022
2022
-
[18]
Cflow-ad: Real-time unsupervised anomaly detection with localizationviaconditionalnormalizingflows
Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. Cflow-ad: Real-time unsupervised anomaly detection with localizationviaconditionalnormalizingflows. InProceedingsoftheIEEE/CVFwinterconferenceonapplications of computer vision, pages 98–107, 2022
2022
-
[19]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead,AlexanderCBerg,Wan-YenLo,etal.Segmentanything.InProceedingsoftheIEEE/CVFinternational conference on computer vision, pages 4015–4026, 2023
2023
-
[20]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...
2021
-
[21]
Winclip: Zero-/few-shotanomalyclassificationandsegmentation
Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. Winclip: Zero-/few-shotanomalyclassificationandsegmentation. InProceedingsoftheIEEE/CVFConferenceonComputer Vision and Pattern Recognition, pages 19606–19616, 2023
2023
-
[22]
Promptad: Zero-shot anomaly detection using text prompts
Yiting Li, Adam Goodge, Fayao Liu, and Chuan-Sheng Foo. Promptad: Zero-shot anomaly detection using text prompts. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1093–1102, 2024. 8 MIAS-SAM: Medical Image Anomaly Segmentation without t...
2024
-
[23]
Adaclip: Adaptingclipwithhybridlearnablepromptsforzero-shotanomalydetection
Yunkang Cao, Jiangning Zhang, Luca Frittoli, Yuqi Cheng, Weiming Shen, and Giacomo Boracchi. Adaclip: Adaptingclipwithhybridlearnablepromptsforzero-shotanomalydetection. InEuropeanConferenceonComputer Vision, pages 55–72. Springer, 2024
2024
-
[24]
Position-guided prompt learning for anomaly detection in chest x-rays
Zhichao Sun, Yuliang Gu, Yepeng Liu, Zerui Zhang, Zhou Zhao, and Yongchao Xu. Position-guided prompt learning for anomaly detection in chest x-rays. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 567–577. Springer, 2024
2024
-
[25]
The faiss library.arXiv preprint arXiv:2401.08281, 2024
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library.arXiv preprint arXiv:2401.08281, 2024
2024 arXiv
-
[26]
The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification.arXiv preprint arXiv:2107.02314, 2021
Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Kalpathy-Cramer, Felipe C Kitamura, Sarthak Pati, et al. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification.a...
2021 arXiv
-
[27]
The liver tumor segmentation benchmark (lits).Medical Image Analysis, 84:102680, 2023
Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, et al. The liver tumor segmentation benchmark (lits).Medical Image Analysis, 84:102680, 2023
2023
-
[28]
Miccai multi-atlas labelingbeyondthecranialvault–workshopandchallenge
Bennett Landman, Zhoubing Xu, J Igelsias, Martin Styner, Thomas Langerak, and Arno Klein. Miccai multi-atlas labelingbeyondthecranialvault–workshopandchallenge. InProc.MICCAIMulti-AtlasLabelingBeyondCranial Vault—Workshop Challenge, volume 5, page 12, 2015
2015
-
[29]
Automated segmentation of macular edema in oct using deep neural networks.Medical image analysis, 55:216–227, 2019
Junjie Hu, Yuanyuan Chen, and Zhang Yi. Automated segmentation of macular edema in oct using deep neural networks.Medical image analysis, 55:216–227, 2019
2019
-
[30]
Segment anything in medical images.Nature Communications, 15(1):654, 2024
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images.Nature Communications, 15(1):654, 2024. 9
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.