REVIEW 4 major objections 5 minor 35 references
Quest for a clinically relevant medical image segmentation metric: the definition and implementation of Medical Similarity Index
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper introduces the Medical Similarity Index (MSI), a per-point average of Gaussian-weighted bidirectional local distances that lets users penalize inside versus outside segmentation errors differently.
desk verdict A useful, open pipeline for a clinical segmentation metric, but the core formula breaks at perfect agreement and the prostate example is partly circular. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the signed bidirectional local distance $\mathrm{BLD}^{\pm}(p,R)$, which for each test point combines the closest distance from $p$ to the reference contour with the largest distance from a reference point to the test contour, with sign indicating whether $p$ is inside or outside $R$. Combined with the weight function $\mathrm{WF}(d,l)=\exp(-d^2/(2(10/l)^2))$ and the inside/outside penalty levels $i_l$ and $o_l$, it turns local contour error into a score between 0 and 1; averaging those scores gives MSI. The $l$ parameter acts as a length-scale control, so the same formula can penalize small deviations severely or tolerate them, depending on the clinical task.
What would settle it
Run the released implementation on a contour identical to its reference. If the result is anything other than 1, or if moving a single test point onto the reference boundary causes a discontinuous jump from a positive score to zero, the boundary-case ambiguity in Eq. 10 is confirmed and the metric is not well defined at perfect agreement.
Extended reading notes
Core claim
The central claim is that clinical usefulness of a segmentation can be measured by MSI, defined as the average over all test-contour points of a Gaussian weighted score on a bidirectional local distance. For each test point $p$ and reference contour $R$, the score is $\mathrm{WF}(\mathrm{BLD}^{\pm}(p,R), i_l)$ if $p$ lies inside $R$ and $\mathrm{WF}(\mathrm{BLD}^{\pm}(p,R), o_l)$ if $p$ lies outside $R$, where $\mathrm{WF}(d,l)=\exp(-d^2/(2(10/l)^2))$ and $\mathrm{BLD}^{\pm}$ is the signed maximum of the forward minimal distance and backward maximal distance. MSI is the mean of these per-point scores after aligning the test and reference centers of mass. Because $i_l$ and $o_l$ are independe
Load-bearing premise
The metric's behavior for test points lying exactly on the reference contour is unresolved: the prose says their score is zero, but Eq. 10 excludes them from the average, so an exactly matching contour would score 0 instead of 1 unless the implementation silently adds a boundary rule.
Editorial extensions
If this is right
- If MSI is accepted, a low MSI together with high Dice and Jaccard can serve as a red flag for clinically unacceptable deviations, as in the prostate bladder-reaching slice.
- Because $i_l$ and $o_l$ are user-set, the same metric can be tuned to favor conservative tumor-volume estimates or organ-at-risk protection without changing the formula.
- The open pipeline extends MSI beyond single contours to multi-slice, multi-mask images, with automatic contour pairing and a concave-mask splitting algorithm.
- Slices where automatic pairing fails are assigned MSI zero by default, making them explicit quality failures rather than silently contributing optimistic scores.
Reading between the lines
- The paper leaves implicit that a calibrated MSI threshold could become a practical acceptance rule for auto-segmentation in a given clinical workflow; this would require collecting MSI values across many expert-approved and expert-rejected cases.
- The asymmetric inside/outside penalty structure generalizes naturally to a spatially varying risk map: instead of constant $i_l$ and $o_l$, one could use local penalties derived from dose distributions or organ proximity.
- A user should test the boundary convention before relying on the released code: because Eq. 10 excludes points lying exactly on the reference contour, an exact match could score 0 unless the implementation adds an explicit boundary rule.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a new segmentation evaluation metric, the Medical Similarity Index (MSI), intended to be clinically adaptable via user-selected inside/outside penalty levels. The metric is defined as an average of Gaussian-weighted bidirectional local distances between test and reference contours. The authors provide an open-source Python pipeline with contour pairing and mask splitting, and demonstrate the workflow on fibroid and prostate MRI datasets, claiming that MSI can flag a clinically unacceptable prostate segmentation that Dice, Jaccard, and average Hausdorff distance do not.
Significance. If the metric were well defined and validated, the idea of a tunable, clinically oriented similarity index would be useful, and the open-source pipeline, Colab notebooks, and mask-splitting utilities are practical contributions. However, the central mathematical definition has a load-bearing inconsistency at boundary/intersection points, and the clinical demonstration is partly self-confirming because the penalty parameters are chosen to encode the conclusion. The current manuscript therefore does not support the claim that MSI is a reliable or clinically relevant metric.
major comments (4)
- [Section 2.2, Eqs. (8)-(10)] The definition of MSI is internally inconsistent for points lying on the reference contour. I(R) is defined as the set of points strictly inside R, and O(R) is defined as I(R)\R (with I(R) the complement of I(R)); thus points on R belong to neither set. For a perfect match T=R, both sums in Eq. (10) are empty. The sentence after Eq. (10) says the MCF score of intersection points is zero, which would make MSI=0 for exact agreement. A similarity metric that assigns total dissimilarity to a perfect segmentation cannot support the paper's central claim. This is not a minor technicality: exact agreement is the anchor case for any similarity index.
- [Section 2.2, Eq. (3)] BMaxD is defined as a maximum over the set of reference points p_r such that ||p_r - p_test|| = dmin(p_r,T). For a test point that is not the nearest test point for any reference point, this set is empty and BMaxD is undefined. No convention is given for this case, yet BLD and hence MCF depend on BMaxD. The metric is therefore not defined for all test points in a discrete contour. A default (e.g., BMaxD=0 when the set is empty) or an explicit exclusion rule is needed.
- [Section 2.2 and Fig. 4] The relationship between the formal definition and the implemented pipeline is unclear. The formal metric uses BLD as the distance argument in WF (Eqs. 4-10), but the pipeline described in Fig. 4 steps 3-8 aligns the center of mass, computes BLD for pairing, moves the test contour back, and then calculates 'distances between the paired points in the original location' before scoring. If the reported MSI values are based on those original-location paired distances, they do not implement Eq. (10). The paper must reconcile the definition with the actual computation.
- [Section 3.4, Fig. 13] The demonstration that 'only the MSI characterizes the segmentation correctly' is not independent. The authors select il=1, ol=10, a parameter combination that encodes a strong penalty for outer deviation, and then show that MSI is low when an outer deviation exists. This shows tunability, but it does not establish clinical relevance or superiority over Dice/Jaccard/Hausdorff. A prespecified parameter choice, sensitivity analysis across the full test set, and a clear clinical endpoint or decision rule would be needed to support the claim.
minor comments (5)
- [Section 2.2, Eq. (9)] The notation O(R) = I(R)\R with the parenthetical 'where I(R) denotes the complement of I(R)' is circular and confusing. Since I(R) was just defined as the interior, the complement notation needs to be stated explicitly and consistently.
- [Throughout] There are numerous typographical errors, including 'healty' (Background), 'the the', 'uncertainity', 'intellgience', and 'clnical'. These should be corrected in a thorough language edit.
- [Tables] Table 1 appears twice: once for MSI hyperparameter values in Section 3.3 and once in the Appendix. The appendix table should be renumbered to avoid ambiguity.
- [Section 4] The Discussion states that slices requiring manual intervention 'receive MSI score of zero' by default, but this rule is not described in Section 2.2 or in the MSI definition. Clarify whether this is part of the metric or an implementation choice.
- [Fig. 4] The flowchart indicates that the test contour is moved back to its original location before distance calculation, but the formal definition in Section 2.2 does not mention this alignment step. Please state explicitly which coordinate frame is used in Eq. (10) and in the reported MSI values.
Circularity Check
Clinical validation is tuned: ol=10 is chosen to penalize the known outer error and then MSI is said to uniquely flag it; Eq. 10's boundary exclusion makes MSI=0 for perfect match.
-
fitted input called prediction
[Section 3.4, Fig. 13 and surrounding text]
"The impact of the outer deviation is crucial in the current clinical application, that is why we need the desired metric to represent the segmentation defects which lays out of the reference segmentation. ... The MSI value was calculated using il= 1, ol= 10 hyperparameters. ... Only the MSI characterizes the segmentation correctly."
The outer penalty level ol=10 is chosen precisely because the known error is an outer deviation reaching the bladder; the low MSI (0.403) is then a direct mathematical consequence of that user-set parameter in Eq. 7/10. The conclusion that 'Only the MSI characterizes the segmentation correctly' is therefore a restatement of the input preference rather than an independent finding. No external clinical outcome is used to validate MSI; the desired property (penalizing outer deviation) is encoded in the chosen weight and read back out as evidence of clinical relevance.
-
self definitional
[Section 2.2, Eqs. (8)-(10)]
"O(R) = I(R)\R, where I(R) denotes the complement of I(R) ... if the reference and test contours intersect, the MCF score of the intersection points is zero."
Equation (10) sums only over p∈T∩I(R) and p∈T∩O(R). By Eqs. (8)-(9), a point lying exactly on R belongs to neither set. The quoted sentence assigns zero score to intersection points. When T=R (perfect segmentation), every test point is an intersection/boundary point, so both sums are empty (or contribute zero) and MSI=0 by construction. Thus the metric's own definition assigns the worst possible value to exact agreement, contradicting the paper's claim that MSI is a similarity index that 'characterizes the segmentation correctly.' This is a self-definitional failure of the central formula, not a missing technical detail.
full rationale
The metric's construction (Eqs. 1-10) is self-contained and builds on external prior work [15]; the self-citations (Refs. 6 and 8) are background and code, not load-bearing. No uniqueness theorem or smuggled ansatz is involved. However, the key clinical demonstration in Sec. 3.4 is circular: the authors set ol=10 because the known failure is an outer deviation, so MSI's low value is forced by the chosen penalty level, and the statement 'Only the MSI characterizes the segmentation correctly' merely reads the input preference back out. In addition, Eqs. (8)-(10) exclude boundary points from both sums, and the prose assigns zero score to intersection points; for a coincident test and reference contour, every test point is on the boundary, so MSI=0 by definition, making the metric fail at the most important case. These two issues affect the central claim of clinical relevance, though the pipeline and open-source code are useful contributions. Score 6 reflects partial circularity in the validation demonstration plus a self-definitional flaw in the metric's perfect-match behavior.
Assumptions & free parameters
free parameters (6)
- il (inside penalty level) =
1, 5, 10 in examples
- ol (outside penalty level) =
1, 5, 10 in examples
- Scale factor 10 in WF(d,l) =
10
- MIN_AREA threshold =
not specified
- Area/convex hull ratio threshold =
1.2 (default)
- Convexity defect length threshold =
0.5 * max length
assumptions (5)
- domain assumption Discrete point sets on 2D image boundaries adequately represent contours for clinical evaluation.
- domain assumption Euclidean distance is the appropriate geometric measure of segmentation error.
- domain assumption The Gaussian weight function WF(d,l) is an appropriate mapping from boundary distance to clinical similarity score.
- domain assumption Bidirectional local distance (BLD) captures clinically relevant contour deviation.
- domain assumption Closest center-of-mass pairing correctly matches corresponding reference and test contours.
Cite this review
Pith. "Pith review of Quest for a clinically relevant medical image segmentation metric: the definition and implementation of Medical Similarity Index." pith.science (2026). https://pith.science/paper/HA7CT3V3
@misc{pith2026250809722,
author = {Pith},
title = {Pith review of: Quest for a clinically relevant medical image segmentation metric: the definition and implementation of Medical Similarity Index},
year = {2026},
howpublished = {\url{https://pith.science/paper/HA7CT3V3}},
note = {Machine review of arXiv:2508.09722}
}
read the original abstract
Background: In the field of radiology and radiotherapy, accurate delineation of tissues and organs plays a crucial role in both diagnostics and therapeutics. While the gold standard remains expert-driven manual segmentation, many automatic segmentation methods are emerging. The evaluation of these methods primarily relies on traditional metrics that only incorporate geometrical properties and fail to adapt to various applications. Aims: This study aims to develop and implement a clinically relevant segmentation metric that can be adapted for use in various medical imaging applications. Methods: Bidirectional local distance was defined, and the points of the test contour were paired with points of the reference contour. After correcting for the distance between the test and reference center of mass, Euclidean distance was calculated between the paired points, and a score was given to each test point. The overall medical similarity index was calculated as the average score across all the test points. For demonstration, we used myoma and prostate datasets; nnUNet neural networks were trained for segmentation. Results: An easy-to-use, sustainable image processing pipeline was created using Python. The code is available in a public GitHub repository along with Google Colaboratory notebooks. The algorithm can handle multislice images with multiple masks per slice. Mask splitting algorithm is also provided that can separate the concave masks. We demonstrate the adaptability with prostate segmentation evaluation. Conclusions: A novel segmentation evaluation metric was implemented, and an open-access image processing pipeline was also provided, which can be easily used for automatic measurement of clinical relevance of medical image segmentation.}
Reference graph
Works this paper leans on
-
[15]
Quantitative evaluation of image segmentation incorporating medical consideration functions
Kim H, Monroe JI, Lo S, Yao M, Harari PM, Machtay M, et al. Quantitative evaluation of image segmentation incorporating medical consideration functions. Medical physics. 2015;42(6Part1):3013–3023
work page 2015
-
[1]
Sherer MV, Lin D, Elguindi S, Duke S, Tan LT, Cacicedo J, et al. Metrics to evaluate the performance of auto-segmentation for radiation treatment planning: A critical review. Radiotherapy and Oncology. 2021;160:185–191. https://doi. org/10.1016/j.radonc.2021.05.003
-
[2]
Segedin B, Petric P. Uncertainties in target volume delineation in radiother- apy—are they relevant and what can we do about them? Radiology and Oncology. 2016;50(3):254–262. https://doi.org/10.1515/raon-2016-0023
-
[3]
Peters LJ, O’Sullivan B, Giralt J, Fitzgerald TJ, Trotti A, Bernier J, et al. Crit- ical impact of radiotherapy protocol compliance and quality in the treatment of advanced head and neck cancer: results from TROG 02.02. Journal of Clinical Oncology. 2010;28(18):2996–3001. https://doi.org/10.1200/JCO.2009.27.4498. 19
-
[4]
Buelens P, Willems S, Vandewinckele L, Crijns W, Maes F, Weltens C. Clinical evaluation of a deep learning model for segmentation of target volumes in breast cancer radiotherapy. Radiotherapy and Oncology. 2022 June;171:84–90. https: //doi.org/10.1016/j.radonc.2022.04.015
-
[5]
Validation of image segmentation and regis- tration algorithms
Warfield SK, Zou KH, Wells WM. Validation of image segmentation and regis- tration algorithms. Statistics in Medicine. 2004;23(2):311–329. https://doi.org/ 10.1002/sim.1723
-
[6]
Fazekas S, Budai BK, Stollmayer R, Kaposi PN, B´ erczi V. Artificial intelligence and neural networks in radiology – Basics that all radiology residents should know. Imaging. 2022;14(2):73 – 81. https://doi.org/10.1556/1647.2022.00104
-
[7]
The predictive value of segmentation metrics on dosimetry in organs at risk of the brain
Poel R, R¨ ufenacht E, Hermann E, Scheib S, Manser P, Aebersold DM, et al. The predictive value of segmentation metrics on dosimetry in organs at risk of the brain. Medical Image Analysis. 2021 October;73:102161. https://doi.org/10. 1016/j.media.2021.102161
Show all 35 references
-
[8]
Accessed: 2025-06-21
Fazekas S.: bld: Bidirectional Local Distance in Python. Accessed: 2025-06-21. https://github.com/szuzina/bld
2025
-
[9]
nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation
Isensee F, Jaeger PF, Kohl SAA, Petersen J, Maier-Hein KH. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods. 2021;18:203–211. https://doi.org/10.1038/s41592-020-01008-z
2021 doi
-
[10]
Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space
Liu Q, Chen C, Qin J, Dou Q, Heng PA. Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition; 2021. p. 1013–1023
2021
-
[11]
Evaluation of Multi- modal Algorithms for the Segmentation of Multiparametric MRI Prostate Images
Nai YH, Teo BW, Tan NL, Tan CH, Lim TP, Poh CL. Evaluation of Multi- modal Algorithms for the Segmentation of Multiparametric MRI Prostate Images. Journal of Healthcare Engineering. 2020;2020:8861035. https://doi.org/10.1155/ 2020/8861035
2020
-
[12]
NCI-ISBI 2013 challenge: automated segmentation of prostate structures
Bloch N, Madabhushi A, Huisman H, Freymann J, Kirby J, Grauer M, et al. NCI-ISBI 2013 challenge: automated segmentation of prostate structures. The Cancer Imaging Archive. 2015;370(6):5
2013
-
[13]
Computer-aided detection and diagnosis for prostate cancer based on mono and multi-parametric MRI: a review
Lema ˆ ıtre G, Mart ´ ı R, Freixenet J, Vilanova JC, Walker PM, Meriaudeau F. Computer-aided detection and diagnosis for prostate cancer based on mono and multi-parametric MRI: a review. Computers in biology and medicine. 2015;60:8–31
2015
-
[14]
Evaluation of prostate segmentation algorithms for MRI: The PROMISE12 chal- lenge
Litjens G, Toth R, van de Ven W, Hoeks C, Kerkstra S, van Ginneken B, et al. Evaluation of prostate segmentation algorithms for MRI: The PROMISE12 chal- lenge. Medical Image Analysis. 2014;18(2):359–373. https://doi.org/10.1016/j. 20 media.2013.12.002
2014 doi
-
[16]
An efficient algorithm for calculating the exact Hausdorff distance
Taha AA, Hanbury A. An efficient algorithm for calculating the exact Hausdorff distance. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2015 Nov;37(11):2153–2163. https://doi.org/10.1109/TPAMI.2015.2408351
2015
-
[17]
Towards a guideline for evaluation metrics in medical image segmentation
M¨ uller D, Soto-Rey I, Kramer F. Towards a guideline for evaluation metrics in medical image segmentation. BMC Research Notes. 2022;15(1). https://doi.org/ 10.1186/s13104-022-06096-y
2022 doi
-
[18]
On the usage of average Hausdorff distance for segmentation performance assess- ment: hidden error when used for ranking
Aydin OU, Taha AA, Hilbert A, Khalil AA, Galinovic I, Fiebach JB, et al. On the usage of average Hausdorff distance for segmentation performance assess- ment: hidden error when used for ranking. European Radiology Experimental. 2021;5(1):4. https://doi.org/10.1186/s41747-020-00200-2
2021 doi
-
[19]
Object detection
Amit Y, Felzenszwalb P, Girshick R. Object detection. In: Computer Vision: A Reference Guide. Springer; 2021. p. 875–883
2021
-
[20]
The OpenCV Library
Bradski G. The OpenCV Library. Dr Dobb’s Journal of Software Tools. 2000
2000
-
[21]
PyTorch: An imperative style, high-performance deep learning library
Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, et al. PyTorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:191201703. 2019
2019
-
[22]
Accessed: 2025- 06-04
American Cancer Society.: Key Statistics for Prostate Cancer. Accessed: 2025- 06-04. Available from: https://www.cancer.org/cancer/types/prostate-cancer/ about/key-statistics.html
2025
-
[23]
Automated prostate multi-regional segmentation in magnetic resonance using fully convolutional neural networks
Jimenez-Pastor A, Lopez-Gonzalez R, Fos-Guarinos B, Garcia-Castro F, Wit- tenberg M, Torregrosa-Andr´ es A, et al. Automated prostate multi-regional segmentation in magnetic resonance using fully convolutional neural networks. European Radiology. 2023 July;33(7):5087–5096. Epu...
2023 doi
-
[24]
Global epidemiological characteristics of uterine fibroids
Li B, Wang F, Chen L, Tong H. Global epidemiological characteristics of uterine fibroids. Archives of Medical Science. 2023;19(6):1802–1810. https://doi.org/10. 5114/aoms/171786
2023
-
[25]
Asymptomatic uterine fibroids
Divakar H. Asymptomatic uterine fibroids. Best Practice & Research Clinical Obstetrics & Gynaecology. 2008 August;22(4):643–654. PMID: 18375184. https: //doi.org/10.1016/j.bpobgyn.2008.01.007. 21
2008 doi
-
[26]
Diagnosis and management of uterine fibroids: current trends and future strategies
Ahmad A, Kumar M, Bhoi NR, Badruddeen, Akhtar J, Khan MI, et al. Diagnosis and management of uterine fibroids: current trends and future strategies. Journal of Basic and Clinical Physiology and Pharmacology. 2023;34(3):291–310. https: //doi.org/doi:10.1515/jbcpp-2022-0219
2023 doi
-
[27]
An Instance Segmentation Model Based on Deep Learning for Intelligent Diagnosis of Uterine Myomas in MRI
Pan H, Zhang M, Bai W, Li B, Wang H, Geng H, et al. An Instance Segmentation Model Based on Deep Learning for Intelligent Diagnosis of Uterine Myomas in MRI. Diagnostics. 2023;13(9). https://doi.org/10.3390/diagnostics13091525
2023 doi
-
[28]
Conceptual review of outcome metrics and measures used in clinical evaluation of artificial intelligence in radiology
Park SH, Han K, Lee JG. Conceptual review of outcome metrics and measures used in clinical evaluation of artificial intelligence in radiology. Radiologia Medica. 2024 November;129(11):1644–1655. Epub 2024 Sep 3. https://doi.org/10.1007/ s11547-024-01886-9
2024
-
[29]
Blood vessel segmentation algorithms - Review of methods, datasets and evaluation metrics
Moccia S, De Momi E, El Hadji S, Mattos LS. Blood vessel segmentation algorithms - Review of methods, datasets and evaluation metrics. Computer Methods and Programs in Biomedicine. 2018 May;158:71–91. Epub 2018 Feb 10. https://doi.org/10.1016/j.cmpb.2018.02.001
2018 doi
-
[30]
Retinal OCT image segmen- tation with deep learning: A review of advances, datasets, and evaluation metrics
Zhang H, Yang B, Li S, Zhang X, Li X, Liu T, et al. Retinal OCT image segmen- tation with deep learning: A review of advances, datasets, and evaluation metrics. Computerized Medical Imaging and Graphics. 2025 July;123:102539. Epub 2025 Apr 4. https://doi.org/10.1016/j.compmedi...
2025
-
[31]
A Review of the Met- rics Used to Assess Auto-Contouring Systems in Radiotherapy
Mackay K, Bernstein D, Glocker B, Kamnitsas K, Taylor A. A Review of the Met- rics Used to Assess Auto-Contouring Systems in Radiotherapy. Clinical Oncology. 2023;35(6):354–369. https://doi.org/https://doi.org/10.1016/j.clon.2023.01.016
2023 doi
-
[32]
A Review of Advancements and Chal- lenges in Liver Segmentation
Wei D, Jiang Y, Zhou X, Wu D, Feng X. A Review of Advancements and Chal- lenges in Liver Segmentation. Journal of Imaging. 2024 August 21;10(8):202. https://doi.org/10.3390/jimaging10080202
2024 doi
-
[33]
van den Oever LB, van Veldhuizen W A, Cornelissen LJ, Spoor DS, Willems TP, Kramer G, et al. Qualitative Evaluation of Common Quantitative Met- rics for Clinical Acceptance of Automatic Segmentation: a Case Study on Heart Contouring from CT Images by Deep Learning Algorithms. ...
2022
-
[34]
Evaluation of Segmentation algorithms for Medical Imaging
Fenster A, Chiu B. Evaluation of Segmentation algorithms for Medical Imaging. In: 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference
2005
-
[2005]
7186–7189
p. 7186–7189. Declarations • Ethics approval and consent to participate 22 This study was approved by the Institutional Review Board (Semmelweis University Regional and Institutional Committee of Science and Research Ethics, SE-RKEB: 172/2022). As this was a retrospective stud...
2022
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.