{"id":"7874fad8-bd7d-4c02-a495-766dde80c065","arxiv_id":"2508.09722","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"MSI is a clinically weighted, bidirectional boundary-distance metric with an open-source pipeline for evaluating medical image segmentations.","lead":"This paper defines a new segmentation metric, MSI, that scores how well an automatic outline matches a clinician's reference, with separate penalties for inside and outside errors. It ships an open-source Python pipeline and Colab notebooks, demonstrated on fibroid and prostate MRI.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 10 has no consistent rule for test points on the reference contour; for a perfect match all test points are boundary/intersection points and MSI evaluates to 0, not 1, so the central metric fails at exact agreement.","rationale":"The reader’s weakest assumption isolates a genuine defect in Eq. 10: the treatment of boundary/intersection points is undefined and, taken literally, makes MSI=0 for T=R. This is the most load-bearing concern because the metric itself is the paper’s central novel contribution; the clinical-relevance demonstration in Sec. 3.4 cannot be interpreted if the metric is not a valid similarity measure. The concern is not about disagreement with consensus or a styling preference; it is an internal inconsistency in the definition. I also note the clinical claim rests on a single selected slice with manually chosen il/ol values, but the boundary defect is sufficient to block acceptance as written. The paper’s code, pipeline, and mask-splitting tool are independently useful and deserve credit, but they do not fix Eq. 10. Since the verdict should remain REJECT, I mark it UNCHANGED.","tokens_in":12847,"tokens_out":7143,"duration_ms":89735,"concrete_test":"Use the released GitHub/Colab code to compute MSI for an identical reference and test mask on one slice (e.g., a prostate or fibroid mask) with il=ol=1. If the output is 1, the code must be using an undocumented boundary convention (e.g., classifying boundary points as inside or special-casing T=R), contradicting Eq. 10; if the output is 0, the identity failure is confirmed. Independently, substitute T=R into Eq. 10 and check whether any p satisfies the membership conditions; none does, giving an empty sum and MSI=0 unless a separate boundary rule is introduced.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The metric’s definition is internally inconsistent in the most important case. Eq. 10 averages MCF_i(p,R) over p∈T∩I(R) and MCF_o(p,R) over p∈T∩O(R), where Eq. 8 defines I(R) as points strictly inside R and Eq. 9 defines O(R) as the complement of I(R). A test point lying exactly on R is therefore in neither summand. The sentence after Eq. 10 then says that if the contours intersect, the MCF score of the intersection points is zero. When T=R, every test point is an intersection point and lies on the boundary; the sum is empty (or, under the stated zero rule, contributes zero), so MSI=0 for a perfect segmentation. A similarity index that scores exact agreement as total dissimilarity cannot support the claim that MSI is clinically meaningful or that 'only MSI characterizes the segmentation correctly' (Sec. 3.4). The issue is not a missing technicality: it changes the value of the metric precisely at the point where all other metrics are anchored. The open-source pipeline and mask splitting are useful, but they do not repair the definitional gap in the central contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new segmentation evaluation metric, the Medical Similarity Index (MSI), intended to be clinically adaptable via user-selected inside/outside penalty levels. The metric is defined as an average of Gaussian-weighted bidirectional local distances between test and reference contours. The authors provide an open-source Python pipeline with contour pairing and mask splitting, and demonstrate the workflow on fibroid and prostate MRI datasets, claiming that MSI can flag a clinically unacceptable prostate segmentation that Dice, Jaccard, and average Hausdorff distance do not.","tokens_in":13178,"tokens_out":4515,"duration_ms":49755,"significance":"If the metric were well defined and validated, the idea of a tunable, clinically oriented similarity index would be useful, and the open-source pipeline, Colab notebooks, and mask-splitting utilities are practical contributions. However, the central mathematical definition has a load-bearing inconsistency at boundary/intersection points, and the clinical demonstration is partly self-confirming because the penalty parameters are chosen to encode the conclusion. The current manuscript therefore does not support the claim that MSI is a reliable or clinically relevant metric.","major_comments":[{"comment":"The definition of MSI is internally inconsistent for points lying on the reference contour. I(R) is defined as the set of points strictly inside R, and O(R) is defined as I(R)\\R (with I(R) the complement of I(R)); thus points on R belong to neither set. For a perfect match T=R, both sums in Eq. (10) are empty. The sentence after Eq. (10) says the MCF score of intersection points is zero, which would make MSI=0 for exact agreement. A similarity metric that assigns total dissimilarity to a perfect segmentation cannot support the paper's central claim. This is not a minor technicality: exact agreement is the anchor case for any similarity index.","section":"Section 2.2, Eqs. (8)-(10)"},{"comment":"BMaxD is defined as a maximum over the set of reference points p_r such that ||p_r - p_test|| = dmin(p_r,T). For a test point that is not the nearest test point for any reference point, this set is empty and BMaxD is undefined. No convention is given for this case, yet BLD and hence MCF depend on BMaxD. The metric is therefore not defined for all test points in a discrete contour. A default (e.g., BMaxD=0 when the set is empty) or an explicit exclusion rule is needed.","section":"Section 2.2, Eq. (3)"},{"comment":"The relationship between the formal definition and the implemented pipeline is unclear. The formal metric uses BLD as the distance argument in WF (Eqs. 4-10), but the pipeline described in Fig. 4 steps 3-8 aligns the center of mass, computes BLD for pairing, moves the test contour back, and then calculates 'distances between the paired points in the original location' before scoring. If the reported MSI values are based on those original-location paired distances, they do not implement Eq. (10). The paper must reconcile the definition with the actual computation.","section":"Section 2.2 and Fig. 4"},{"comment":"The demonstration that 'only the MSI characterizes the segmentation correctly' is not independent. The authors select il=1, ol=10, a parameter combination that encodes a strong penalty for outer deviation, and then show that MSI is low when an outer deviation exists. This shows tunability, but it does not establish clinical relevance or superiority over Dice/Jaccard/Hausdorff. A prespecified parameter choice, sensitivity analysis across the full test set, and a clear clinical endpoint or decision rule would be needed to support the claim.","section":"Section 3.4, Fig. 13"}],"minor_comments":[{"comment":"The notation O(R) = I(R)\\R with the parenthetical 'where I(R) denotes the complement of I(R)' is circular and confusing. Since I(R) was just defined as the interior, the complement notation needs to be stated explicitly and consistently.","section":"Section 2.2, Eq. (9)"},{"comment":"There are numerous typographical errors, including 'healty' (Background), 'the the', 'uncertainity', 'intellgience', and 'clnical'. These should be corrected in a thorough language edit.","section":"Throughout"},{"comment":"Table 1 appears twice: once for MSI hyperparameter values in Section 3.3 and once in the Appendix. The appendix table should be renumbered to avoid ambiguity.","section":"Tables"},{"comment":"The Discussion states that slices requiring manual intervention 'receive MSI score of zero' by default, but this rule is not described in Section 2.2 or in the MSI definition. Clarify whether this is part of the metric or an implementation choice.","section":"Section 4"},{"comment":"The flowchart indicates that the test contour is moved back to its original location before distance calculation, but the formal definition in Section 2.2 does not mention this alignment step. Please state explicitly which coordinate frame is used in Eq. (10) and in the reported MSI values.","section":"Fig. 4"}],"recommendation":"reject","confidential_remarks":"The paper's open-source pipeline is useful, but the central metric as defined fails at exact agreement and the implementation appears to diverge from the stated formula. These are load-bearing issues for the claimed contribution. The clinical demonstration is also post hoc in its parameter choice. I do not see a straightforward local revision that would make the central claim defensible without redefining the metric and re-running the evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [colleague],\n\nQuick take: the paper packages a plausible idea — a bidirectional local distance score with user-tunable inside/outside penalties — into a clean, open Python pipeline. That is genuinely useful. But the central formula has a boundary-case bug that is not a technicality: for a perfect match, the metric evaluates to 0, not 1, because test points lying exactly on the reference contour are excluded from both sums in Eq. 10. That breaks the metric at the exact point where every other segmentation metric is anchored.\n\nWhat is new: MSI is a specific combination of Kim et al.'s bidirectional local distance and medical consideration functions, with a Gaussian weight and a center-of-mass alignment step. The paper cites [15] for the building blocks, so the novelty claim is honest. The code, the Google Colab notebook, and the mask-splitting/contour-pairing utilities are real contributions; someone evaluating radiotherapy contours could adapt them.\n\nWhat is not solid: first, the boundary case. Eq. 8 defines I(R) as strictly inside, and Eq. 9 defines O(R) as the complement minus the boundary. Points on R are in neither set. The sentence after Eq. 10 says intersection points get score zero. If T=R, every test point is on the boundary, so the sums are empty (or all zeros) and MSI=0. The paper never defines a rule for this case, and the text even garbles the O(R) definition. This is a load-bearing flaw: the index cannot distinguish a perfect segmentation from a completely unacceptable one without an implicit patch that is not stated.\n\nSecond, the clinical validation is thin. The prostate example in Sec. 3.4 uses ol=10, which is precisely the parameter that encodes a preference against outer deviation, and then shows that MSI, with that parameter, penalizes outer deviation. That is circular; it demonstrates that the metric implements the chosen penalty, not that the penalty is clinically predictive. The appendix adds slices but still only one patient. No systematic study crosses different penalty levels or independent clinical endpoints.\n\nThe citation pattern is honest; the paper repeatedly credits Kim et al. and other prior work. The code is reproducible, which I weigh in the paper's favor.\n\nBottom line: the pipeline and the metric idea deserve attention, but the definitional bug and the circular demonstration are serious. I would encourage the authors to fix the boundary rule and validate against multiple clinicians or dosimetric outcomes. As is, I would not reject the paper out of hand — the core idea is salvageable — but the current version should not be accepted without those repairs. A serious editor could send it to review, with the expectation of major revisions.","headline":"A useful, open pipeline for a clinical segmentation metric, but the core formula breaks at perfect agreement and the prostate example is partly circular.","tokens_in":13660,"tokens_out":4206,"would_cite":false,"duration_ms":42799,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces the Medical Similarity Index (MSI), a per-point average of Gaussian-weighted bidirectional local distances that lets users penalize inside versus outside segmentation errors differently.","keywords":["medical image segmentation","segmentation evaluation metric","Medical Similarity Index","bidirectional local distance","Dice score","Hausdorff distance","clinical relevance","prostate MRI"],"falsifier":"Run the released implementation on a contour identical to its reference. If the result is anything other than 1, or if moving a single test point onto the reference boundary causes a discontinuous jump from a positive score to zero, the boundary-case ambiguity in Eq. 10 is confirmed and the metric is not well defined at perfect agreement.","tokens_in":12780,"feed_emoji":"🩻","tokens_out":8497,"duration_ms":84390,"temperature":0.7,"pith_summary":"The paper proposes Medical Similarity Index (MSI), a clinically oriented replacement for overlap and distance metrics in medical image segmentation. MSI averages, over every point of the automatic test contour, a Gaussian score applied to a bidirectional local distance to the expert reference contour, with separate user-set penalties for errors inside versus outside the reference. The authors argue this lets the metric reflect clinical risk: an outward error that reaches the bladder can be scored harshly even when Dice and Jaccard remain high. They demonstrate on a prostate MRI slice where Dice 0.939 and Jaccard 0.886 look good but MSI 0.403 correctly marks the segmentation as unacceptable. The paper also releases a Python pipeline with contour pairing and concave-mask splitting so the metric can be applied to multi-slice, multi-mask studies.","feed_headline":"MSI catches bladder-risking prostate overgrowth that Dice misses","feed_subtitle":"New metric scores every boundary point with separate inside/outside penalties, turning clinical risk into a number.","key_machinery":"The central object is the signed bidirectional local distance $\\mathrm{BLD}^{\\pm}(p,R)$, which for each test point combines the closest distance from $p$ to the reference contour with the largest distance from a reference point to the test contour, with sign indicating whether $p$ is inside or outside $R$. Combined with the weight function $\\mathrm{WF}(d,l)=\\exp(-d^2/(2(10/l)^2))$ and the inside/outside penalty levels $i_l$ and $o_l$, it turns local contour error into a score between 0 and 1; averaging those scores gives MSI. The $l$ parameter acts as a length-scale control, so the same formula can penalize small deviations severely or tolerate them, depending on the clinical task.","core_discovery":"The central claim is that clinical usefulness of a segmentation can be measured by MSI, defined as the average over all test-contour points of a Gaussian weighted score on a bidirectional local distance. For each test point $p$ and reference contour $R$, the score is $\\mathrm{WF}(\\mathrm{BLD}^{\\pm}(p,R), i_l)$ if $p$ lies inside $R$ and $\\mathrm{WF}(\\mathrm{BLD}^{\\pm}(p,R), o_l)$ if $p$ lies outside $R$, where $\\mathrm{WF}(d,l)=\\exp(-d^2/(2(10/l)^2))$ and $\\mathrm{BLD}^{\\pm}$ is the signed maximum of the forward minimal distance and backward maximal distance. MSI is the mean of these per-point scores after aligning the test and reference centers of mass. Because $i_l$ and $o_l$ are independe","pith_inferences":["The paper leaves implicit that a calibrated MSI threshold could become a practical acceptance rule for auto-segmentation in a given clinical workflow; this would require collecting MSI values across many expert-approved and expert-rejected cases.","The asymmetric inside/outside penalty structure generalizes naturally to a spatially varying risk map: instead of constant $i_l$ and $o_l$, one could use local penalties derived from dose distributions or organ proximity.","A user should test the boundary convention before relying on the released code: because Eq. 10 excludes points lying exactly on the reference contour, an exact match could score 0 unless the implementation adds an explicit boundary rule."],"forward_implications":["If MSI is accepted, a low MSI together with high Dice and Jaccard can serve as a red flag for clinically unacceptable deviations, as in the prostate bladder-reaching slice.","Because $i_l$ and $o_l$ are user-set, the same metric can be tuned to favor conservative tumor-volume estimates or organ-at-risk protection without changing the formula.","The open pipeline extends MSI beyond single contours to multi-slice, multi-mask images, with automatic contour pairing and a concave-mask splitting algorithm.","Slices where automatic pairing fails are assigned MSI zero by default, making them explicit quality failures rather than silently contributing optimistic scores."],"supporting_citations":[{"why":"Supplies the bidirectional local distance and medical consideration functions that MSI adapts into its scoring formula.","marker":"[15]"},{"why":"Shows that 23 standard segmentation metrics have limited predictive value for treatment quality, motivating a clinically oriented metric.","marker":"[7]"},{"why":"Provides the nnU-Net framework whose un-fine-tuned outputs are used to stress-test the pipeline with many segmentation errors.","marker":"[9]"},{"why":"Supplies the public prostate MRI dataset used to demonstrate that MSI flags a bladder-reaching outer deviation.","marker":"[10]"},{"why":"Documents pitfalls in metric implementation and usage in medical image segmentation, supporting the need for an explicit deterministic pipeline.","marker":"[17]"},{"why":"Reviews metrics for auto-segmentation evaluation and frames the clinical evaluation gap the paper addresses.","marker":"[1]"}],"fun_headline_variants":["MSI catches prostate overgrowth that Dice misses","New metric MSI adapts to clinical imaging needs","Prostate overgrowth caught by MSI, not Dice","MSI: a clinically relevant segmentation metric","MSI turns clinical risk into a measurable score"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The metric's behavior for test points lying exactly on the reference contour is unresolved: the prose says their score is zero, but Eq. 10 excludes them from the average, so an exactly matching contour would score 0 instead of 1 unless the implementation silently adds a boundary rule.","fun_headline_variants_meta":{"raw":{"variants":["MSI catches prostate overgrowth that Dice misses","New metric MSI adapts to clinical imaging needs","Prostate overgrowth caught by MSI, not Dice","MSI: a clinically relevant segmentation metric","MSI turns clinical risk into a measurable score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000358,"raw_usage":{"total_tokens":1831,"prompt_tokens":853,"completion_tokens":978,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":905}},"tokens_in":597,"tokens_out":978,"duration_ms":10315,"temperature":1.0,"reasoning_tokens":905,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:51:17.040841+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released implementation on a contour identical to its reference. If the result is anything other than 1, or if moving a single test point onto the reference boundary causes a discontinuous jump from a positive score to zero, the boundary-case ambiguity in Eq. 10 is confirmed and the metric is not well defined at perfect agreement.","supporting_citations":[{"cited_title":"Quantitative evaluation of image segmentation incorporating medical consideration functions","cited_arxiv_id":null,"evidence_quote":"Supplies the bidirectional local distance and medical consideration functions that MSI adapts into its scoring formula."},{"cited_title":"The predictive value of segmentation metrics on dosimetry in organs at risk of the brain","cited_arxiv_id":null,"evidence_quote":"Shows that 23 standard segmentation metrics have limited predictive value for treatment quality, motivating a clinically oriented metric."},{"cited_title":"Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space","cited_arxiv_id":null,"evidence_quote":"Supplies the public prostate MRI dataset used to demonstrate that MSI flags a bladder-reaching outer deviation."},{"cited_title":"Metrics to evaluate the performance of auto-segmentation for radiation treatment planning: A critical review","cited_arxiv_id":null,"evidence_quote":"Reviews metrics for auto-segmentation evaluation and frames the clinical evaluation gap the paper addresses."}],"review_version":1}