REVIEW 3 major objections 6 minor 16 references
The paper shows that a frozen anatomy-specific CNN and a frozen promptable foundation model can be coupled at inference time—without retraining or manual prompts—to lift median Dice on hip MRI from 0.71 to 0.92 and cut boundary error fourfo
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 00:35 UTC pith:AVVCAJDC
load-bearing objection A well-built inference-time optimization framework whose headline gains are partly in-sample — worth refereeing, but the authors should be asked to validate the chosen policy on a true holdout. the 3 major comments →
MedSAM2-Anatomy: Training-Free Inference-Time Optimization for Musculoskeletal Segmentation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that domain shift degrades a frozen network's delineation (its surface tracing) far faster than it degrades its localization (its spatial placement of the structure), and this asymmetry can be exploited at inference time. By generating multiple prompt hypotheses from the expert's coarse prior and fusing the foundation model's candidate masks with a union rule plus a laterality plausibility guard, the framework recovers boundaries the expert cannot draw and suppresses the prompt sensitivity that makes an unguided foundation model inconsistent across anatomical sides. No weight is updated and no user supplies a prompt. The gain concentrates exactly where the expert's bound
What carries the argument
The load-bearing mechanism is the inference-time execution policy, whose four tunable components are: automatic prompt generation from a frozen expert's binary prior, placement of N=3 equidistant seed slices along the structure's longitudinal extent, propagation of each seed through a frozen promptable 3D foundation model, and voxel-wise union fusion (threshold τ=1) of the resulting N candidate masks. A laterality plausibility guard rejects any bilateral prior whose predicted volume is below 15% of its contralateral partner before prompting, blocking the failure mode where a truncated-field sliver drifts onto the opposite anatomy and contaminates the union. The ablation identifies the fusion
Load-bearing premise
The reported gains rest on the assumption that the policy configuration chosen on a 30-instance subset of the same target-domain cohorts—three prompt slices, union fusion, and the 15% laterality threshold—will transfer to new scans, even though the margin over the default majority vote is not statistically significant (p=0.058) and the primary cohorts are explicitly in-sample with respect to that selection.
What would settle it
Run the same frozen expert plus frozen foundation model pipeline, with three prompt slices, union fusion, and the 15% laterality threshold, on a fresh institutional cohort of hip MRI or shoulder CT acquired with a different scanner or protocol, scored against independent expert contours; if the average median Dice gain over the frozen expert is not comparable to the reported +0.21 on hip MRI (or the lower tail of per-instance Dice reappears), the policy selection was overfit and the general claim of training-free inference-time improvement would be falsified for these settings.
If this is right
- Surgical-planning pipelines can adopt a new institutional imaging protocol without retraining or manual prompting, provided a small target-domain cohort is available to select the policy once.
- Automatic multi-hypothesis prompting removes the laterality asymmetry and single-prompt failure cases observed with an unguided foundation model, making promptable 3D segmentation viable for batch processing.
- The finding that the optimal fusion rule depends on expert prior reliability implies that no single inference-time recipe is universal; a confidence-aware adaptive fusion that weights the prior per case is needed to interpolate between regimes.
- The method's cost is bounded: compute scales linearly with the prompt budget N, and N=3 gives the best accuracy/compute trade-off on both target cohorts.
- Because either frozen component can be replaced without changing the framework, stronger future expert or foundation models can be dropped in directly, and adding a new anatomy requires only declaring an expert and a label.
Where Pith is reading between the lines
- If the localization-survives / delineation-degrades asymmetry generalizes, similar inference-time policies could combine other frozen expert-localizers with other promptable foundation models across anatomies and modalities, effectively upgrading frozen systems without retraining.
- The 15% laterality threshold was calibrated on observed sliver-to-partner volume ratios (0.0003–0.047) of the target cohort; a more portable version could replace it with a model-free ratio test derived from anatomical volume priors, enabling use on structures without a clear bilateral partner.
- A testable extension of the paper's confidence-aware fusion proposal is to compute a per-case prior-reliability signal (e.g., prior-to-candidate volume ratio or inter-candidate spread) and switch between union and majority fusion within a single scan; this predicts improved performance on cohorts mixing in-distribution and out-of-distribution anatomies.
- The paper's own in-sample selection caveat implies the strongest external check is a fresh institutional cohort: if the 0.71→0.92 hip MRI gain does not reproduce with the same frozen-configuration policy, the framework's practical value would rest on per-site policy re-selection rather than on a universal inference-time improvement.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MedSAM2-Anatomy, a training-free inference-time optimization framework that couples a frozen expert CNN (TotalSegmentator) with a frozen promptable foundation model (MedSAM2). The expert's segmentation is used as an anatomical prior to automatically generate N equidistant prompt slices; each prompt is propagated through MedSAM2; the N candidate masks are fused using a union rule (Eq. 6, τ=1); and a laterality plausibility guard (Eq. 8, ρ=0.15) rejects implausible priors before prompting. No weights are updated and no manual prompts are used. On the Balgrist-V0 target-domain cohorts, the authors report average median DSC increases from 0.7105 to 0.9242 on hip MRI and from 0.8851 to 0.9216 on shoulder CT, with median HD95 on hip MRI falling from 22.04 mm to 4.99 mm. On the public TotalSegmentator CT/MRI test sets, the frozen expert remains strongest, and the authors interpret this as evidence that the optimal fusion strategy depends on expert-prior reliability. The paper includes an ablation identifying the fusion layer plus plausibility guard as the key components, and reports a deterministic, reproducible pipeline.
Significance. If the target-domain gains are validated out-of-sample, this is a useful and practical contribution: it shows that frozen segmentation models can be improved at inference time without retraining or manual prompting, using only a routing policy over existing public checkpoints. The study has notable strengths: paired evaluation on 272 instances, deterministic inference with cached priors and pinned flags, public code, a clean ablation that isolates the policy from model weights, and honest negative results (e.g., hole filling hurts, larger N does not help). The central caveat is that the policy was selected on a subset of the same Balgrist-V0 cohorts used for the primary evaluation, and the selected arm's margin over the default is not statistically significant (p=0.058). The paper discloses this in §4.5, but the abstract's 'independent' framing obscures it. A holdout evaluation would materially strengthen the evidence; without it, the headline 0.71→0.92 gain is an in-sample estimate and should not be read as a demonstrated transferable effect.
major comments (3)
- [§3.3, §3.4, Table 2] The primary target-domain claim is measured on the same Balgrist-V0 cohorts from which the policy (N=3, τ=1, ρ=0.15) was selected. The ablation set of 15 subjects/30 instances is a subset of the 24-subject primary evaluation, so the Table 2 gains are partly in-sample. The guard's benefit is concentrated in one failure case that lies in the selection set, and the selected configuration's improvement over majority vote is not significant (p=0.058). I do not regard the headline 'independent Balgrist-V0' gain as an out-of-sample estimate. A holdout evaluation on the 9 Balgrist-V0 subjects not used in §3.3, or a nested cross-validation over the 24 subjects, is needed before the abstract's 0.71→0.92 claim can be accepted as transferable.
- [§3.4, Table 2] No confidence intervals or significance tests are reported for the headline DSC/HD95 improvements. The shoulder CT gain over MedSAM2 is small in absolute terms (average median DSC 0.9216 vs 0.9146), and per-structure sample sizes are small (12 subjects per study, with left/right split). Paired bootstrap intervals or Wilcoxon tests over the 272 paired instances would show whether the margins are identifiable. Without them, the reported ranking overstates its reliability—especially on structures such as right scapula, where MedSAM2-Anatomy is numerically below MedSAM2 (0.8481 vs 0.8498).
- [§2.7, §4.3] Equation (8) with ρ=0.15 is justified by an observed gap between sliver ratios (0.0003–0.047) and genuine pairs (near unity), but the ratio-based form and the threshold were chosen after inspecting the same target-domain data used for primary evaluation. With the guard activating on only 3 of 30 ablation instances, the risk of threshold overfitting to those cases is non-negligible. The paper acknowledges the in-sample status in §4.5. To make Eq. (8) a principled component, the authors should report sensitivity of the primary results to ρ over a plausible range (e.g., 0.05–0.5), ideally on the 9 held-out subjects or a fresh institutional cohort, and should state whether the 3 guard-activated instances include the single failure case described in §3.3.
minor comments (6)
- [Table 1(b)] The N=1 row contains only dashes; the text calls it a 'conceptual baseline.' If single-prompt MedSAM2 was not actually run, the row should be removed or explicitly labeled as not evaluated, to avoid implying a measured comparison.
- [Abstract / §3.2] The word 'independent' in 'independent Balgrist-V0 cohorts' is ambiguous. The cohorts are independent of the models' training data, but the policy was selected on a subset of them. Please qualify this in the abstract (e.g., 'independent of model training, with policy selection on a subset disclosed in §3.3–3.4').
- [§3.2] The handling of HD95 when a method returns no mask is described in prose only. A table footnote stating the effective per-cell sample size for the Balgrist-V0 MRI HD95 medians (where one MedSAM2 instance is dropped) would improve interpretability.
- [§3.5] The claim that automatic prompt optimization improves the frozen foundation model on the public cohorts by +0.0081 and +0.0028 average median DSC is presented without uncertainty or a significance test. Given the small margins and sparse per-structure data, this should be softened or accompanied by a paired test.
- [§4.6 / Reference [15]] Typographical spacing in 'V oxTell' (should be 'VoxTell'); check the rendering in the final version.
- [Figure 3] The caption's structure list 'Hip Femur Humerus Scapula' does not match the panel ordering in the top row (Acetabulum, Femur, Humerus, Scapula). Harmonize the ordering for clarity.
Circularity Check
No circular derivation: the reported gains are empirical outputs of frozen models; the disclosed in-sample policy selection is a validity concern, not logical circularity.
full rationale
The paper's derivation chain is self-contained. Candidate masks are outputs of two frozen public checkpoints (TotalSegmentator and MedSAM2) routed through explicit, fixed equations: prompt placement (eqs. 3-4), propagation (eq. 5), threshold fusion (eq. 6), and laterality plausibility (eq. 8). No equation is defined in terms of the reported DSC/HD95, and no fitted parameter is renamed as a prediction. The one self-citation (ref [14]) motivates equidistant prompt placement but is not load-bearing; the prompt-budget ablation independently tests N. The main caveat is disclosed in §3.2 and §4.5: the policy hyperparameters (N=3, tau=1, rho=0.15) were selected on a 30-instance subset of the same Balgrist-V0 cohorts used for primary evaluation, so the Balgrist rows are in-sample with respect to that selection, the margin over majority voting is not significant (p=0.058), and the abstract's 'independent' refers to independence from model training rather than from policy selection. This is selection bias / in-sample optimism about generalization to a fresh cohort, not a self-referential derivation: the reported medians are not forced by construction, and the secondary TotalSegmentator benchmarks plus the disclosed non-significance bound the claim. No circular step meets the evidence bar of Eq.-to-Eq. reduction or fitted-parameter-renamed-as-prediction.
Axiom & Free-Parameter Ledger
free parameters (5)
- Prompt budget N =
3
- Fusion threshold tau =
1 (union)
- Plausibility ratio rho =
0.15
- Intensity preprocessing =
CT window [0,255], MRI clip at [0.5,99.5] percentiles
- Seed placement scheme =
Equidistant slices within bounding box
axioms (5)
- domain assumption The expert prior's localization (position and extent) survives domain shift even when its delineation degrades.
- domain assumption MedSAM2, prompted by a 2D mask/bbox from an arbitrary slice, produces useful 3D candidate segmentations across CT and MRI on the Balgrist protocol.
- domain assumption The Balgrist-V0 reference contours are independent, expert-validated ground truths suitable for cross-domain evaluation.
- domain assumption DSC 0.85 is a meaningful clinical acceptance threshold for bone segmentation.
- domain assumption Mapping TotalSegmentator's 'hip' (os coxae) label to the surgically relevant acetabulum sub-region preserves label comparability.
read the original abstract
High-resolution 3D segmentation of hip and shoulder anatomy from CT and MRI is essential for surgical planning, yet frozen segmentation models often fail under domain shift. CNN-based expert models are fully automatic but lack adaptability, whereas promptable foundation models generalize better but require manual prompting. We present MedSAM2-Anatomy, a training-free inference-time optimization framework that improves frozen segmentation models without retraining or human interaction. A frozen expert model generates anatomical priors that are automatically converted into multiple prompt hypotheses for a frozen 3D foundation model. Candidate masks are fused while anatomically implausible priors are rejected. No model weights are updated and no manual prompts are required. TotalSegmentator and MedSAM2 are used as representative expert and foundation models, allowing the contribution of the inference policy to be isolated. Evaluation on the independent Balgrist-V0 CT and MRI cohorts shows that inference-time optimization increases median Dice from 0.71 to 0.92 on hip MRI and from 0.89 to 0.92 on shoulder CT, while reducing median HD95 on hip MRI from 22.0 mm to 5.0 mm. On public TotalSegmentator benchmarks, the expert model remains strongest, indicating that the optimal fusion strategy depends on the reliability of the expert prior. These results demonstrate that training-free inference-time optimization provides a practical strategy for improving frozen segmentation models without manual prompting.
Figures
Reference graph
Works this paper leans on
-
[1]
U-net: Con- volutional networks for biomedical image segmentation,
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Con- volutional networks for biomedical image segmentation,” Medical Image Computing and Computer-Assisted Inter- vention, pp. 234–241, 2015
2015
-
[2]
nnu-net: a self-configuring method for deep learning-based biomedical image segmentation,
F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnu-net: a self-configuring method for deep learning-based biomedical image segmentation,”Nature Methods, vol. 18, no. 2, pp. 203–211, 2021
2021
-
[3]
TotalSegmentator: Robust segmentation of 104 anatomic structures in CT images,
J. Wasserthal, H.-C. Breit, M. T. Meyer, M. Pradella, D. Hinck, A. W. Sauter, T. Heye, D. T. Boll, J. Cyriac, S. Yang, M. Bach, and M. Segeroth, “TotalSegmentator: Robust segmentation of 104 anatomic structures in CT images,”Radiol.: Artif. Intell., vol. 5, no. 5, Sep. 2023, art. no. e230024
2023
-
[4]
TotalSegmentator MRI: Robust sequence-independent segmentation of multiple anatomic structures in MRI,
T. Akinci D’Antonoli, L. K. Berger, A. K. Indrakanti, N. Vishwanathan, J. Weiss, M. Jung, Z. Berkarda, A. Rau, M. Reisert, T. Küstner, A. Walter, E. M. Merkle, D. Boll, H.-C. Breit, A. P. Nicoli, M. Segeroth, J. Cyr- iac, S. Yang, and J. Wasserthal, “TotalSegmentator MRI: Robust sequence-independent segmentation of multiple anatomic structures in MRI,”Rad...
2025
-
[5]
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rol- land, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Loet al., “Segment anything,”arXiv preprint arXiv:2304.02643, 2023
Pith/arXiv arXiv 2023
-
[6]
Sam 2: Segment anything in images and videos,
N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafsonet al., “Sam 2: Segment anything in images and videos,”arXiv preprint arXiv:2408.00714, 2024
Pith/arXiv arXiv 2024
-
[7]
Seg- ment anything in medical images,
J. Ma, Y . He, F. Li, L. Han, C. You, and B. Wang, “Seg- ment anything in medical images,”Nature Communica- tions, vol. 15, no. 1, p. 654, 2024
2024
-
[8]
MedSAM2: Segment anything in 3D medical images and videos,
J. Ma, S. Yang, C. Zhang, Y . Gu, Z. He, J. Zhang, Y . Zhao, L. Wang, R. Zhang, D. Chen, A. Wong, and B. Wang, “MedSAM2: Segment anything in 3D medical images and videos,”arXiv preprint arXiv:2408.03322, 2024
Pith/arXiv arXiv 2024
-
[9]
SIT-SAM: A semantic-integration trans- former that adapts the segment anything model to medical imaging,
Y . Shenet al., “SIT-SAM: A semantic-integration trans- former that adapts the segment anything model to medical imaging,”Biomed. Signal Process. Control, vol. 97, p. 108086, 2025
2025
-
[10]
SegmentAnyBone: A universal model that segments any bone at any location on MRI,
H. Gu, R. Colglazier, H. Dong, J. Zhang, Y . Chen, Z. Yildiz, Y . Chen, L. Li, J. Yang, J. Willhite, A. M. Meyer, A. S. Bergamasco, G. Sgroi, E. Kim, J. Lee, M. Faraj, S. Ronnekleiv-Kelly, M. R. Bashir, and M. A. Mazurowski, “SegmentAnyBone: A universal model that segments any bone at any location on MRI,”arXiv preprint arXiv:2401.12974, 2024
Pith/arXiv arXiv 2024
-
[11]
Foundation models for generalist medical artificial intelligence,
M. Moor, O. Banerjee, Z. S. H. Abad, H. M. Krumholz, J. Leskovec, E. J. Topol, and P. Rajpurkar, “Foundation models for generalist medical artificial intelligence,”Na- ture, vol. 616, no. 7956, pp. 259–265, 2023
2023
-
[12]
Generalist mod- els in medical image segmentation: A survey and compar- ison,
A. Moglia, M. Leccardi, M. Cavicchioli, A. Maccarini, M. Marcon, L. Mainardi, and P. Cerveri, “Generalist mod- els in medical image segmentation: A survey and compar- ison,”arXiv preprint arXiv:2506.10825, 2025
arXiv 2025
-
[13]
nnInteractive: Redefining 3D promptable segmentation,
F. Isensee, M. Rokuss, L. Krämer, S. Dinkelacker, A. Ravindran, F. Stritzke, B. Hamm, T. Wald, M. Langen- berg, C. Ulrich, J. Deissler, R. Floca, and K. Maier-Hein, “nnInteractive: Redefining 3D promptable segmentation,” arXiv preprint arXiv:2503.08373, 2025
Pith/arXiv arXiv 2025
-
[14]
A multiclass radiomics method-based WHO severity scale for improving COVID- 19 patient assessment and disease characterization from CT scans,
J. A. Garcia-Henao, A. Depotter, D. V . Bower, H. Ba- jercius, P. T. Todorovaet al., “A multiclass radiomics method-based WHO severity scale for improving COVID- 19 patient assessment and disease characterization from CT scans,”Invest. Radiol., vol. 58, no. 12, pp. 882–893, 2023
2023
-
[15]
V oxTell: Free-text promptable universal 3D medical image segmen- tation,
M. Rokuss, M. Langenberg, Y . Kirchhoff, F. Isensee, B. Hamm, C. Ulrich, S. Regnery, L. Bauer, E. Katsi- giannopulos, T. Norajitra, and K. Maier-Hein, “V oxTell: Free-text promptable universal 3D medical image segmen- tation,”arXiv preprint arXiv:2511.11450, 2025
arXiv 2025
-
[16]
MedGemma 1.5 technical report,
A. Sellergren, C. Gao, F. Mahvar, T. Kohlberger, F. Jamil, M. Traverse, A. Tono, B. Sadjad, L. Yang et al., “MedGemma 1.5 technical report,”arXiv preprint arXiv:2604.05081, 2026. 13
Pith/arXiv arXiv 2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.