Pith. sign in

REVIEW 3 major objections 6 minor 16 references

The paper shows that a frozen anatomy-specific CNN and a frozen promptable foundation model can be coupled at inference time—without retraining or manual prompts—to lift median Dice on hip MRI from 0.71 to 0.92 and cut boundary error fourfo

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-04 00:35 UTC pith:AVVCAJDC

load-bearing objection A well-built inference-time optimization framework whose headline gains are partly in-sample — worth refereeing, but the authors should be asked to validate the chosen policy on a true holdout. the 3 major comments →

arxiv 2608.00195 v1 pith:AVVCAJDC submitted 2026-07-31 eess.IV cs.CVcs.LG

MedSAM2-Anatomy: Training-Free Inference-Time Optimization for Musculoskeletal Segmentation

classification eess.IV cs.CVcs.LG
keywords inference-time optimizationfrozen modelspromptable foundation modelsmedical image segmentationdomain shiftmusculoskeletal imagingmask fusionplausibility filtering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper seeks to establish that frozen segmentation models—networks whose weights cannot be updated—can still be improved under domain shift by optimizing the policy that routes data through them, rather than the models themselves. It couples a frozen anatomy-specific CNN (used as a coarse localizer) with a frozen promptable 3D foundation model (used as a delineator): the CNN's prior is automatically converted into several equidistant prompt hypotheses, each propagated independently, and the resulting candidate masks are fused by union after anatomically implausible priors are rejected. On an independent institutional hip MRI cohort, average median Dice rises from 0.71 to 0.92 and median HD95 falls from 22 mm to 5 mm; on shoulder CT the gain is smaller but consistent. The paper also shows the optimal fusion rule is not universal: on public benchmarks where the expert prior is already reliable, the same policy underperforms the expert, indicating the best policy depends on prior reliability.

Core claim

The central claim is that domain shift degrades a frozen network's delineation (its surface tracing) far faster than it degrades its localization (its spatial placement of the structure), and this asymmetry can be exploited at inference time. By generating multiple prompt hypotheses from the expert's coarse prior and fusing the foundation model's candidate masks with a union rule plus a laterality plausibility guard, the framework recovers boundaries the expert cannot draw and suppresses the prompt sensitivity that makes an unguided foundation model inconsistent across anatomical sides. No weight is updated and no user supplies a prompt. The gain concentrates exactly where the expert's bound

What carries the argument

The load-bearing mechanism is the inference-time execution policy, whose four tunable components are: automatic prompt generation from a frozen expert's binary prior, placement of N=3 equidistant seed slices along the structure's longitudinal extent, propagation of each seed through a frozen promptable 3D foundation model, and voxel-wise union fusion (threshold τ=1) of the resulting N candidate masks. A laterality plausibility guard rejects any bilateral prior whose predicted volume is below 15% of its contralateral partner before prompting, blocking the failure mode where a truncated-field sliver drifts onto the opposite anatomy and contaminates the union. The ablation identifies the fusion

Load-bearing premise

The reported gains rest on the assumption that the policy configuration chosen on a 30-instance subset of the same target-domain cohorts—three prompt slices, union fusion, and the 15% laterality threshold—will transfer to new scans, even though the margin over the default majority vote is not statistically significant (p=0.058) and the primary cohorts are explicitly in-sample with respect to that selection.

What would settle it

Run the same frozen expert plus frozen foundation model pipeline, with three prompt slices, union fusion, and the 15% laterality threshold, on a fresh institutional cohort of hip MRI or shoulder CT acquired with a different scanner or protocol, scored against independent expert contours; if the average median Dice gain over the frozen expert is not comparable to the reported +0.21 on hip MRI (or the lower tail of per-instance Dice reappears), the policy selection was overfit and the general claim of training-free inference-time improvement would be falsified for these settings.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Surgical-planning pipelines can adopt a new institutional imaging protocol without retraining or manual prompting, provided a small target-domain cohort is available to select the policy once.
  • Automatic multi-hypothesis prompting removes the laterality asymmetry and single-prompt failure cases observed with an unguided foundation model, making promptable 3D segmentation viable for batch processing.
  • The finding that the optimal fusion rule depends on expert prior reliability implies that no single inference-time recipe is universal; a confidence-aware adaptive fusion that weights the prior per case is needed to interpolate between regimes.
  • The method's cost is bounded: compute scales linearly with the prompt budget N, and N=3 gives the best accuracy/compute trade-off on both target cohorts.
  • Because either frozen component can be replaced without changing the framework, stronger future expert or foundation models can be dropped in directly, and adding a new anatomy requires only declaring an expert and a label.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the localization-survives / delineation-degrades asymmetry generalizes, similar inference-time policies could combine other frozen expert-localizers with other promptable foundation models across anatomies and modalities, effectively upgrading frozen systems without retraining.
  • The 15% laterality threshold was calibrated on observed sliver-to-partner volume ratios (0.0003–0.047) of the target cohort; a more portable version could replace it with a model-free ratio test derived from anatomical volume priors, enabling use on structures without a clear bilateral partner.
  • A testable extension of the paper's confidence-aware fusion proposal is to compute a per-case prior-reliability signal (e.g., prior-to-candidate volume ratio or inter-candidate spread) and switch between union and majority fusion within a single scan; this predicts improved performance on cohorts mixing in-distribution and out-of-distribution anatomies.
  • The paper's own in-sample selection caveat implies the strongest external check is a fresh institutional cohort: if the 0.71→0.92 hip MRI gain does not reproduce with the same frozen-configuration policy, the framework's practical value would rest on per-site policy re-selection rather than on a universal inference-time improvement.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes MedSAM2-Anatomy, a training-free inference-time optimization framework that couples a frozen expert CNN (TotalSegmentator) with a frozen promptable foundation model (MedSAM2). The expert's segmentation is used as an anatomical prior to automatically generate N equidistant prompt slices; each prompt is propagated through MedSAM2; the N candidate masks are fused using a union rule (Eq. 6, τ=1); and a laterality plausibility guard (Eq. 8, ρ=0.15) rejects implausible priors before prompting. No weights are updated and no manual prompts are used. On the Balgrist-V0 target-domain cohorts, the authors report average median DSC increases from 0.7105 to 0.9242 on hip MRI and from 0.8851 to 0.9216 on shoulder CT, with median HD95 on hip MRI falling from 22.04 mm to 4.99 mm. On the public TotalSegmentator CT/MRI test sets, the frozen expert remains strongest, and the authors interpret this as evidence that the optimal fusion strategy depends on expert-prior reliability. The paper includes an ablation identifying the fusion layer plus plausibility guard as the key components, and reports a deterministic, reproducible pipeline.

Significance. If the target-domain gains are validated out-of-sample, this is a useful and practical contribution: it shows that frozen segmentation models can be improved at inference time without retraining or manual prompting, using only a routing policy over existing public checkpoints. The study has notable strengths: paired evaluation on 272 instances, deterministic inference with cached priors and pinned flags, public code, a clean ablation that isolates the policy from model weights, and honest negative results (e.g., hole filling hurts, larger N does not help). The central caveat is that the policy was selected on a subset of the same Balgrist-V0 cohorts used for the primary evaluation, and the selected arm's margin over the default is not statistically significant (p=0.058). The paper discloses this in §4.5, but the abstract's 'independent' framing obscures it. A holdout evaluation would materially strengthen the evidence; without it, the headline 0.71→0.92 gain is an in-sample estimate and should not be read as a demonstrated transferable effect.

major comments (3)
  1. [§3.3, §3.4, Table 2] The primary target-domain claim is measured on the same Balgrist-V0 cohorts from which the policy (N=3, τ=1, ρ=0.15) was selected. The ablation set of 15 subjects/30 instances is a subset of the 24-subject primary evaluation, so the Table 2 gains are partly in-sample. The guard's benefit is concentrated in one failure case that lies in the selection set, and the selected configuration's improvement over majority vote is not significant (p=0.058). I do not regard the headline 'independent Balgrist-V0' gain as an out-of-sample estimate. A holdout evaluation on the 9 Balgrist-V0 subjects not used in §3.3, or a nested cross-validation over the 24 subjects, is needed before the abstract's 0.71→0.92 claim can be accepted as transferable.
  2. [§3.4, Table 2] No confidence intervals or significance tests are reported for the headline DSC/HD95 improvements. The shoulder CT gain over MedSAM2 is small in absolute terms (average median DSC 0.9216 vs 0.9146), and per-structure sample sizes are small (12 subjects per study, with left/right split). Paired bootstrap intervals or Wilcoxon tests over the 272 paired instances would show whether the margins are identifiable. Without them, the reported ranking overstates its reliability—especially on structures such as right scapula, where MedSAM2-Anatomy is numerically below MedSAM2 (0.8481 vs 0.8498).
  3. [§2.7, §4.3] Equation (8) with ρ=0.15 is justified by an observed gap between sliver ratios (0.0003–0.047) and genuine pairs (near unity), but the ratio-based form and the threshold were chosen after inspecting the same target-domain data used for primary evaluation. With the guard activating on only 3 of 30 ablation instances, the risk of threshold overfitting to those cases is non-negligible. The paper acknowledges the in-sample status in §4.5. To make Eq. (8) a principled component, the authors should report sensitivity of the primary results to ρ over a plausible range (e.g., 0.05–0.5), ideally on the 9 held-out subjects or a fresh institutional cohort, and should state whether the 3 guard-activated instances include the single failure case described in §3.3.
minor comments (6)
  1. [Table 1(b)] The N=1 row contains only dashes; the text calls it a 'conceptual baseline.' If single-prompt MedSAM2 was not actually run, the row should be removed or explicitly labeled as not evaluated, to avoid implying a measured comparison.
  2. [Abstract / §3.2] The word 'independent' in 'independent Balgrist-V0 cohorts' is ambiguous. The cohorts are independent of the models' training data, but the policy was selected on a subset of them. Please qualify this in the abstract (e.g., 'independent of model training, with policy selection on a subset disclosed in §3.3–3.4').
  3. [§3.2] The handling of HD95 when a method returns no mask is described in prose only. A table footnote stating the effective per-cell sample size for the Balgrist-V0 MRI HD95 medians (where one MedSAM2 instance is dropped) would improve interpretability.
  4. [§3.5] The claim that automatic prompt optimization improves the frozen foundation model on the public cohorts by +0.0081 and +0.0028 average median DSC is presented without uncertainty or a significance test. Given the small margins and sparse per-structure data, this should be softened or accompanied by a paired test.
  5. [§4.6 / Reference [15]] Typographical spacing in 'V oxTell' (should be 'VoxTell'); check the rendering in the final version.
  6. [Figure 3] The caption's structure list 'Hip Femur Humerus Scapula' does not match the panel ordering in the top row (Acetabulum, Femur, Humerus, Scapula). Harmonize the ordering for clarity.

Circularity Check

0 steps flagged

No circular derivation: the reported gains are empirical outputs of frozen models; the disclosed in-sample policy selection is a validity concern, not logical circularity.

full rationale

The paper's derivation chain is self-contained. Candidate masks are outputs of two frozen public checkpoints (TotalSegmentator and MedSAM2) routed through explicit, fixed equations: prompt placement (eqs. 3-4), propagation (eq. 5), threshold fusion (eq. 6), and laterality plausibility (eq. 8). No equation is defined in terms of the reported DSC/HD95, and no fitted parameter is renamed as a prediction. The one self-citation (ref [14]) motivates equidistant prompt placement but is not load-bearing; the prompt-budget ablation independently tests N. The main caveat is disclosed in §3.2 and §4.5: the policy hyperparameters (N=3, tau=1, rho=0.15) were selected on a 30-instance subset of the same Balgrist-V0 cohorts used for primary evaluation, so the Balgrist rows are in-sample with respect to that selection, the margin over majority voting is not significant (p=0.058), and the abstract's 'independent' refers to independence from model training rather than from policy selection. This is selection bias / in-sample optimism about generalization to a fresh cohort, not a self-referential derivation: the reported medians are not forced by construction, and the secondary TotalSegmentator benchmarks plus the disclosed non-significance bound the claim. No circular step meets the evidence bar of Eq.-to-Eq. reduction or fitted-parameter-renamed-as-prediction.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The framework introduces no new physical or computational entities; the 'execution policy' is a conceptual re-description of the routing strategy. The main hidden cost is the selection of N, tau, and rho on the evaluation cohort, which the paper discloses but the abstract does not.

free parameters (5)
  • Prompt budget N = 3
    Selected on the 30-instance ablation subset (Table 1b); N=4 and N=5 did not significantly improve median DSC (Friedman test p=0.20).
  • Fusion threshold tau = 1 (union)
    Union fusion selected over majority vote and soft-fusion variants in the ablation (Table 1a); majority vote was found to erode thin structures.
  • Plausibility ratio rho = 0.15
    Eq. (8); threshold set after observing sliver/partner volume ratios (0.0003-0.047 for slivers, near unity for genuine pairs). The 'empty region' claim is asserted, not shown as a distribution.
  • Intensity preprocessing = CT window [0,255], MRI clip at [0.5,99.5] percentiles
    Fixed preprocessing choices described in section 2.9; not fitted but part of the policy.
  • Seed placement scheme = Equidistant slices within bounding box
    Eq. (4); deterministic placement is a design choice inherited from the authors' prior AssessNet-19 framework.
axioms (5)
  • domain assumption The expert prior's localization (position and extent) survives domain shift even when its delineation degrades.
    Load-bearing for treating TotalSegmentator as a prompt generator; supported by the paper's measurements on two target cohorts (sections 2.3, 4.1) but not a universal law.
  • domain assumption MedSAM2, prompted by a 2D mask/bbox from an arbitrary slice, produces useful 3D candidate segmentations across CT and MRI on the Balgrist protocol.
    Relied on throughout section 2.5; the basis of the multi-hypothesis propagation.
  • domain assumption The Balgrist-V0 reference contours are independent, expert-validated ground truths suitable for cross-domain evaluation.
    Section 3.1 asserts independence; we cannot independently verify the annotation quality or the absence of hidden protocol correlations.
  • domain assumption DSC 0.85 is a meaningful clinical acceptance threshold for bone segmentation.
    Invoked in section 3.2 to interpret results; conventional but not derived from downstream planning metrics.
  • domain assumption Mapping TotalSegmentator's 'hip' (os coxae) label to the surgically relevant acetabulum sub-region preserves label comparability.
    Section 3.1 harmonizes labels across cohorts; the mapping may not perfectly align the anatomical definitions.

pith-pipeline@v1.3.0-alltime-deepseek · 16087 in / 11875 out tokens · 105755 ms · 2026-08-04T00:35:49.993442+00:00 · methodology

0 comments
read the original abstract

High-resolution 3D segmentation of hip and shoulder anatomy from CT and MRI is essential for surgical planning, yet frozen segmentation models often fail under domain shift. CNN-based expert models are fully automatic but lack adaptability, whereas promptable foundation models generalize better but require manual prompting. We present MedSAM2-Anatomy, a training-free inference-time optimization framework that improves frozen segmentation models without retraining or human interaction. A frozen expert model generates anatomical priors that are automatically converted into multiple prompt hypotheses for a frozen 3D foundation model. Candidate masks are fused while anatomically implausible priors are rejected. No model weights are updated and no manual prompts are required. TotalSegmentator and MedSAM2 are used as representative expert and foundation models, allowing the contribution of the inference policy to be isolated. Evaluation on the independent Balgrist-V0 CT and MRI cohorts shows that inference-time optimization increases median Dice from 0.71 to 0.92 on hip MRI and from 0.89 to 0.92 on shoulder CT, while reducing median HD95 on hip MRI from 22.0 mm to 5.0 mm. On public TotalSegmentator benchmarks, the expert model remains strongest, indicating that the optimal fusion strategy depends on the reliability of the expert prior. These results demonstrate that training-free inference-time optimization provides a practical strategy for improving frozen segmentation models without manual prompting.

Figures

Figures reproduced from arXiv: 2608.00195 by Armando Hoch, Benedikt Herzog, Benjamin Vella, Bettina Hochreiter, Carmen Castroviejo Fernandez, Cindy Guerrero Toro, Felix \"Ottl, John Garcia Henao, Matthias Biner, Nicholas B\"unger, Norman Juchler, Rico Br\"utsch, Sebastiano Caprara, Sven Hirsch.

Figure 1
Figure 1. Figure 1: MedSAM2-Anatomy as an inference-time optimisation pipeline. A frozen expert network produces the prior [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Multi-cohort evaluation set for hip and shoulder segmentation on CT and MRI. Each column is one cohort under [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-structure DSC distributions across the primary target-domain and secondary public benchmark cohorts. Inference [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison across the four evaluation cohorts, one subject per row. Columns show the source image, the [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 6 linked inside Pith

  1. [1]

    U-net: Con- volutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Con- volutional networks for biomedical image segmentation,” Medical Image Computing and Computer-Assisted Inter- vention, pp. 234–241, 2015

  2. [2]

    nnu-net: a self-configuring method for deep learning-based biomedical image segmentation,

    F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnu-net: a self-configuring method for deep learning-based biomedical image segmentation,”Nature Methods, vol. 18, no. 2, pp. 203–211, 2021

  3. [3]

    TotalSegmentator: Robust segmentation of 104 anatomic structures in CT images,

    J. Wasserthal, H.-C. Breit, M. T. Meyer, M. Pradella, D. Hinck, A. W. Sauter, T. Heye, D. T. Boll, J. Cyriac, S. Yang, M. Bach, and M. Segeroth, “TotalSegmentator: Robust segmentation of 104 anatomic structures in CT images,”Radiol.: Artif. Intell., vol. 5, no. 5, Sep. 2023, art. no. e230024

  4. [4]

    TotalSegmentator MRI: Robust sequence-independent segmentation of multiple anatomic structures in MRI,

    T. Akinci D’Antonoli, L. K. Berger, A. K. Indrakanti, N. Vishwanathan, J. Weiss, M. Jung, Z. Berkarda, A. Rau, M. Reisert, T. Küstner, A. Walter, E. M. Merkle, D. Boll, H.-C. Breit, A. P. Nicoli, M. Segeroth, J. Cyr- iac, S. Yang, and J. Wasserthal, “TotalSegmentator MRI: Robust sequence-independent segmentation of multiple anatomic structures in MRI,”Rad...

  5. [5]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rol- land, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Loet al., “Segment anything,”arXiv preprint arXiv:2304.02643, 2023

  6. [6]

    Sam 2: Segment anything in images and videos,

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafsonet al., “Sam 2: Segment anything in images and videos,”arXiv preprint arXiv:2408.00714, 2024

  7. [7]

    Seg- ment anything in medical images,

    J. Ma, Y . He, F. Li, L. Han, C. You, and B. Wang, “Seg- ment anything in medical images,”Nature Communica- tions, vol. 15, no. 1, p. 654, 2024

  8. [8]

    MedSAM2: Segment anything in 3D medical images and videos,

    J. Ma, S. Yang, C. Zhang, Y . Gu, Z. He, J. Zhang, Y . Zhao, L. Wang, R. Zhang, D. Chen, A. Wong, and B. Wang, “MedSAM2: Segment anything in 3D medical images and videos,”arXiv preprint arXiv:2408.03322, 2024

  9. [9]

    SIT-SAM: A semantic-integration trans- former that adapts the segment anything model to medical imaging,

    Y . Shenet al., “SIT-SAM: A semantic-integration trans- former that adapts the segment anything model to medical imaging,”Biomed. Signal Process. Control, vol. 97, p. 108086, 2025

  10. [10]

    SegmentAnyBone: A universal model that segments any bone at any location on MRI,

    H. Gu, R. Colglazier, H. Dong, J. Zhang, Y . Chen, Z. Yildiz, Y . Chen, L. Li, J. Yang, J. Willhite, A. M. Meyer, A. S. Bergamasco, G. Sgroi, E. Kim, J. Lee, M. Faraj, S. Ronnekleiv-Kelly, M. R. Bashir, and M. A. Mazurowski, “SegmentAnyBone: A universal model that segments any bone at any location on MRI,”arXiv preprint arXiv:2401.12974, 2024

  11. [11]

    Foundation models for generalist medical artificial intelligence,

    M. Moor, O. Banerjee, Z. S. H. Abad, H. M. Krumholz, J. Leskovec, E. J. Topol, and P. Rajpurkar, “Foundation models for generalist medical artificial intelligence,”Na- ture, vol. 616, no. 7956, pp. 259–265, 2023

  12. [12]

    Generalist mod- els in medical image segmentation: A survey and compar- ison,

    A. Moglia, M. Leccardi, M. Cavicchioli, A. Maccarini, M. Marcon, L. Mainardi, and P. Cerveri, “Generalist mod- els in medical image segmentation: A survey and compar- ison,”arXiv preprint arXiv:2506.10825, 2025

  13. [13]

    nnInteractive: Redefining 3D promptable segmentation,

    F. Isensee, M. Rokuss, L. Krämer, S. Dinkelacker, A. Ravindran, F. Stritzke, B. Hamm, T. Wald, M. Langen- berg, C. Ulrich, J. Deissler, R. Floca, and K. Maier-Hein, “nnInteractive: Redefining 3D promptable segmentation,” arXiv preprint arXiv:2503.08373, 2025

  14. [14]

    A multiclass radiomics method-based WHO severity scale for improving COVID- 19 patient assessment and disease characterization from CT scans,

    J. A. Garcia-Henao, A. Depotter, D. V . Bower, H. Ba- jercius, P. T. Todorovaet al., “A multiclass radiomics method-based WHO severity scale for improving COVID- 19 patient assessment and disease characterization from CT scans,”Invest. Radiol., vol. 58, no. 12, pp. 882–893, 2023

  15. [15]

    V oxTell: Free-text promptable universal 3D medical image segmen- tation,

    M. Rokuss, M. Langenberg, Y . Kirchhoff, F. Isensee, B. Hamm, C. Ulrich, S. Regnery, L. Bauer, E. Katsi- giannopulos, T. Norajitra, and K. Maier-Hein, “V oxTell: Free-text promptable universal 3D medical image segmen- tation,”arXiv preprint arXiv:2511.11450, 2025

  16. [16]

    MedGemma 1.5 technical report,

    A. Sellergren, C. Gao, F. Mahvar, T. Kohlberger, F. Jamil, M. Traverse, A. Tono, B. Sadjad, L. Yang et al., “MedGemma 1.5 technical report,”arXiv preprint arXiv:2604.05081, 2026. 13