Pith. sign in

REVIEW 4 major objections 5 minor 19 references

Universality Reconsidered: Rethinking the Validation of Foundation Models for General-Purpose 3D Medical Segmentation

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Pith's one-line read This paper claims that five leading zero-shot 3D medical segmentation models, despite strong benchmark scores, drop sharply or collapse to near-zero Dice on paired whole-body PET/CT and PET/MRI scans, so published benchmarks overstate real-

desk verdict The paired whole-body dataset is a genuine contribution, but the headline claim about catastrophic collapse of semantic models is not yet proven because the evaluation protocol is under-specified and the near-zero Dice scores look like artifacts. read the letter →

arxiv 2602.07643 v2 pith:MNR235DC submitted 2026-02-07 cs.CV

classification cs.CV
keywords 3Dmedicalimagesegmentationfoundationmodelszero-shotgeneralizationmodalitydiscrepancyPET/CTPET/MRIwhole-bodyillusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that current 3D medical segmentation foundation models are not truly general-purpose. Using a newly collected paired benchmark of 490 whole-body PET/CT and 464 whole-body PET/MRI scans with voxel-wise annotations for 13 organs, it shows that five representative zero-shot models perform far below their literature-reported Dice scores (a standard measure of segmentation overlap), and several collapse to near-zero on almost all organs, especially on PET. The paper attributes this to a systemic structural bias: models are trained and evaluated almost entirely on structural imaging such as CT and MRI, so functional imaging like PET lies outside their effective competence. If the finding holds, published benchmarks for these models overstate their real-world generalization, and validation practice should shift toward paired whole-body structural-plus-functional imaging rather than regional structural datasets.

What carries the argument

The key mechanism is the UMD benchmark: a newly collected set of 490 whole-body PET/CT and 464 whole-body PET/MRI scans with voxel-wise annotations for 13 organs. Because the structural and functional volumes come from the same patient in the same diagnostic session, anatomy, scale, and orientation are held constant, turning imaging modality into a controlled independent variable. This paired design is what lets the paper attribute performance drops to modality rather than to organ shape or task complexity, and because the data are newly collected rather than recompiled from public sources, they serve as a leakage-free out-of-distribution test of zero-shot generalization.

What would settle it

Run the UMD evaluation with one matched prompt type for all models—for instance, only text/class prompts, or only point prompts sampled without ground-truth masks—and check whether the near-zero Dice collapses persist. If they do not persist, the generalization gap is partly a prompt-type artifact; if they do, the paper's structural-bias verdict is confirmed.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that imaging modality—not anatomical task complexity—is the decisive variable causing current foundation models to fail. Because each subject's structural and functional volumes were acquired in the same session, the same organs in the same patient appear in both CT or MRI and PET; evaluating the models zero-shot on all of them isolates the modality effect. The result is a systemic pattern: semantic/text-guided models collapse to near-zero Dice on nearly every organ, point-prompted models keep some continuity but stay well below clinical reliability, and even the best-performing model loses well over a third of its reported average score. The paper

Load-bearing premise

The load-bearing premise is that the zero-shot prompting protocol is fair across model families—text/class prompts for some models versus simulated point prompts derived from ground-truth masks for others; if those oracle points leak target location, point-based models are tested under easier conditions and the cross-model comparison is confounded, although the per-model structural-to-functional gap is less affected.

Editorial extensions

If this is right

  • If the paper is right, literature-reported Dice scores for these foundation models overstate how they will perform on unseen whole-body scans.
  • Zero-shot use of these models on functional imaging such as PET is clinically unreliable; several models effectively fail on nearly all organs.
  • Competence on structural imaging does not transfer to functional imaging, so 'general-purpose' claims need to be redefined to include functional modalities.
  • New validation protocols should use paired, un-leaked, whole-body data that isolate modality as the primary variable.
  • Progress in this area requires multimodal training and evaluation that includes functional imaging, not simply larger structural datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the modality gap is as systematic as reported, a testable prediction is that training a model on even a modest amount of PET data should close much of the PET gap, implying the failure is a training-distribution problem rather than an architectural ceiling.
  • Beyond the paper: the total collapse of text/class-prompted models suggests prompt interface may be a confound; a matched-prompt comparison across models would separate prompt-interface effects from model capability and might change the ranking of model families.
  • Beyond the paper: the same paired-benchmark logic could be extended to other functional-structural pairs (for example SPECT/CT or dynamic contrast MRI) and to pathological lesions, testing whether the structural bias is a general law of current medical foundation models.
  • Beyond the paper: the paper's own visual comparison shows collapse even within the CT domain when acquisition parameters shift, so scanner and reconstruction variation is a natural next axis to isolate with the same paired design.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces the UMD dataset of 490 paired whole-body PET/CT and 464 paired PET/MRI scans with 13-organ annotations, and evaluates five 3D medical segmentation foundation models (SAM-Med3D-turbo, SegVol, nnInteractive, VISTA3D, SAT) in a zero-shot setting. The central claim is that these models are not truly general-purpose: on the UMD benchmark they show large drops from literature-reported Dice scores, with the semantic/text-prompt models (VISTA3D, SAT) collapsing to near-zero Dice on nearly all organs, especially on PET. The authors interpret this as a systemic structural bias and a 'generalization illusion' in current benchmarks.

Significance. If the empirical results are reproducible, the UMD dataset and the paired-modality protocol are valuable contributions: the intra-subject design controls for anatomy and is a sensible way to isolate modality effects, and the dataset appears to be genuinely new, avoiding data leakage. The paper also provides a useful critique of current evaluation practices. However, the strength of the central claim depends on the credibility of the near-zero Dice scores for VISTA3D and SAT on CT and PET, which are not yet supported by a documented and validated evaluation protocol. The cross-model comparison is also confounded by the use of oracle point prompts for some models and text/class prompts for others.

major comments (4)
  1. [Appendix, Implementation Details; Table 1; Table 3] The cross-model comparison is confounded by prompt type. Point-based models (SAM-Med3D-turbo, SegVol, nnInteractive) receive simulated point prompts generated from ground-truth masks, while VISTA3D and SAT receive class IDs/text. This provides the point-based models with oracle localization information, making their relatively higher Dice scores a consequence of the interaction protocol rather than of underlying segmentation ability. The paper's claim that 'semantic-guided models catastrophically fail' while point-based models 'maintain continuity' is therefore not a fair comparison. Please report results under a common prompt protocol (e.g., a fixed number of anatomy-derived points for every model) or explicitly analyze the effect of prompt type. The paired-modality comparison within a model is less affected, but the 'generalization illusion' framing relies on the cross-model contrast.
  2. [Table 1, Table 3, Appendix, Implementation Details] The near-zero Dice scores for VISTA3D and SAT, even on CT, are suspicious and may be a protocol artifact rather than evidence of model failure. Table 3 reports UMD CT averages of 0.071 for VISTA3D and 0.018 for SAT, yet Table 1 shows VISTA3D achieving 0.9186 on the liver (described in the text as a CT success). Such an abrupt collapse on almost all other organs is not typical of domain shift and could be caused by preprocessing mismatches (resampling, intensity windowing, orientation, cropping whole-body volumes) or by prompting with class IDs/texts that are not in the model's vocabulary. The Appendix only says official codebases and standardized protocols were used; it does not specify the exact preprocessing, prompt strings, class IDs, or whether the models produced non-empty predictions. Please provide a reproducible protocol and a sanity check (e.g., running the same pipeline on in-d
  3. [Table 3 and surrounding discussion] The comparison between UMD Dice and 'reported performance' is not controlled. The reported numbers come from different test datasets, different evaluation settings, and, for SAT, different units (Dice vs. other metrics), which the table footnote partially acknowledges. Despite this, the text interprets the discrepancy as 'overfitting to curated data distributions' and 'generalization illusion.' This is an uncontrolled cross-dataset comparison. The nnU-Net baseline trained on 10 UMD cases is a useful reference, but it does not separate distribution shift from the claimed structural bias. Please present Table 3 as an indicative comparison only, and base the central claims on the controlled paired-modality analysis.
  4. [Tables 1 and 2] The statistical shading and color coding are not self-explanatory. The captions say red/blue/teal indicate PET/CT/MRI, but the row order is not labeled, and a monochrome print or a color-blind reader cannot infer which number is PET and which is CT. For example, VISTA3D's liver cell 0.9186 is called a CT success in the text, but the table structure does not make this clear. Additionally, no statistical test is named, no sample size is given for the p-values, and no multiple-comparison correction is described. Please make table rows explicit and specify the test procedure.
minor comments (5)
  1. [Abstract] The sentence 'While emerging 3D medical foundation models are envisioned as versatile tools with offer general-purpose capabilities' contains a grammar error ('with offer').
  2. [Table 1 header] The model name is misspelled as 'nnIteractive' in Table 1; elsewhere in the paper it is 'nnInteractive.'
  3. [Section 5] 'Despite high reported benchmarks, these models remain limited generalization ability' is ungrammatical. Suggest rewording to 'these models remain limited in their generalization ability' or similar.
  4. [Figure 2] The figure legend refers to 'Task 1AbdomenCT' etc., and the formatting appears corrupted. Please ensure the text in figure panels is readable and the contrast between 'confounded' and 'isolated' designs is clear.
  5. [Section 4, paragraph 1] When discussing the 'isolated outlier' nature of VISTA3D's liver CT success, the paper does not mention that liver is the most frequent organ in VISTA3D's training data. This context might help readers interpret the result, though it is not required.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: measured zero-shot performance on an external paired whole-body benchmark; self-citations are background only.

full rationale

The paper is an empirical evaluation. Its central claim that current 3D foundation models are not truly general-purpose is supported by Dice measurements of five pre-existing models on a newly collected UMD dataset, not by a derivation that reduces to its inputs. There are no fitted parameters relabeled as predictions, no quantity defined in terms of the target conclusion, and no uniqueness theorem imported from the authors' prior work. The authors' self-citations are background only (e.g., PET signal characteristics [16], related 2.5D methods [13], prior foundation-model surveys) and are not load-bearing for the empirical result. The prompt asymmetry (text/class IDs for VISTA3D/SAT vs. ground-truth-simulated point prompts for point-based models) and the absence of detailed preprocessing documentation are validity and fairness concerns, not circularity; they could affect whether the near-zero semantic-model scores are real or artifacts. Similarly, comparing literature-reported benchmark numbers to UMD numbers across different test sets is a comparability concern, and the paper itself marks this with '*' and '/', so it is not a covert circular reduction. The benchmark is an independent external testbed for pre-existing models, so the circularity burden is minimal and no specific circular step can be identified.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No fitted free parameters. The central claim depends on five domain assumptions, the most fragile being the oracle point-prompt protocol and implicit co-registration of the paired scans. The benchmark is new, but these assumptions are load-bearing for the stated conclusions.

assumptions (5)
  • domain assumption Intra-subject paired PET/CT and PET/MRI acquisitions are sufficiently co-registered to treat imaging modality as the isolated independent variable.
    Section 3 Benchmark Design claims 'intrinsic spatial and anatomical consistency' but no registration algorithm, QA, or alignment verification is reported.
  • ad hoc to paper Ground-truth-derived point prompts constitute a fair zero-shot interaction protocol.
    Appendix Implementation Details: point prompts are sampled from GT masks, leaking target location and not representing realistic user interaction.
  • domain assumption Reported benchmark scores in Table 3 are informative for the generalization gap despite different test data.
    Table 3 asterisks say test data differ and cannot be directly compared, yet the headline 'stark discrepancy' relies on that comparison.
  • domain assumption The five selected models are representative of current 3D medical segmentation foundation models.
    Section 3 selects SAM-Med3D-turbo, SegVol, SAT-Pro, VISTA3D, and nnInteractive; the conclusion broadens to all current foundation models.
  • domain assumption Dice similarity on 13 whole-body organs is an adequate proxy for clinical generalization.
    The paper's own limitation statement restricts evaluation to whole-body organs, limited prompt types, and Dice only, yet the central conclusion is stated broadly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Universality Reconsidered: Rethinking the Validation of Foundation Models for General-Purpose 3D Medical Segmentation." pith.science (2026). https://pith.science/paper/MNR235DC

@misc{pith2026260207643,
  author       = {Pith},
  title        = {Pith review of: Universality Reconsidered: Rethinking the Validation of Foundation Models for General-Purpose 3D Medical Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MNR235DC}},
  note         = {Machine review of arXiv:2602.07643}
}
read the original abstract

Foundation models have emerged as a transformative paradigm in 3D medical imaging, with the promise of unified quantitative analysis across diverse targets and imaging modalities. Yet the prevailing conception of universality remains incomplete. Current models are predominantly developed and evaluated on datasets largely concentrated around a limited set of imaging modalities and anatomical regions. In this Perspective, we evaluate representative 3D segmentation foundation models using paired whole-body structural and functional imaging data. Our analysis reveals a substantial gap between benchmark-reported performance and real-world generalization, with marked degradation on previously unseen data and particularly severe failures on functional imaging modalities. These findings suggest that current foundation models remain far from achieving true universality. We argue that progress requires not only scaling models and datasets, but also a reconsideration of how universality is defined and validated, extending evaluation beyond regional structural benchmarks toward whole-body structural and functional imaging. Our observations highlight the need to distinguish benchmark success from genuine clinical generalization. Bridging this gap will be essential for translating foundation models from controlled evaluation settings to real-world medical practice.

Figures

Figures reproduced from arXiv: 2602.07643 by the authors.

Figure 1
Figure 1. The structural bias in data distribution of general-purpose medical segmenta￾tion foundation models. A profound disparity is observed between structural imaging (CT and MRI) and functional imaging (PET), with the latter constituting a negligible fraction of the total data across all models. slice-by-slice and reconstructed through post-processing. This strategy inher￾ently overlooks the critical volumetric contextua… view at source ↗
Figure 2
Figure 2. (a) Existing validation protocols typically assess models on heterogeneous datasets where modality is intrinsically entangled with specific anatomical tasks. This approach prevents an isolated measurement of modality-specific robustness, as perfor￾mance variations are confounded by varying task complexities. (b) In contrast, our evaluation utilizes paired whole-body PET/CT and PET/MRI data. By performing simultaneou… view at source ↗
Figure 3
Figure 3. Visual comparison of failure cases of SegVol with performance across in-domain and out-of-domain UMD datasets. This stark illustrates the generalization illusion, where high scores on benchmark datasets fail to translate into robust clinical utility on unseen distributions [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative assessment of modality discrepancy via nnInteractive under￾scores a fundamental performance gap. Despite exhibiting the highest relative general￾ization ability in our evaluation, the model remains incapable of mapping anatomical priors onto the distinct si…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 1 linked inside Pith

  1. [1]

    Advances in neural Information Processing Systems 37, 110746–110783

    Du, Y., Bai, F., Huang, T., Zhao, B.: Segvol: Universal and interactive volumetric medical image segmentation. Advances in neural Information Processing Systems 37, 110746–110783. (2024) 2, 5

  2. [2]

    Nature Machine In- telligence pp

    Gatidis, S., Früh, M., Fabritius, M.P., Gu, S., Nikolaou, K., Fougère, C.L., Ye, J., He, J., Peng, Y., Bi, L., et al.: Results from the autopet challenge on fully automated lesion segmentation in oncologic pet/ct imaging. Nature Machine In- telligence pp. 1–10 (2024) 4

  3. [3]

    Scientific Data9(1), 601 (2022) 4

    Gatidis, S., Hepp, T., Früh, M., La Fougère, C., Nikolaou, K., Pfannenberg, C., Schölkopf, B., Küstner, T., Cyran, C., Rubin, D.: A whole-body fdg-pet/ct dataset with manually annotated tumor lesions. Scientific Data9(1), 601 (2022) 4

  4. [4]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    He, Y., Guo, P., Tang, Y., Myronenko, A., Nath, V., Xu, Z., Yang, D., Zhao, C., Si- mon, B., Belue, M., et al.: Vista3d: A unified segmentation foundation model for 3d medical imaging. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 20863–20873 (2025) 2, 5

  5. [5]

    arXiv preprint arXiv:2503.08373 (2025) 2, 5

    Isensee, F., Rokuss, M., Krämer, L., Dinkelacker, S., Ravindran, A., Stritzke, F., Hamm, B., Wald, T., Langenberg, M., Ulrich, C., et al.: nninteractive: Redefining 3d promptable segmentation. arXiv preprint arXiv:2503.08373 (2025) 2, 5

  6. [6]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023) 1

  7. [7]

    the results of the emidec challenge

    Lalande, A., Chen, Z., Pommier, T., Decourselle, T., Qayyum, A., Salomon, M., Ginhac, D., Skandarani, Y., Boucher, A., Brahim, K., et al.: Deep learning methods for automatic evaluation of delayed enhancement-mri. the results of the emidec challenge. Medical Image Analysis79, 102428 (2022) 4

  8. [8]

    Nature Communications15, 1–9 (2024) 1

    Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications15, 1–9 (2024) 1

Show all 19 references
  1. [9]

    Nature Methods20(7), 953–955 (2023) 1

    Ma, J., Wang, B.: Towards foundation models of biological image segmentation. Nature Methods20(7), 953–955 (2023) 1

  2. [10]

    Ma, J., Zhang, Y., Gu, S., Zhu, C., Ge, C., Zhang, Y., An, X., Wang, C., Wang, Q., Liu, X., Cao, S., Zhang, Q., Liu, S., Wang, Y., Li, Y., He, J., Yang, X.: Abdomenct- 1k: Is abdominal organ segmentation a solved problem? IEEE Transactions on Pattern Analysis and Machine Intel...

  3. [11]

    Medical Image Analysis89, 102918 (2023) 1

    Mazurowski, M.A., Dong, H., Gu, H., Yang, J., Konz, N., Zhang, Y.: Segment anything model for medical image analysis: an experimental study. Medical Image Analysis89, 102918 (2023) 1

  4. [12]

    IEEE Transactions on Neural Networks and Learning Systems (2025) 2, 5

    Wang, H., Guo, S., Ye, J., Deng, Z., Cheng, J., Li, T., Chen, J., Su, Y., Huang, Z., Shen, Y., et al.: Sam-med3d: a vision foundation model for general-purpose seg- mentation on volumetric medical images. IEEE Transactions on Neural Networks and Learning Systems (2025) 2, 5

  5. [13]

    Computerized Medical Imaging and Graphics p

    Zhang, Y., Liao, Q., Ding, L., Zhang, J.: Bridging 2d and 3d segmentation networks for computation-efficient volumetric medical image segmentation: An empirical study of 2.5 d solutions. Computerized Medical Imaging and Graphics p. 102088 (2022) 2

  6. [14]

    Medical Image Analysis106, 103733 (2025) 1 10 Yichi Zhanget al

    Zhang, Y., Lv, B., Xue, L., Zhang, W., Liu, Y., Fu, Y., Cheng, Y., Qi, Y.: Semisam+: Rethinking semi-supervised medical image segmentation in the era of foundation models. Medical Image Analysis106, 103733 (2025) 1 10 Yichi Zhanget al

  7. [15]

    Computers in Biology and Medicine p

    Zhang, Y., Shen, Z., Jiao, R.: Segment anything model for medical image seg- mentation: Current applications and future directions. Computers in Biology and Medicine p. 108238 (2024) 1

  8. [16]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    Zhang, Y., Xue, L., Zhang, W., Li, L., Liu, Y., Jiang, C., Cheng, Y., Qi, Y.: Seganypet:Universalpromptablesegmentationfrompositronemissiontomography images. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 21107–21116 (2025) 2, 4

  9. [17]

    Nature methods22(1), 166–176 (2025) 1

    Zhao, T., Gu, Y., Yang, J., Usuyama, N., Lee, H.H., Kiblawi, S., Naumann, T., Gao, J., Crabtree, A., Abel, J., et al.: A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities. Nature methods22(1), 166–176 (2025) 1

  10. [18]

    NPJ Digital Medicine8(1), 566 (2025) 2, 5

    Zhao, Z., Zhang, Y., Wu, C., Zhang, X., Zhou, X., Zhang, Y., Wang, Y., Xie, W.: Large-vocabulary segmentation for medical images with text prompts. NPJ Digital Medicine8(1), 566 (2025) 2, 5

  11. [19]

    Zhou, N., Zou, K., Ren, K., Luo, M., He, L., Wang, M., Chen, Y., Zhang, Y., Chen, H., Fu, H.: Medsam-u: Uncertainty-guided auto multi-prompt adaptation for reliable medsam. IEEE Transactions on Circuits and Systems for Video Technology (2025) 1 Title Suppressed Due to Excessiv...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.