Pith. sign in

REVIEW 4 major objections 5 minor 24 references

A radiology report alone can now pick which organs to segment

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 14:33 UTC pith:YLBYCKPC

load-bearing objection A plausible modular design for report-driven segmentation, but the routing evaluation is self-referential and the extension claim is untested. the 4 major comments →

arxiv 2607.14116 v1 pith:YLBYCKPC submitted 2026-05-08 cs.CL cs.AIcs.CV

ReportMedSAM: Guiding Segmentation Through Radiology Reports

Anghong Du , Theodoros N. Arvanitis , Colin Watts , Alejandro F. Frangi , Le Zhang This is my paper
classification cs.CL cs.AIcs.CV
keywords radiology reportsmedical image segmentationconcept bankMixture-of-Expertssemantic collapsevision-language modelorgan segmentationreport-driven segmentation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ReportMedSAM aims to show that a free-form radiology report is enough to decide which organs to segment in a CT scan, with no manual prompt or predefined phrase list. It replaces discrete rule-based text extraction with a learned 'concept bank' of organ anchors in a frozen medical-language embedding space. During inference, the report is embedded, matched to these anchors by cosine similarity, and the top-matching organs activate organ-specific segmentation experts. If this works, clinical segmentation can be driven directly by the dictated report, new organs can be added without retraining old ones, and synonym substitutions should not hurt accuracy. The paper reports stable Dice scores across typical and synonymous descriptions and lower false activation than rule-based baselines.

Core claim

The paper claims that a report-driven segmentation framework can move from discrete extraction to a learnable semantic interface. The concept bank—one trainable embedding per organ, optimized in a frozen vision-language text space via contrastive learning and an inter-concept orthogonality penalty—maps a whole clinical narrative to a small set of target organs. Each activated organ then conditions a shared segmentation decoder through an organ-specific prompt generator, so the decoder stays frozen and only light projection layers are trained. The reported results on an abdominal CT benchmark indicate that this design keeps segmentation accuracy when clinical synonyms are substituted, and tha

What carries the argument

The concept bank: a set of trainable embedding vectors, one per anatomical structure, living in the embedding space of a frozen medical text encoder. It is the semantic interface between free-form reports and segmentation. Inter-concept orthogonality keeps organ anchors distinct (avoiding semantic collapse), while cosine similarity between the report embedding and each concept performs routing that activates organ-specific Mixture-of-Experts modules. This decoupling is what lets new concepts and experts be added without retraining existing components.

Load-bearing premise

The system assumes that keyword/regular-expression matching correctly identifies which parts of a report refer to each organ, and that the same matching is a good enough gold standard for evaluating routing.

What would settle it

Show a report that refers to an organ only indirectly (e.g., 'cirrhosis' without the word 'liver') and check whether the concept for liver is activated; or a report with explicit negation ('no splenic abnormality') and check whether the spleen expert fires anyway. Either would reveal whether routing truly understands clinical language or just pattern-matches keywords.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Free-form reports can replace manual prompts and rule-based extraction at inference, simplifying clinical integration.
  • Synonym robustness (e.g., 'renal' vs 'kidney') transfers to unseen clinical expressions through the frozen encoder's semantic manifold.
  • New organs can be added as parameter-isolated concept-expert pairs, enabling incremental scaling without catastrophic interference.
  • Routing confidence (threshold or Top-K) gives a tunable trade-off between missed organs and false activations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the concept bank is trained on reports from a single institution or language, domain shift in report style could still break routing; a multi-site evaluation would test this.
  • The same concept-bank routing could be repurposed for other report-conditioned tasks like lesion detection or report-driven image retrieval, since the mechanism is generic text-to-anchor matching.
  • A self-training loop could eventually replace the keyword-derived anchor supervision with routing confidence, removing the last rule-based component—but the paper does not demonstrate this.
  • The claim of 'seamless extension' rests on sequential training; a stronger stress test would add many new organs simultaneously with little data per expert.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ReportMedSAM, a framework that takes a free-form radiology report as input and predicts organ segmentation masks without manual prompts. A trainable concept bank is learned inside a frozen BiomedCLIP text space by contrastive alignment with organ-specific report excerpts extracted via keyword matching and regular expressions. During inference, the full report embedding is matched against these concepts to select target organs, which then activate organ-specific Mixture-of-Experts segmentation modules built on a frozen LiteMedSAM backbone. The method is evaluated on AbdomenAtlas 3.0 for liver, spleen, pancreas, and kidney. The paper reports competitive Dice scores under typical anatomical terms, claims robustness to synonyms, and asserts seamless parameter-isolated extension to new organs.

Significance. If the central claims are substantiated, the paper would offer a practically relevant alternative to discrete phrase-based prompting for report-driven segmentation, with a decoupled design that could allow incremental addition of new organs without retraining existing experts. The use of a frozen medical VLM and a learnable concept bank is a reasonable architecture, and the intended experiments address an important real-world gap. The paper would be strengthened by machine-checked or fully reproducible results, but code release is only promised. The key weaknesses are that the routing evaluation is potentially circular, the synonym-robustness claim is not actually tested with unseen expressions, and the scalability claim is asserted without experimental evidence. These issues are central to the paper's headline contributions, so the current version does not yet establish the stated claims.

major comments (4)
  1. [§2.2, §3, Table 2] The routing evaluation is self-referential. The concept bank is trained on organ-specific excerpts selected by the rule-based operators g_k ('keyword matching and regular expressions'), and the gold standard for the routing evaluation in Table 2 is derived from 'organ-specific text distilled from reports via keyword matching.' Because the same operators seed both the learned concepts and the routing ground truth, Table 2's FAR/Precision cannot validate free-form report understanding; the model could be learning to mimic g_k. The operators g_k are never specified, and no independent gold standard (e.g., expert annotation of report intent or organ mentions) is provided. This is load-bearing for the claim that the framework 'effectively interprets free-form reports.' Please specify g_k, test on reports where g_k is known to fail or use an external gold standard, and report agreement between
  2. [Table 1, §3, §4] The synonym-robustness claim is not demonstrated. In Table 1, the 'Typical' and 'Synonyms' rows for the proposed method report exactly identical Dice values for every organ (0.810/0.681/0.396/0.707, avg 0.647). This strongly indicates the tested synonyms are already within the vocabulary covered by the keyword/regex-derived concept bank, so the model sees no genuinely new input at test time. The paper claims robustness to 'unseen clinical expressions' (Section 4), but no experiment uses expressions outside the keyword-covered set. The identical numbers also make it impossible to assess variance. Please test on paraphrased or rephrased reports that are not generated from the same keyword list, or at least provide the exact synonym list and explain how it was constructed.
  3. [§3, Conclusion] The headline claim of 'seamless, non-interfering extension to novel clinical tasks' is asserted without experimental support. Section 3 states that results 'were obtained via a sequential incremental training protocol with frozen experts, empirically validating the framework's scalability,' but no interference metrics, adding schedule, or comparison with joint training are reported. The scalar 'scalability' is not visible in any table or figure. To support this claim, the authors should report Dice before and after adding a new expert, show that old organs are unaffected, and ideally compare incremental training with retraining all experts jointly. Without such data, the parameter-isolated extension is only a design property, not an empirical result.
  4. [Eq. (1), Table 1] The routing configuration used to obtain the main segmentation results in Table 1 is not specified. Equation (1) defines the selection operator σ with either threshold τ or Top-K′, and Table 2 reports Top-K variants (K=1..4), but Table 1 does not state which configuration (e.g., which τ or K) produced the reported Dice. This makes the central experiment non-reproducible and prevents assessment of how routing choices affect segmentation accuracy. Please report the exact routing configuration used for Table 1 and, if possible, provide sensitivity analysis of Dice with respect to K and τ.
minor comments (5)
  1. [Abstract/Introduction] The author list in the header ('J. Liu et al.') does not match the affiliation list ('Anghong Du et al.'). Please correct.
  2. [§2.3] The notation 'V(k)_n = MLP_k(Vn)' is confusing because Vn is already a set of spatial tokens; clarify whether MLP is applied per-token.
  3. [§2.4, Eq. (2)] The loss weights β_pos and β_neg are defined as 0.5 and 0.3, but also 'negative supervision scaled by a factor of 0.3' appears later. Clarify the relationship.
  4. [Fig. 2] The right panel is described as 'semantic consistency analysis' but the axes are not labeled in the figure caption. Please specify what C and D represent in panel (right).
  5. [Table 1] The baseline row labels are malformed ('RadGraphXL BioMedParse' appears as a single entry). Please split into separate rows for readability.

Circularity Check

1 steps flagged

Routing evaluation is self-referential: keyword matching builds the concept bank and also supplies the routing gold standard.

specific steps
  1. fitted input called prediction [Section 2.2 (concept bank training) and Section 3 (dataset/evaluation)]
    "we construct organ-associated report subsets using lightweight rule-based operators g_k(·), implemented via keyword matching and regular expressions ... Official masks and reports provide ground truth, with bounding boxes derived automatically from masks via tight enclosing boxes and organ-specific text distilled from reports via keyword matching."

    The concept bank is trained on excerpts selected by the same kind of rule-based operator g_k that generates the routing gold standard ('organ-specific text distilled from reports via keyword matching'). Consequently, Table 2's FAR/Precision is computed against keyword-derived labels, so the evaluation cannot distinguish genuine semantic understanding from learning to mimic g_k. The held-out split does not remove the circularity because the label function remains identical in kind to the training-data selector; the model is never tested against an independent human annotation of which organs the report intends.

full rationale

ReportMedSAM's pixel-level segmentation Dice is evaluated against official AbdomenAtlas 3.0 masks, so the segmentation quality claim is externally grounded. The circularity is confined to the report-to-organ routing evaluation. Section 2.2 trains the concept bank on excerpts chosen by 'keyword matching and regular expressions' (g_k), while Section 3 derives the routing ground truth from 'organ-specific text distilled from reports via keyword matching.' Therefore Table 2's FAR/Precision can only show agreement between the learned routing and the keyword operator, not independent free-form understanding. The 'synonym robustness' evidence is also limited: exact Dice equality across Typical/Synonyms is consistent with synonym-invariant routing, but the paper never specifies g_k, so the claimed robustness to 'unseen clinical expressions' is not demonstrated against vocabulary outside the operator. No author self-citation chain is load-bearing; the architecture is built on externally published LiteMedSAM/BiomedCLIP. The 'seamless, non-interfering extension' claim is asserted without interference measurements, but that is a missing-support issue, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper does not introduce new physical entities. Its free parameters are the concept embeddings and the routing threshold, and its key assumptions are that the frozen BiomedCLIP space already separates clinical organ language and that keyword-based report filtering provides trustworthy training and evaluation labels. The latter assumption is load-bearing and partially circular because the same kind of filtering labels both the concepts and the routing benchmark.

free parameters (3)
  • Organ concept embeddings C_k = not reported (learned 512-d vectors for liver, spleen, pancreas, kidney)
    Trainable vectors optimized in Stage I via InfoNCE-style contrastive learning; they are the semantic anchors used for target routing.
  • Routing threshold τ / Top-K K′ = τ swept from -0.05 to 0.25; K tested 1-4
    The selection operator σ is a free choice determining activated organs; no principled method for picking τ or K′ is given.
  • Loss weights and balance factors = β_pos=0.5, β_neg=0.3; λ_seg=1.0, λ_box=0.5, λ_bg=0.1; γ=0.05; temperature=0.07
    Hand-set hyperparameters that control the negative-branch suppression and prompt alignment; results depend on them though they are not the main claim.
axioms (4)
  • domain assumption Frozen BiomedCLIP text manifold preserves clinical synonym proximity and organ discriminability
    The whole routing relies on cosine similarity between a full-report embedding and learned organ concepts in this pre-trained space; no evaluation of the clinical text domain is provided beyond the reported experiments.
  • domain assumption Keyword matching and regular expressions produce correct organ-relevant report excerpts and correct target-organ labels
    Stage I training ('lightweight rule-based operators') and evaluation ('organ-specific text distilled from reports via keyword matching') both assume the rules capture organ semantics; this is the load-bearing premise behind the 'no discrete extraction' claim.
  • ad hoc to paper Inter-concept orthogonality (L_ortho) prevents semantic collapse and improves segmentation
    The paper shows reduced cross-correlation in Fig 2 but no ablation linking L_ortho to final Dice; it is an algorithmic assumption rather than a demonstrated mechanism.
  • domain assumption Frozen LiteMedSAM mask decoder can be steered by synthesized prompts generated from a concept-conditioned cross-attention module
    The segmentation branch assumes a lightweight generator can replace manual point/box prompts; the paper does not compare against alternative prompt-conditioning schemes.

pith-pipeline@v1.3.0-alltime-deepseek · 8066 in / 12506 out tokens · 123024 ms · 2026-08-02T14:33:49.556976+00:00 · methodology

0 comments
read the original abstract

Free-form radiology reports contain rich clinical descriptions, yet converting them for reliable segmentation remains challenging due to the inherent variability of natural language. Existing pipelines often rely on predefined organ phrases or brittle rule-based inference-time extraction, which limits their scalability to novel anatomical structures and makes them sensitive to linguistic variations. To address this, we propose ReportMedSAM, a report-driven framework that replaces discrete extraction with a learnable concept bank. By leveraging a frozen medical vision-language encoder (BiomedCLIP), we align organ-level concept embeddings with large-scale clinical corpora through contrastive learning, establishing mutually orthogonal semantic anchors. Our approach explicitly mitigates organ-level semantic collapse and ensures high robustness against diverse clinical synonyms (e.g., "renal" vs. "kidney" ). During inference, a clinical report is embedded and matched against this concept bank to dynamically activate task-specific Mixture-of-Experts (MoE) modules. This decoupled design allows new concepts and experts to be added without retraining existing components, providing a parameter-isolated extension mechanism while keeping previously learned experts unchanged. Evaluated on the AbdomenAtlas 3.0 dataset, ReportMedSAM effectively interprets free-form reports, achieves competitive segmentation accuracy, and demonstrates seamless, non-interfering extension to novel clinical tasks.

Figures

Figures reproduced from arXiv: 2607.14116 by Alejandro F. Frangi, Anghong Du, Colin Watts, Le Zhang, Theodoros N. Arvanitis.

Figure 1
Figure 1. Figure 1: Overview of our proposed segmentation framework. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Quantitative analysis of report-driven routing and the learned semantic [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 4 linked inside Pith

  1. [1]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Bassi, P.R., Li, W., Chen, J., Zhu, Z., Lin, T., Decherchi, S., Cavalli, A., Wang, K., Yang, Y., Yuille, A.L., et al.: Learning segmentation from radiology reports. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 305–315. Springer (2025)

    2025

  2. [2]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Bassi, P.R., Yavuz, M.C., Hamamci, I.E., Er, S., Chen, X., Li, W., Menze, B., Decherchi, S., Cavalli, A., Wang, K., et al.: Radgpt: Constructing 3d image-text tumor datasets. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 23720–23730 (2025)

    2025

  3. [3]

    arXiv preprint arXiv:2511.16719 (2025)

    Carion, N., Gustafson, L., Hu, Y.T., Debnath, S., Hu, R., Suris, D., Ryali, C., Alwala,K.V.,Khedr,H.,Huang,A.,etal.:Sam3:Segmentanythingwithconcepts. arXiv preprint arXiv:2511.16719 (2025)

    Pith/arXiv arXiv 2025

  4. [4]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Cheng, J., Fu, B., Ye, J., Wang, G., Li, T., Wang, H., Li, R., Yao, H., Cheng, J., Li, J., et al.: Interactive medical image segmentation: A benchmark dataset and baseline. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 20841–20851 (2025)

    2025

  5. [5]

    In: Findings of the Association for Computational Linguistics: ACL 2024

    Delbrouck,J.B.,Chambon,P.,Chen,Z.,Varma,M.,Johnston,A.,Blankemeier,L., Van Veen, D., Bui, T., Truong, S., Langlotz, C.: Radgraph-xl: A large-scale expert- annotated dataset for entity and relation extraction from radiology reports. In: Findings of the Association for Computational Linguistics: ACL 2024. pp. 12902– 12915 (2024)

    2024

  6. [6]

    Journal of Machine Learning Research 23(120), 1–39 (2022)

    Fedus,W.,Zoph,B.,Shazeer,N.:Switchtransformers:Scalingtotrillionparameter models with simple and efficient sparsity. Journal of Machine Learning Research 23(120), 1–39 (2022)

    2022

  7. [7]

    ICLR1(2), 3 (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. ICLR1(2), 3 (2022)

    2022

  8. [8]

    In: Proceedings of the AAAI conference on artificial intelligence

    Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 590–597 (2019)

    2019

  9. [9]

    Scientific data6(1), 317 (2019) 10 J

    Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data6(1), 317 (2019) 10 J. Liu et al

    2019

  10. [10]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023)

    2023

  11. [11]

    In: Findings of the Association for Computational Linguistics: EACL 2024

    Lewis, M., Nayak, N., Yu, P., Merullo, J., Yu, Q., Bach, S., Pavlick, E.: Does clip bind concepts? probing compositionality in large image models. In: Findings of the Association for Computational Linguistics: EACL 2024. pp. 1487–1500 (2024)

    2024

  12. [12]

    Advances in Neural Information Processing Systems36, 28541–28564 (2023)

    Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., Gao, J.: Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems36, 28541–28564 (2023)

    2023

  13. [13]

    Nature communications15(1), 654 (2024)

    Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature communications15(1), 654 (2024)

    2024

  14. [14]

    arXiv preprint arXiv:2412.16085 (2024)

    Ma, J., Li, F., Kim, S., Asakereh, R., Le, B.H., Nguyen-Vu, D.K., Pfefferle, A., Wei, M., Gao, R., Lyu, D., et al.: Efficient medsams: Segment anything in medical images on laptop. arXiv preprint arXiv:2412.16085 (2024)

    Pith/arXiv arXiv 2024

  15. [15]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  16. [16]

    Journal of the American Medical Informatics Association17(5), 507–513 (2010)

    Savova, G.K., Masanz, J.J., Ogren, P.V., Zheng, J., Sohn, S., Kipper-Schuler, K.C., Chute, C.G.: Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications. Journal of the American Medical Informatics Association17(5), 507–513 (2010)

    2010

  17. [17]

    Nejm Ai1(3), AIoa2300138 (2024)

    Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al.: Towards generalist biomedical ai. Nejm Ai1(3), AIoa2300138 (2024)

    2024

  18. [18]

    In: The Fourteenth International Conference on Learning Representa- tions (2026)

    Wang, H., Xu, X., Wang, L., Pu, B., Li, X.: Atttok: Marrying attribute tokens with generative pre-trained vision-language models towards medical image under- standing. In: The Fourteenth International Conference on Learning Representa- tions (2026)

    2026

  19. [19]

    In: Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing

    Wang, Z., Wu, Z., Agarwal, D., Sun, J.: Medclip: Contrastive learning from un- paired medical images and text. In: Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing. pp. 3876–3887 (2022)

    2022

  20. [20]

    Advances in Neural Information Processing Systems37, 137432–137473 (2024)

    Wu, R., Papyan, V.: Linguistic collapse: Neural collapse in (large) language models. Advances in Neural Information Processing Systems37, 137432–137473 (2024)

    2024

  21. [21]

    arXiv preprint arXiv:2303.00915 (2023)

    Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., et al.: Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915 (2023)

    Pith/arXiv arXiv 2023

  22. [22]

    arXiv preprint arXiv:2405.12971 (2024)

    Zhao, T., Gu, Y., Yang, J., Usuyama, N., Lee, H.H., Naumann, T., Gao, J., Crab- tree, A., Abel, J., Moung-Wen, C., et al.: Biomedparse: a biomedical foundation model for image parsing of everything everywhere all at once. arXiv preprint arXiv:2405.12971 (2024)

    Pith/arXiv arXiv 2024

  23. [23]

    International journal of computer vision130(9), 2337–2348 (2022)

    Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. International journal of computer vision130(9), 2337–2348 (2022)

    2022

  24. [24]

    Advances in neural information processing systems36, 19769–19782 (2023)

    Zou, X., Yang, J., Zhang, H., Li, F., Li, L., Wang, J., Wang, L., Gao, J., Lee, Y.J.: Segment everything everywhere all at once. Advances in neural information processing systems36, 19769–19782 (2023)

    2023