Pith. sign in

REVIEW 3 major objections 5 minor 23 references

Annotation-Efficient Task Guidance for Medical Segment Anything

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SAM-Mix beats fully supervised U-Net on liver segmentation using only 50 labeled CT slices and 10 training epochs.

desk verdict The proposed CAM-guided SAM pipeline is new and the numbers are consistent, but the headline gains are not attributable to the proposed mechanism because all SAM-based models start from pretrained weights while the fully supervised baselines start from scratch. read the letter →

arxiv 2412.08575 v1 pith:GTTI7THS submitted 2024-12-11 cs.CV

Tyler Ward , Abdullah-Al-Zubaer Imran This is my paper · ORCID
classification cs.CV
keywords medicalimagesegmentationSegmentAnythingModelGradCAMmultitasklearningsemi-supervisedliverCTpromptgenerationLoRA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SAM-Mix claims that a medical image segmenter built on the Segment Anything Model can be trained with almost no dense annotation: an auxiliary classifier trained only on slice-level labels (liver present or absent) produces GradCAM heatmaps, which are thresholded into bounding boxes that automatically prompt SAM. With just 50 labeled slices—0.04% of the LiTS training labels—and 10 epochs, this end-to-end framework reaches a Dice score of 0.948 on in-domain liver segmentation, beating a fully supervised U-Net trained on the whole dataset, and a 25.4 percentage-point higher Dice than U-Net on a cross-domain test set. The paper's point is that classification-guided attention can substitute for dense manual annotation in prompting a foundation segmenter, and that this guidance improves both accuracy and generalization.

What carries the argument

The load-bearing mechanism is the CAM-to-prompt connection: a ResNet-38 classifier, trained with focal loss on coarse labels $C=\max(y_i)$ extracted from the same masks, produces GradCAMs $g=\sum_i w_i^{(c)} F_{\mathrm{last},i}$; thresholding at $\tau = 0.5\cdot\max(g)$ yields a binary mask whose bounding box becomes SAM's prompt. The prompt encoder turns this box into sparse embeddings that steer SAM's mask decoder, while a LoRA adapter (rank 8) inserted into SAM's ViT attention makes fine-tuning parameter-efficient. The classifier and segmenter are optimized jointly (focal loss plus Dice loss), so the prompt generator is trained by the same objective as the segmentation.

What would settle it

On the LiTS test set, compute the intersection-over-union and center-of-mass error between each GradCAM-derived bounding box and the liver ground-truth mask, then correlate them with per-slice Dice. If a substantial fraction of boxes fail to enclose the liver while SAM-Mix still predicts high Dice, the claimed guidance mechanism is contradicted; if boxes are accurate but Dice is low, the segmentation branch would be the limiting factor.

Watch

Extended reading notes

Core claim

The central claim is that connecting an auxiliary classification task to the SAM segmentation branch through automatically generated GradCAM-based bounding-box prompts yields a semi-supervised multitask learner that outperforms fully supervised baselines. When the framework (SAM-Mix) is trained for 90% fewer epochs on only 50 labeled 2D slices, it improves in-domain liver Dice by 5.1 percentage points over the best fully supervised baseline (U-Net) and achieves a 25.4 percentage-point Dice improvement on the cross-domain TotalSegmentator test set, with a 62.26% lower Hausdorff distance. The same configuration with only 5 labeled slices still reaches Dice 0.919 in-domain and 0.807 cross-domain. The authors argue this demonstrates annotation-efficient task guidance: image-level labels, via CAMs, provide enough spatial information to prompt SAM effectively while the end-to-end training keeps the prompts aligned with the segmentation objective.

Load-bearing premise

The method assumes that the auxiliary classifier's GradCAM, thresholded at half its maximum activation, produces a bounding box that reliably encloses the liver; if the CAM fires on non-liver cues or misses the organ, the prompts mislead SAM and the reported gains depend on that luck.

Editorial extensions

If this is right

  • Medical image segmentation can be performed accurately with a tiny fraction of the manual annotations, since coarse image-level labels suffice to make SAM segment effectively.
  • The same framework transfers across scanners and protocols: the 25.4 percentage-point cross-domain improvement implies that CAM-guided prompts improve generalization over full-supervision training.
  • Training cost drops dramatically: 10 epochs versus 100 for the fully supervised baselines, and LoRA limits the number of trainable parameters, making SAM fine-tuning feasible in low-resource settings.
  • The approach removes manual or semi-automated prompt generation for SAM, automating what was previously a human-in-the-loop bottleneck.
  • Even at 5 labeled slices, the method remains competitive (Dice 0.919 in-domain), suggesting that the guidance is robust to extreme label scarcity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the mechanism would measure how often the GradCAM-derived bounding box actually encloses the liver; the paper does not report this, so the causal role of the prompts remains an inference from the end-to-end Dice.
  • The same task guidance could generalize from bounding-box to point or mask prompts, or to multi-organ segmentation, where a single classifier would need to localize several structures at once—likely stressing the thresholded-CAM assumption.
  • Because the coarse labels $C=\max(y_i)$ are derived from the same masks used for segmentation, the method still needs a small seed set of full masks; a further extension is to test whether labels from a different modality or noisy annotations still yield the benefit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SAM-Mix, a multitask framework that combines an auxiliary ResNet-38 classifier with SAM for liver segmentation in abdominal CT. Slice-level labels are manufactured from the segmentation masks via C = max(y_i), GradCAM maps are thresholded to produce bounding boxes, and these boxes prompt a SAM image encoder adapted with LoRA. The method is trained end-to-end on 5, 50, or 100 labeled slices and evaluated on LiTS (in-domain) and TotalSegmentator (cross-domain). The authors report that SAM-Mix-50 outperforms fully supervised U-Net trained on the full LiTS training set, with a Dice improvement of about 5 percentage points in-domain and 25 percentage points cross-domain. The paper also compares against a two-stage SAM-PP pipeline and a limited-supervision MultiMix baseline.

Significance. If the reported gains are robust, the method would be a practically useful demonstration that a promptable foundation model can be steered by cheaply manufactured GradCAM boxes, substantially reducing annotation burden. The paper uses public datasets, reports five-seed averages, and provides code, which are strengths. The core idea is simple and the experimental design is internally consistent across Table 1 and Table 2. However, the central comparison is confounded by the use of pretrained SAM weights with LoRA against randomly initialized baselines, and the missing controlled baseline (SAM-LoRA with ground-truth prompts) prevents attributing the improvement to the proposed task-guidance mechanism. The headline arithmetic also contains errors. These issues are load-bearing for the paper's central claim and need to be resolved before the contribution can be assessed.

major comments (3)
  1. [§3.2, Table 1] The central comparison is confounded by the use of a pretrained foundation model. The Implementation Details state that all fully supervised baselines are randomly initialized and trained for 100 epochs, whereas the SAM-based models are initialized from SAM's pretrained weights and trained for only 10 epochs. The reported gain of SAM-Mix-50 over U-Net (Dice 0.948 vs 0.897) could therefore be driven by the pretrained representation plus LoRA adaptation rather than by the proposed GradCAM-guided prompts. A control experiment that fine-tunes SAM with LoRA on the same 50 labeled slices using ground-truth bounding-box prompts (or full-image prompts) is missing; without it, the central claim that the task guidance is responsible for the improvement is unsupported.
  2. [§3.3, abstract] The headline numerical claims are internally inconsistent and arithmetically incorrect. The abstract states a 5.1% Dice improvement, Section 3.3 states 5.9%, and the difference in Table 1 is 0.948 - 0.897 = 0.051 (5.1 percentage points). Additionally, 50/11437 is 0.44%, not the stated 0.04%. These numbers must be corrected and made consistent across the abstract, the results section, and the tables.
  3. [§2, Eq. (5)-(6)] No direct evidence is provided that the automatically generated bounding boxes are reliable. The auxiliary classifier is trained on labels C = max(y_i) that are derived from the same ground-truth masks, and the GradCAM threshold tau = 0.5 * max(g) is a free parameter (omega) that is never varied. Reporting prompt-box quality (e.g., box IoU or recall of the liver ROI) and an ablation over omega would substantiate the mechanism claim that classification-guided attention, rather than the SAM backbone itself, drives the result. This evidence is load-bearing because the entire novelty rests on the CAM-to-prompt connection.
minor comments (5)
  1. [§1, §3.2] The method is described as semi-supervised throughout, but no unlabeled data are used; all training examples carry segmentation masks (5, 50, or 100 slices). The terminology should be changed to limited-supervision or few-shot supervised to avoid confusion with standard semi-supervised learning.
  2. [§2, Eq. (3)] The focal loss hyperparameters alpha and gamma in Eq. (3) are not specified in Implementation Details; please provide the chosen values.
  3. [§3.2, Data] The preprocessing step 'extract only the middle 30% slices' should state the resulting number of slices per scan and clarify whether the 5/50/100 labeled slices are drawn from this filtered set.
  4. [Table 1, Table 2] The supervision column is incomplete for several rows: nnU-Net, TransU-Net, and the SAM-PP/SAM-Mix variants do not have explicit 'Full' or 'Limited' entries, which makes the table harder to interpret. Please make the supervision status explicit for every row.
  5. [§5, References] References [9] and [13] are both from the same group and describe the same method in different venues; please distinguish them at first citation and ensure the author lists are complete and consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SAM-Mix's reported gains rest on held-out test evaluations; self-citations to MultiMix are inspirational and not load-bearing.

full rationale

The paper contains no derivation that reduces to its own inputs. The auxiliary classification labels C = max(y_i) are manufactured from the same ground-truth segmentation masks, but this is a standard weak-supervision label transformation rather than a circular prediction: at inference the classifier and GradCAM prompt generation operate on images without masks, and the reported Dice/Hausdorff results are evaluated on held-out test slices. The citations to MultiMix and S2C are used for inspiration and context, not as a proof or uniqueness theorem, and no fitted parameter is renamed as a prediction. The experimental comparison is potentially confounded because SAM-based models start from pretrained SAM weights with LoRA while fully supervised baselines are randomly initialized, but that is a benchmarking/correctness concern, not circularity. No circular step can be exhibited, so the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper does not introduce new physical or mathematical entities. The central method rests on hand-set hyperparameters (GradCAM threshold, LoRA rank) and domain assumptions about label-derived CAMs and slice selection.

free parameters (2)
  • GradCAM threshold factor omega (tau = 0.5) = 0.5
    The threshold that converts GradCAM activations into binary prompt masks is set by hand ('For simplicity... set the threshold tau to 0.5', Section 2), and it directly controls the bounding box prompts.
  • LoRA rank r = 8
    The rank of the low-rank adaptation is chosen as 8 (Section 2), a hyperparameter that controls adapter capacity and is not derived from data.
assumptions (4)
  • domain assumption The slice-level class label C = max(y_i) fully captures liver presence, so a classifier trained on these labels yields GradCAMs that localize the liver.
    Invoked in Section 2, Auxiliary Classification; if the classifier learns shortcuts (e.g., image contrast) instead of liver location, the automatic prompts are unreliable.
  • domain assumption The middle 30% of slices in each CT scan is representative of the segmentation task.
    Invoked in Section 3.2 Data; this restriction is applied before splitting, so test performance reflects only the chosen slice range.
  • domain assumption SAM's pretrained weights transfer to abdominal CT and LoRA rank 8 is sufficient to adapt them.
    Invoked in Section 2 Segmentation and Section 3.2; the strong baselines are randomly initialized while SAM-Mix inherits foundation-model features.
  • domain assumption Dice loss and binary focal loss with no explicit weighting combine without degrading either branch.
    The total training objective is implied by Equations 3 and 9; no loss-weighting schedule is reported, so the balance is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Annotation-Efficient Task Guidance for Medical Segment Anything." pith.science (2026). https://pith.science/paper/GTTI7THS

@misc{pith2026241208575,
  author       = {Pith},
  title        = {Pith review of: Annotation-Efficient Task Guidance for Medical Segment Anything},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GTTI7THS}},
  note         = {Machine review of arXiv:2412.08575}
}
read the original abstract

Medical image segmentation is a key task in the imaging workflow, influencing many image-based decisions. Traditional, fully-supervised segmentation models rely on large amounts of labeled training data, typically obtained through manual annotation, which can be an expensive, time-consuming, and error-prone process. This signals a need for accurate, automatic, and annotation-efficient methods of training these models. We propose SAM-Mix, a novel multitask learning framework for medical image segmentation that uses class activation maps produced by an auxiliary classifier to guide the predictions of the semi-supervised segmentation branch, which is based on the SAM framework. Experimental evaluations on the public LiTS dataset confirm the effectiveness of SAM-Mix for simultaneous classification and segmentation of the liver from abdominal computed tomography (CT) scans. When trained for 90% fewer epochs on only 50 labeled 2D slices, representing just 0.04% of the available labeled training data, SAM-Mix achieves a Dice improvement of 5.1% over the best baseline model. The generalization results for SAM-Mix are even more impressive, with the same model configuration yielding a 25.4% Dice improvement on a cross-domain segmentation task. Our code is available at https://github.com/tbwa233/SAM-Mix.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 17 canonical work pages

  1. [1]

    INTRODUCTION Segmentation of targets such as organs, lesions, and tumors is one of the most important tasks in the downstream medical imaging pipeline, playing a vital role in enhancing the diagnos- tic and monitoring capabilities of clinicians. Traditional U-Net [1]-based architectures have long been one of the most popular and reliable methods for perfo...

    work page Pith review arXiv 2024

  2. [2]

    , xN } is a set of abdominal CT slices and Y = {y1, y2,

    METHODS: SAM-MIX To formulate the problem, we assume a data distribution p(X , Y ) over D where X = {x1, x2, . . . , xN } is a set of abdominal CT slices and Y = {y1, y2, . . . , yN } is the set of corresponding ground truth segmentation maps. A networkGϕ is created with parameters ϕ such that Gϕ(X ) → Y . To aid in segmentation, an auxiliary classificati...

    work page

  3. [3]

    Data We validate our proposed SAM-Mix method with the pub- licly available Liver Tumor Segmentation (LiTS) Benchmark [16]

    EXPERIMENTAL EV ALUATION 3.1. Data We validate our proposed SAM-Mix method with the pub- licly available Liver Tumor Segmentation (LiTS) Benchmark [16]. We split the dataset into training (100 scans), validation (5 scans), and test (25 scans) sets, with 11,437, 1,139, and 4,827 slices respectively in each set. The binary labels are then generated based on...

    work page

  4. [4]

    Through an innovative auxiliary classification prediction, segmentation prompts are generated automatically

    CONCLUSIONS We have presented a novel multitask learning approach to annotation-efficient medical image segmentation (SAM-Mix). Through an innovative auxiliary classification prediction, segmentation prompts are generated automatically. This annotation-efficient task guidance enables SAM to accurately segment liver from abdominal CT images requiring no ma...

    work page

  5. [5]

    U-Net: Convolutional networks for biomedical im- age segmentation,

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox, “U-Net: Convolutional networks for biomedical im- age segmentation,” in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th In- ternational Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, 2015, pp. 234– 241

    work page 2015

  6. [6]

    Segment anything,

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick, “Segment anything,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4015–4026

    work page 2023

  7. [7]

    SPPNet: A single- point prompt network for nuclei image segmentation,

    Qing Xu, Wenwei Kuang, Zeyu Zhang, Xueyao Bao, Haoran Chen, and Wenting Duan, “SPPNet: A single- point prompt network for nuclei image segmentation,” in International Workshop on Machine Learning in Medical Imaging. Springer, 2023, pp. 227–236

    work page 2023

  8. [8]

    PP-SAM: Per- turbed prompts for robust adaption of segment anything model for polyp segmentation,

    Md Mostafijur Rahman, Mustafa Munir, Debesh Jha, Ulas Bagci, and Radu Marculescu, “PP-SAM: Per- turbed prompts for robust adaption of segment anything model for polyp segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 4989–4995

    work page 2024

Show all 23 references
  1. [9]

    EVF-SAM: Early vision-language fu- sion for text-prompted segment anything model,

    Yuxuan Zhang, Tianheng Cheng, Rui Hu, Lei Liu, Heng Liu, Longjin Ran, Xiaoxin Chen, Wenyu Liu, and Xing- gang Wang, “EVF-SAM: Early vision-language fu- sion for text-prompted segment anything model,” arXiv preprint arXiv:2406.20076, 2024

    2024 arXiv

  2. [10]

    MaskSAM: Towards auto-prompt SAM with mask 5 classification for medical image segmentation,

    Bin Xie, Hao Tang, Bin Duan, Dawen Cai, and Yan Yan, “MaskSAM: Towards auto-prompt SAM with mask 5 classification for medical image segmentation,” arXiv preprint arXiv:2403.14103, 2024

    2024 arXiv

  3. [11]

    Segment anything in medical images,

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang, “Segment anything in medical images,” Nature Communications, vol. 15, no. 1, pp. 654, 2024

    2024

  4. [12]

    Self-supervised, semi-supervised, multi- context learning for the combined classification and seg- mentation of medical images,

    Abdullah-Al-Zubaer Imran, Chao Huang, Hui Tang, Wei Fan, Yuan Xiao, Dingjun Hao, Zhen Qian, and Demetri Terzopoulos, “Self-supervised, semi-supervised, multi- context learning for the combined classification and seg- mentation of medical images,” in Proceedings of the AAAI Con...

    2020

  5. [13]

    MultiMix: sparingly- supervised, extreme multitask learning from medical im- ages,

    Ayaan Haque, Abdullah-Al-Zubaer Imran, Adam Wang, and Demetri Terzopoulos, “MultiMix: sparingly- supervised, extreme multitask learning from medical im- ages,” in 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 2021, pp. 693–696

    2021

  6. [14]

    Segment anything model (SAM) enhanced pseudo labels for weakly supervised semantic segmentation,

    Tianle Chen, Zheda Mai, Ruiwen Li, and Wei-lun Chao, “Segment anything model (SAM) enhanced pseudo labels for weakly supervised semantic segmentation,” arXiv preprint arXiv:2305.05803, 2023

    2023 arXiv

  7. [15]

    Weakly-supervised semantic segmentation with image-level labels: from traditional models to foundation models,

    Zhaozheng Chen and Qianru Sun, “Weakly-supervised semantic segmentation with image-level labels: from traditional models to foundation models,” arXiv preprint arXiv:2310.13026, 2023

    2023 arXiv

  8. [16]

    From SAM to CAMs: Exploring segment anything model for weakly supervised semantic segmentation,

    Hyeokjun Kweon and Kuk-Jin Yoon, “From SAM to CAMs: Exploring segment anything model for weakly supervised semantic segmentation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 19499–19509

    2024

  9. [17]

    Generalized multi-task learning from substantially unlabeled multi-source med- ical image data,

    Ayaan Haque, Abdullah-Al-Zubaer Imran, Adam Wang, Demetri Terzopoulos, et al., “Generalized multi-task learning from substantially unlabeled multi-source med- ical image data,” Machine Learning for Biomedical Imaging, vol. 1, no. October 2021 issue, pp. 1–25, 2021

    2021

  10. [18]

    Wider or deeper: Revisiting the resnet model for visual recognition,

    Zifeng Wu, Chunhua Shen, and Anton Van Den Hengel, “Wider or deeper: Revisiting the resnet model for visual recognition,” Pattern Recognition, vol. 90, pp. 119–133, 2019

    2019

  11. [19]

    LoRA: Low-rank adaptation of large language models,

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, “LoRA: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  12. [20]

    The liver tumor segmentation benchmark (LiTS),

    Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene V orontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Sze- skin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, Fabian Loh¨ofer, Julian Walter Holch, Wieland Sommer, Felix Hofmann, Alexandre Hostettler, Naama L...

    2023

  13. [21]

    TotalSeg- mentator: Robust segmentation of 104 anatomic struc- tures in ct images,

    Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, Michael Bach, and Martin Segeroth, “TotalSeg- mentator: Robust segmentation of 104 anatomic struc- tures in ct ima...

    2023

  14. [22]

    nnU-Net: a self- configuring method for deep learning-based biomedical image segmentation,

    Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein, “nnU-Net: a self- configuring method for deep learning-based biomedical image segmentation,” Nature Methods, vol. 18, no. 2, pp. 203–211, 2021

    2021

  15. [23]

    Tran- sUNet: Rethinking the U-Net architecture design for medical image segmentation through the lens of trans- formers,

    Jieneng Chen, Jieru Mei, Xianhang Li, Yongyi Lu, Qi- hang Yu, Qingyue Wei, Xiangde Luo, Yutong Xie, Ehsan Adeli, Yan Wang, Matthew P. Lungren, Shaoting Zhang, Lei Xing, Le Lu, Alan Yuille, and Yuyin Zhou, “Tran- sUNet: Rethinking the U-Net architecture design for medical image...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.