{"id":"c613a099-4144-4939-8c93-6777680a0a5e","arxiv_id":"2412.12660","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SEG-SAM adds a semantic-aware decoder, LLM text summaries, and a mask-agreement loss to SAM for unified binary and semantic medical segmentation.","lead":"SEG-SAM adds semantic understanding to the Segment Anything Model so one medical segmentation model returns both a binary mask and a category label. It reports higher accuracy than existing SAM-based and task-specific methods, but the evaluation protocol and missing artifacts need closer scrutiny before the claim is accepted.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semantic-segmentation headline is not yet supported: the task-specific baselines (nnUNet, U-Mamba) are retrained as unified 2D slice classifiers, a setting foreign to their 3D design, and the closest semantic SAM baseline MaskSAM is absent from all comparisons.","rationale":"After reading SEG-SAM, the architecture is coherent: an independent semantic decoder with SO/CO tokens, LLM text enhancement, and a consistency loss are all plausible and the ablations show monotone gains. The binary comparison in Table 1 is the strongest part of the paper—all methods are retrained under the same prompting protocol, so the 73.15 vs 65.45 DSC improvement over SAM-Med2D is a controlled result. However, the abstract's second clause—'outperforms ... task-specific methods in semantic medical segmentation'—is the differentiator, and that is where the evidence is weakest. The task-specific baselines are not given their own task-specific setup; instead, nnUNet and U-Mamba, which are 3D and rely on volumetric context, are fed 2D slices from three axes and forced to solve a single 200-way unified segmentation problem. This is a category error in evaluation: it tests a unified model against specialized models that are denied their specialization. The large improvement over U-Mamba (75.28 vs 50.57 mDSC) is more plausibly explained by the baseline being out-of-domain than by SEG-SAM being genuinely superior. Omitting MaskSAM, the one existing SAM-based semantic segmentation method, leaves the 'state-of-the-art SAM-based' semantic comparison incomplete. The reader's weakest assumption about label uniqueness is real but secondary: because prompts select a specific mask during training, overlapping masks do not immediately make the objective ill-posed; the ambiguous case is only when a prompt point lies in an overlap region and the same pixel is supervised to two different classes. That could add noise, but the paper's main unresolved problem is that the semantic result is not compared on equal terms. I therefore recommend the verdict remain CONDITIONAL, with conditions extended to require native-baseline comparisons and MaskSAM.","tokens_in":15553,"tokens_out":10183,"duration_ms":87592,"concrete_test":"Run nnUNet and U-Mamba in their standard 3D per-dataset configuration on the original volumes of KiTS23, BTCV, and AMOS (the cross-dataset benchmarks), and report per-dataset Dice. Simultaneously, train MaskSAM under the same unified 2D point/box prompt protocol as SEG-SAM on Med2D-16M and report its mDSC. If SEG-SAM still exceeds the natively-trained 3D baselines and MaskSAM, the semantic claim is supported; if not, the headline must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2 compares SEG-SAM to nnUNet, UNETR, nnFormer, and U-Mamba after 'train these methods using the same setup as our method on the Med2D-16M dataset.' This means 3D-native architectures are retrained on 2D slices extracted along all three axes, with a single model covering ~200 classes across 8 modalities. That is not how these methods are designed or validated; nnUNet self-configures and expects volumetric context, so flattening it to 2D and merging all datasets strips its inductive biases. The resulting margin (75.28 vs 50.57 mDSC against U-Mamba) is too large to be credible evidence of superiority. In addition, MaskSAM (Ref. [35])—the only prior SAM-based semantic segmentation model cited—is discussed in Sec. 2 but omitted from every experiment, even though it is the most relevant baseline for the semantic claim. The paper's binary comparisons (Table 1) are reasonably controlled, and the ablations (Table 4) show each component helps, so the architectural idea has support. But the headline claim of outperforming task-specific semantic segmentation rests on an unfair protocol; until the baselines are run in their native 3D per-task setting (or on 2D models like U-Net trained per dataset) and MaskSAM is included, the semantic claim is unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SEG-SAM, a unified medical image segmentation model built on SAM. It adds a semantic-aware decoder (with a segmentation-oriented token for the prompted object and classification-oriented tokens for unprompted objects), a text-to-vision semantic enhancement module that injects LLM-generated medical category descriptions into the prompt tokens, and a cross-mask spatial alignment loss that encourages agreement between the semantic and binary decoders. Experiments on Med2D-16M report improved binary segmentation over SAM-based methods and improved semantic segmentation over four conventional architectures, with additional cross-dataset tests on KiTS23, BTCV, and AMOS.","tokens_in":15801,"tokens_out":3813,"duration_ms":35901,"significance":"If the semantic segmentation claim is substantiated, SEG-SAM would be a useful step toward a single promptable model that produces both binary and semantically labeled masks across modalities. The architectural idea is plausible and the ablations (Table 4 and Table 8) indicate that each proposed component contributes. The binary comparison in Table 1 is reasonably controlled, using the same prompt protocol for all SAM-based methods, and the plug-in experiments in Table 5 are informative. However, the headline semantic superiority over task-specific methods currently rests on an unsuitable evaluation protocol and on the omission of the closest prior semantic SAM baseline, so the paper's central claim is not yet supported as written.","major_comments":[{"comment":"The semantic comparison trains nnUNet, UNETR, nnFormer, and U-Mamba as 2D-slice models on a merged 200-class dataset, a setting that strips these 3D architectures of their native inductive biases (e.g., nnUNet's self-configuring pipeline and volumetric context). The resulting margin (75.28 vs 50.57 average mDSC against U-Mamba) is therefore not credible evidence of superiority over task-specific methods. Please evaluate these baselines in their native 3D per-task setting (or at least per-dataset 2D models such as U-Net trained separately on each dataset) and report those results. The abstract's phrase \"task-specific methods\" also does not match Table 2, where all methods are trained on the same multi-dataset merge.","section":"Sec. 4.2, Table 2"},{"comment":"MaskSAM (Ref. [35]), the only cited prior SAM-based semantic segmentation method, is discussed in Related Work but is absent from all experiments. Since the paper claims state-of-the-art semantic medical segmentation, the most relevant baseline must be included with the same training and prompting protocol, or the exclusion must be explicitly justified. The SAM-based semantic baselines that are included (Med2D*, MedSAM*, Med-SA*) are all created by inserting the proposed semantic-aware decoder into other methods, so they do not measure SEG-SAM against existing semantic SAM approaches.","section":"Sec. 2 and Sec. 4.2"},{"comment":"The semantic ground truth is defined as Ms = [Cs, Mb] under the assumption that every binary mask in every merged dataset carries exactly one unambiguous category. The paper does not analyze how many of the ~200 categories and ~16M masks violate this, for example masks that span multiple organs or cases where organ and lesion masks overlap spatially. Under such conditions, the semantic decoder receives contradictory labels and the cross-mask spatial alignment loss becomes ill-posed. Please report statistics on label ambiguity in Med2D-16M and, if ambiguous cases exist, describe how the supervision and the consistency loss handle them.","section":"Sec. 3.2.1, Eq. (10)"}],"minor_comments":[{"comment":"The equation concatenates prompt tokens tp into the text cross-attention query and then appends MLP(t'_text) to tp to form t'_p. This appears to duplicate prompt information; please clarify the exact role of tp inside the cross-attention and whether the final t'_p contains the original tp unchanged.","section":"Eq. (9)"},{"comment":"The IoU loss is written for masks Ms and Mb, but the paper does not specify how these continuous probability maps are binarized before computing the intersection and union. Please state the thresholding or soft-IoU variant used.","section":"Eq. (10)"},{"comment":"The training procedure says 'randomly select 5 foreground objects in each image'; if an image contains fewer than five foreground masks, the paper does not state how sampling is handled. Please clarify.","section":"Sec. 3.3"},{"comment":"The intro claims 'we are the first to introduce semantic learning into SAM,' which is contradicted by the cited MaskSAM work and by other promptable semantic segmentation models; please soften this claim. Also, the supplementary template feeds a highlighted example image to the LLM, so the generated text may describe that example rather than class-general characteristics; please state whether the same description is used for the entire class and discuss the effect on cross-dataset generalization.","section":"Introduction and Sec. 6.2"}],"recommendation":"major_revision","confidential_remarks":"The central architecture has merit and the binary results are credible, but the semantic claim needs a substantially reworked evaluation. I would suggest the editor require the authors to include native 3D per-task baselines and MaskSAM, and to address the label-ambiguity issue, before considering acceptance. The paper also overstates novelty relative to MaskSAM and the 'task-specific' naming is misleading."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the binary segmentation story is credible, but the semantic headline rests on a comparison that isn't fair, and the closest semantic SAM baseline is missing. Still a real architectural contribution worth a serious referee.\n\nWhat's actually new: SEG-SAM is, as far as I know, the first SAM-based model that keeps the original binary decoder and adds a separate semantic-aware decoder with a segmentation-oriented token plus classification-oriented tokens for unprompted objects, with LLM-generated text summaries feeding the prompt tokens and a cross-mask IoU loss tying the two decoders together. That assembly is coherent, and the ablations in Table 4 show each piece earning its keep: removing the consistency loss costs about 1.3 DSC and 0.5 mDSC, and the unprompted-object classification helps more than a full semantic segmentation branch, which actually hurts. The binary comparisons in Table 1 run all SAM variants under the same prompting protocol, so the four-to-six point average DSC gain over MedSAM and SAM-Med2D is believable. The plug-in experiments in Table 5 also support the claim that the decoder helps other SAM finetunes.\n\nNow the soft spots. The headline semantic comparison in Table 2 is the load-bearing problem. nnUNet, UNETR, nnFormer, and U-Mamba are designed as 3D volumetric models with self-configuration; here they are retrained as a single 2D slice classifier over roughly 200 classes from 8 modalities, which strips their inductive biases. The reported margins (75.28 vs 50.57 mDSC over U-Mamba) are therefore not credible evidence of superiority. The right comparison would be per-task native 3D training, or at least strong 2D models like U-Net trained per dataset. MaskSAM, the one prior SAM-based semantic segmentation method cited, is discussed in Sec. 2 but never appears in any table; that omission is hard to justify. I would also want error bars, since all numbers are single-run. And the assumption that every binary mask maps to exactly one semantic category (Sec. 3.2.1) is plausible for organ-level labels but untested for the roughly 200 categories in Med2D-16M; overlapping organs or lesion-within-organ masks would make the semantic ground truth and the consistency loss ill-posed. No code or full text descriptions are released, which slows verification.\n\nWho it's for: people working on unified promptable medical segmentation will get genuine ideas from the decoder design and the ablation methodology. It deserves peer review as a conditional accept: the architecture is worth publishing, but the semantic experiments need a fair protocol, error bars, and the missing baseline before the central claim is established.","headline":"Binary results are credible; the semantic headline isn't supported yet due to an unfair baseline protocol and a missing MaskSAM comparison—but the architecture deserves a serious referee.","tokens_in":16387,"tokens_out":2560,"would_cite":true,"duration_ms":23486,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SEG-SAM claims that adding a semantic-aware decoder, LLM text knowledge, and cross-mask alignment lets one SAM-based model outperform prior SAM-based methods in binary medical segmentation and task-specific methods in semantic medical…","keywords":["medical image segmentation","Segment Anything Model","semantic-aware decoder","text-to-vision semantic enhancement","cross-mask spatial alignment","unified segmentation","Med2D-16M","promptable segmentation"],"falsifier":"Evaluate the method on a dataset constructed to contain spatially overlapping masks of different categories (for example, a tumor region labeled inside a liver region) and compare the semantic decoder's per-category accuracy and the cross-mask alignment loss against a dataset of disjoint categories; if accuracy collapses on overlaps, the unique-category assumption behind $M_s=[C_s, M_b]$ is the breaking point.","tokens_in":15298,"feed_emoji":"🩻","tokens_out":6096,"duration_ms":52804,"temperature":0.7,"pith_summary":"SEG-SAM is a unified medical image segmentation model built on SAM that tries to show one promptable model can produce both accurate binary masks and correct semantic labels without task-specific retraining. It adds a separate semantic-aware decoder to SAM's original mask decoder, feeds it category knowledge extracted from a large language model, and enforces spatial agreement between the two decoders' outputs. On Med2D-16M across CT, MR, PET, and other modalities, the paper reports that this design outperforms prior SAM-based methods on binary segmentation (average DICE 73.15 with points, 81.46 with boxes) and also outperforms task-specific methods on semantic segmentation (average mDSC 75.28 and 80.90). The value of the claim, if right, is that a single model with one prompt can serve diverse clinical segmentation tasks while giving clinicians both the object boundary and its category.","feed_headline":"Adding semantics to SAM beats prior medical segmenters","feed_subtitle":"One prompt returns both binary mask and semantic label, improving average DICE over SAM-based baselines.","key_machinery":"The load-bearing mechanism is the semantic-aware decoder (SAWD), a parallel attention-based decoder that consumes a segmentation-oriented token plus prompt tokens and outputs a semantic mask, plus classification-oriented tokens that predict categories for unprompted objects. The text-to-vision semantic enhancement (T2VSE) module encodes LLM-generated category descriptions and folds them into the prompt tokens via a text summary token and cross-attention. A cross-mask spatial alignment loss, written as $1 - \\text{IoU}(\\hat{M}_s, \\hat{M}_b)$, supervises consistency between the semantic and binary masks. The SAWD is what lets the model keep SAM's original binary decoder while adding semantic learning without conflicting gradient signals.","core_discovery":"The central claim is that semantic supervision, added through a decoder that is architecturally separate from SAM's semantic-agnostic decoder, resolves the binary-versus-semantic conflict and improves both tasks. For a prompted object, a segmentation-oriented token decodes a semantic mask, while classification-oriented tokens predict categories of unprompted objects to supply context. LLM-generated text descriptions of each category are summarized into a token and appended to the visual prompt tokens, and a cross-mask spatial alignment loss pulls the semantic and binary masks toward each other. The paper reports that this combination beats both SAM-based unified models and conventional task-specific segmenters on the Med2D-16M benchmark and generalizes to unseen KiTS23, BTCV, and AMOS datasets.","pith_inferences":["A testable consequence the paper does not run: if a dataset contains overlapping masks of different categories (for example, a tumor region labeled inside a liver region), the assumed one-category-per-mask construction would give the semantic decoder contradictory labels; measuring per-category mDSC on such overlaps would show where the method's ceiling is.","The LLM descriptions are generated once from a fixed template. A cheap sensitivity experiment would replace them with category names only, or with descriptions from a different model, to see how much of the gain is genuinely linguistic versus just an extra token.","The cross-mask alignment loss forces spatial equality between semantic and binary masks; for inherently ambiguous boundaries this could bias one decoder. A soft alignment or uncertainty-weighted variant is a natural extension to test."],"forward_implications":["With one prompt, SEG-SAM can return both a binary mask and a semantic label, so a single deployed model could replace several task-specific segmenters.","The reported gains on binary segmentation imply that semantic context, including unprompted-object categories, is a usable signal for sharpening prompted-object boundaries, not just for labeling.","Because the semantic-aware decoder is a plug-in, the paper's ablation suggests other SAM-based medical segmenters can be upgraded to semantic prediction without full retraining from scratch.","The cross-dataset results imply that semantic knowledge learned from Med2D-16M transfers to new organs and modalities that were not in the training set."],"supporting_citations":[{"why":"Supplies the base SAM architecture, prompt encoder, original semantic-agnostic decoder, and the focal and Dice losses reused by SEG-SAM.","marker":"[22]"},{"why":"Provides the Med2D-16M dataset, the training protocol with five sampled objects per image, and the SAM-Med2D baseline that SEG-SAM builds on and compares against.","marker":"[12]"},{"why":"MedSAM is a key baseline for binary medical segmentation and one of the methods the semantic-aware decoder is plugged into.","marker":"[28]"},{"why":"Medical SAM Adapter serves as another SAM-based baseline and plug-in target for evaluating the semantic-aware decoder.","marker":"[34]"},{"why":"The large language model that generates the medical category descriptions used by the text-to-vision semantic enhancement scheme.","marker":"[1]"},{"why":"Supplies the semantic segmentation loss (cross entropy plus Dice) used to supervise the semantic mask prediction in the SAWD.","marker":"[11]"},{"why":"MaskSAM is a competing approach that also injects category information into SAM prompts, providing a point of comparison for semantic medical segmentation.","marker":"[35]"},{"why":"nnU-Net is a conventional task-specific baseline in the semantic medical segmentation comparison.","marker":"[18]"},{"why":"SAM2 is a baseline in the binary segmentation comparison, representing the updated SAM model family.","marker":"[30]"}],"fun_headline_variants":["Semantics injected into SAM top prior medical segmenters","One prompt, two masks: SEG-SAM outdoes SAM baselines","SEG-SAM weds semantic and binary masks for medical wins","Adding semantic sense to SAM lifts medical segmentation","Medical segmentation: SEG-SAM's semantic twist beats baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole semantic-supervision pipeline rests on the assumption that every binary mask in every training dataset carries exactly one unambiguous category label, so that semantic ground truth can be assembled as the category combined with the binary mask.","fun_headline_variants_meta":{"raw":{"variants":["Semantics injected into SAM top prior medical segmenters","One prompt, two masks: SEG-SAM outdoes SAM baselines","SEG-SAM weds semantic and binary masks for medical wins","Adding semantic sense to SAM lifts medical segmentation","Medical segmentation: SEG-SAM's semantic twist beats baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000403,"raw_usage":{"total_tokens":2088,"prompt_tokens":922,"completion_tokens":1166,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":1095}},"tokens_in":538,"tokens_out":1166,"duration_ms":8453,"temperature":1.0,"reasoning_tokens":1095,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:51:56.765735+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the method on a dataset constructed to contain spatially overlapping masks of different categories (for example, a tumor region labeled inside a liver region) and compare the semantic decoder's per-category accuracy and the cross-mask alignment loss against a dataset of disjoint categories; if accuracy collapses on overlaps, the unique-category assumption behind $M_s=[C_s, M_b]$ is the breaking point.","supporting_citations":[{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"Supplies the base SAM architecture, prompt encoder, original semantic-agnostic decoder, and the focal and Dice losses reused by SEG-SAM."},{"cited_title":"Segment anything in medical images","cited_arxiv_id":null,"evidence_quote":"MedSAM is a key baseline for binary medical segmentation and one of the methods the semantic-aware decoder is plugged into."},{"cited_title":"nnu-net: a self-configuring method for deep learning-based biomedical image segmen- tation","cited_arxiv_id":null,"evidence_quote":"nnU-Net is a conventional task-specific baseline in the semantic medical segmentation comparison."}],"review_version":1}