{"id":"25f2decc-8d01-4a76-8825-33f12d8d3d5c","arxiv_id":"2412.05605","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"RefSAM3D combines 3D adapters, text prompts, and hierarchical cross-attention to adapt SAM for 3D medical segmentation, claiming state-of-the-art Dice scores and CT-to-MRI zero-shot transfer.","lead":"This paper adapts the 2D Segment Anything Model to 3D CT and MRI scans by adding 3D adapters and text prompts, and reports higher Dice scores than several baselines on tumor and organ segmentation tasks. A smart generalist should read it because the method claims strong zero-shot transfer from CT to MRI, which would be valuable in medical imaging if the results hold.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table II's per-dataset text prompts embed acquisition statistics (spacing, slice counts, dimensions), while the Table IV ablation shows removing the text branch costs 16 Dice points on BTCV; the reported SOTA and zero-shot gains may be dataset-identity conditioning, not segmentation ability.","rationale":"The paper's central claim is empirical: RefSAM3D beats a strong baseline set (nnU-Net, Swin-UNETR, MA-SAM, 3DSAM-adapter) on four tumor benchmarks, BTCV multi-organ segmentation, and zero-shot AMOS22 CT/MRI. The adaptation machinery (3D patch embedding, adapter, hierarchical cross-attention, upsampled mask decoder) is a coherent PETL design, and cross-modal referring segmentation is a legitimate idea; the concern is not the concept but the prompt content and the absence of controls. For the central claim to hold, the text prompts must contribute semantics rather than dataset identity. Two pieces of evidence break that assumption. First, Table II: every prompt is a per-dataset constant embedding acquisition statistics (spacing, dimensions, slice counts, modality). Through the trainable MLP and cross-attention, this constant can act as a dataset-ID code. Second, Table IV: removing the text branch costs 16.0 Dice points on BTCV, an order of magnitude larger than RefSAM3D's entire edge over MA-SAM (+1.1 points); a purely semantic prompt should not carry that much weight for the same 13 organs. The zero-shot claims are additionally ambiguous because the paper never states which prompt is used at test time; using the AMOS22 prompt (which names 'MRI' and '15 organs') would confound the cross-modality generalization test. I considered the strongest counter-argument: the frozen CLIP text encoder may tokenize numbers into embeddings that carry little precise statistical content, so the statistics might be nearly inert and the organ names dominate. That is possible, which is why the concrete test is a prompt-sanitized rerun rather than a categorical assertion; but even organ-set and modality words in a per-dataset prompt constitute dataset tagging, and the ablation magnitude indicates the text branch is doing far more than generic semantic conditioning. Corroborating integrity signals: Section IV.A states only CT was used for MM-WHS, while Table II, Figure 5, and the prose report MRI cardiac results; 'same as xxx []' and 'Table ??' placeholders remain; Eq. 7's stated shapes are mismatched (attention weights over L text tokens multiplied by values over C image channels). There is no released code, no formal verification, and no independent check; the strongest claim therefore rests entirely on the paper's own unverifiable tables. Since the reader's low-confidence REJECT is consistent with this analysis, my read does not change the verdict.","tokens_in":16723,"tokens_out":18607,"duration_ms":163770,"concrete_test":"Retrain each dataset variant with the Table II prompts stripped of all acquisition statistics and dataset names, keeping only the semantic target description (e.g., 'Segment kidneys, tumors, and cysts in CT images'), and re-run every reported benchmark plus the BTCV-trained zero-shot evaluation on AMOS22, using only the BTCV prompt at test time. If BTCV Dice falls toward the 72.3 w/o-text ablation level, or the AMOS22 MRI zero-shot figure drops toward the nnU-Net/Swin-UNETR baseline range (12-15%), the reported results were inflated by dataset-identity conditioning. As a secondary confirmation from the same trained models, score identical test volumes under their dataset-specific prompt versus a generic organ-name-only prompt; a shift of more than about 2 Dice points with inputs held fixed would confirm that the text branch, not the segmentation model, carries the dataset signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that RefSAM3D consistently outperforms strong baselines and generalizes zero-shot (Sec. IV.B–C). For that claim to hold, the hand-designed text prompts must supply legitimate semantic guidance. Table II shows they supply much more: each prompt is a per-dataset constant containing acquisition statistics, e.g., KiTS21 'spacing (0.5, 0.44, 0.44) to (5.0, 1.04, 1.04), dimensions (29, 512, 512) to (1059, 512, 796)', MSD pancreas 'resolution 512x512, slices 37 to 751', BTCV 'slice thickness 2.5-5.0 mm, in-plane resolution 0.54x0.54 to 0.98x0.98 mm^2', and AMOS22 'CT and MRI images... varying modalities and resolutions'. Because the prompt is identical for every sample of a dataset and flows through the trainable cross-modal MLP (Eq. 3) and hierarchical cross-attention (Eqs. 6-8), the text branch can act as a dataset-identity embedding that the decoder learns to lean on. The internal ablation is consistent with this: Table IV shows removing the text prompt drops BTCV Dice from 88.3 to 72.3 (-16.0), a swing far larger than RefSAM3D's entire margin over MA-SAM (88.3 vs 87.2). The zero-shot numbers (85.7% AMOS22 CT, 63.2% MRI after BTCV-only training) are uninterpretable unless the test-time prompt is specified: if the AMOS22 prompt (which names both modalities and '15 organs') is used, the model is told the target domain, so the cross-modality result is not a clean generalization test. No code or detailed protocol is released, so the numbers cannot be independently reproduced. Corroborating defects: Sec. IV.A says only CT was used for MM-WHS, yet Table II, Figure 5, and the prose report MRI cardiac results; 'same as xxx []' and 'Table ??' placeholders are unresolved; Eq. 7 as written is dimensionally inconsistent (Ai has L text columns, Vi has C image channels). The load-bearing assumption — that the prompts are semantic-only — is therefore unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RefSAM3D, an adaptation of the Segment Anything Model (SAM) to 3D medical image segmentation. The method introduces a 3D image adapter for volumetric inputs, a cross-modal reference prompt generation module that fuses CLIP text embeddings with hierarchical visual features, and a lightweight 3D mask decoder. The authors report state-of-the-art Dice and Normalized Surface Dice scores on KiTS21, MSD Pancreas, LiTS, and MSD Colon tumor segmentation, average Dice of 88.3% on BTCV multi-organ segmentation, and zero-shot mean Dice of 85.7% on AMOS22 CT and 63.2% on AMOS22 MRI after training only on BTCV CT. Ablations show that removing the text prompt, the cross-modal projector, or the hierarchical fusion each causes a substantial performance drop.","tokens_in":17134,"tokens_out":5929,"duration_ms":52647,"significance":"If the results are valid, RefSAM3D would be a useful contribution: it is a parameter-efficient way to adapt a 2D foundation model to 3D medical volumes, and it reports large gains over strong baselines such as MA-SAM and nnU-Net. The paper also contains positive elements: a clearly described architecture with equations for the adapter and cross-attention, evaluations on multiple public datasets, and ablations of the main components. However, as written, the central performance claims are not verifiable or fairly interpretable. The text prompts encode dataset-specific acquisition statistics and can act as dataset-identity embeddings; the evaluation protocol is incomplete, with an unresolved reference and missing details; the dataset description for the cardiac experiment is self-contradictory; and the zero-shot experiment does not specify the test-time prompt. These issues bear directly on the paper's headline claims of consistent state-of-the-art performance and cross-modal generalization.","major_comments":[{"comment":"The per-dataset text prompts in Table II contain acquisition statistics rather than purely semantic descriptions: KiTS21 includes spacing and dimensions, MSD Pancreas includes resolution and slice counts, BTCV includes slice thickness and in-plane resolution, and AMOS22 names 'CT and MRI images' with 'varying modalities and resolutions.' Since each prompt is a constant for an entire dataset and is passed through the trainable cross-modal MLP (Eq. 3) and hierarchical cross-attention (Eqs. 6-8), the text branch can act as a dataset-identity embedding rather than as anatomical guidance. The ablation in Table IV is consistent with this concern: removing the text prompt lowers BTCV Dice by 16.0 points (88.3 to 72.3), a swing much larger than the method's 1.1-point margin over MA-SAM. Please re-run the experiments with prompts that contain only organ or structure names, or otherwise demonstrate that the dataset statistics do not inflate the reported results, for example by swapping prompts across datasets.","section":"Section IV.A, Table II, Table IV"},{"comment":"The evaluation protocol is underspecified and partly unreferenced. The phrase 'same as xxx []' is unresolved, and baseline descriptions such as 'SAM-B[]' and 'MA-SAM(1 relaxed 3D bbx/slice)' lack precise prompt construction details. The manuscript also does not specify the exact resampling, patch sampling, cropping, and inference-time prompt procedures. Without a complete, self-contained protocol and release of code and trained models, the numbers in Tables I and III cannot be independently reproduced or compared against baselines.","section":"Section IV.A (Implementation Details)"},{"comment":"The dataset description contradicts itself: the subsection is titled 'MRI Cardiac Segmentation' and states that MM-WHS contains 20 CT and 20 MRI scans, but then says 'only the CT scans were used.' Table II gives the prompt as 'MRI images, cardiac structure segmentation,' and Section IV.C, Figure 5, and the accompanying text describe MRI cardiac segmentation. No quantitative cardiac results appear in any table. Please clarify which modality was used, report the corresponding quantitative results, and correct the text.","section":"Section IV.A (MRI Cardiac Segmentation)"},{"comment":"The zero-shot experiment is not well-defined because the test-time prompt is not specified. The model is trained on BTCV CT and then evaluated on AMOS22 CT and MRI; if the AMOS22 prompt from Table II (which names both modalities, 'varying modalities and resolutions', and 15 organs) is used at inference, then the model is informed of the target dataset and modality, so the reported 85.7% CT and 63.2% MRI Dice scores do not measure clean cross-dataset generalization. Please state the exact test-time prompts and, ideally, use prompts that do not encode modality or dataset-specific statistics.","section":"Section IV.C (Generalization Evaluation)"},{"comment":"No error bars or statistical significance tests are reported for the main comparisons, and the only variance given anywhere is the single '±3.1%' in Section IV.C. The claim that RefSAM3D 'consistently outperforms' other methods (Section IV.B) is therefore not statistically supported. Please report mean and standard deviation over at least three runs or cross-validation folds, and state the number of runs used for each table.","section":"Tables I, III, IV and Section IV.C"}],"minor_comments":[{"comment":"Unresolved markers 'same as xxx []', 'SAM-B[]', and 'Table ??' must be replaced with actual citations and table references.","section":"Section IV.A"},{"comment":"The caption 'MRI cardical tumor segmentation' contains a typo, and the text refers to 'cardiac tumor segmentation,' but the MM-WHS task is cardiac structure segmentation, not tumor segmentation.","section":"Figure 5 caption and Section IV.B"},{"comment":"The qualitative paragraph refers to 'our AutoSAM Adapter' when the proposed method is RefSAM3D; please correct the model name.","section":"Section IV.B"},{"comment":"The sentence 'HD increasing from 2.34% to 7.31%' uses percent signs for Hausdorff Distance, which is measured in mm; please fix the units.","section":"Section IV.D"},{"comment":"The sentence 'These results confirm When the Cross-Modal Projector is removed...' is grammatically incomplete and should be rewritten.","section":"Section IV.D"},{"comment":"The MSD Pancreas dataset is cited as reference [31], which is actually the Medical SAM Adapter paper; please cite the dataset source instead.","section":"Section IV.A"},{"comment":"The checkmark layout is ambiguous; please label which stages are included in each row.","section":"Table V"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an early draft with several incomplete references and internal inconsistencies, which suggests it was not carefully checked before submission. I do not see evidence of intentional misconduct, but the prompt-leakage concern must be addressed directly because it affects the validity of the reported state-of-the-art and zero-shot claims. I would recommend requiring a complete evaluation protocol, code release, and corrected experiments with semantic-only prompts before any further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nYou should know two things about arXiv:2412.05605. First, the core idea—adapting SAM for 3D medical segmentation with a 3D adapter plus hierarchical cross-attention between visual features and CLIP text prompts—is a reasonable extension of referring segmentation into volumetric medical imaging. The individual components are known, but the specific combination and the use of per-dataset text prompts as a conditioning signal is new enough to matter. Second, the reported numbers are not trustworthy as presented, and the reason is specific: the text prompts encode dataset statistics, not just semantics.\n\nThe paper does several things well. It builds on relevant prior work (3DSAM-Adapter, MA-SAM, SAM-Med3D) and describes the architecture clearly enough to be implemented. The ablation study is a good instinct, even if the result is damning. And the zero-shot framing is the right question to ask.\n\nThe soft spots are serious. Table II lists prompts that include acquisition metadata: spacing ranges, slice counts, volume dimensions. The prompt is constant for every sample in a dataset and flows through a trainable cross-modal MLP and hierarchical attention. That gives the text branch the ability to act as a dataset-identity embedding. The ablation in Table IV is consistent with this reading: removing the text prompt drops BTCV Dice from 88.3 to 72.3, a 16-point swing, which is larger than the entire margin over MA-SAM (88.3 vs 87.2). The zero-shot results on AMOS22 are uninterpretable unless we know which prompt was used at test time; if the AMOS22 prompt names the modality and organ count, then the model is being told the target domain and the generalization claim is inflated.\n\nThere are also correctness and completeness problems that should not have been in a submission: unresolved placeholders ('same as xxx []', 'Table ??', 'SAM-B[]'), a contradiction about MM-WHS (Section IV.A says only CT was used, yet Table II and Figure 5 report MRI), and Eq. 7 as written is dimensionally inconsistent. No code or detailed protocol is released.\n\nI'm not saying the method is worthless. With prompts limited to semantic labels (e.g., 'liver tumor') and a full evaluation protocol, the architecture might hold up. But as written, the load-bearing claim of state-of-the-art performance is unsupported. I would not cite it in the next year. I would, however, send it to peer review: the idea deserves scrutiny, and a competent reviewer can pin down the prompt issue. Bring it to reading group? Maybe—it's a good case study in how conditioning can leak dataset identity.","headline":"The architecture is a legitimate extension of SAM to 3D medical segmentation, but the per-dataset text prompts encode acquisition statistics, so the reported SOTA and zero-shot gains are likely driven by dataset-identity conditioning, not segmentation ability.","tokens_in":17715,"tokens_out":5336,"would_cite":false,"duration_ms":44686,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adapting SAM to 3D medical volumes with cross-modal text prompts yields state-of-the-art segmentation and strong zero-shot transfer.","keywords":["3D medical image segmentation","Segment Anything Model","cross-modal prompt","text-guided segmentation","volumetric CT and MRI","parameter-efficient adaptation","hierarchical cross-attention","tumor segmentation"],"falsifier":"Run RefSAM3D on the same benchmarks with text prompts reduced to organ names and modality only (no spacing, dimension, or slice-count statistics), keeping everything else fixed. If Dice on KiTS21, MSD, LiTS, and AMOS falls toward the no-prompt ablation level (72.3% on BTCV), the reported advantage is a prompt-leakage artifact; if it stays near the published numbers, the mechanism is semantic.","tokens_in":16514,"feed_emoji":"🩻","tokens_out":10408,"duration_ms":80795,"temperature":0.7,"pith_summary":"RefSAM3D attempts to make the Segment Anything Model (SAM) segment 3D medical volumes instead of 2D natural images. It adds a lightweight 3D adapter to SAM's visual encoder, converts the mask decoder to produce 3D masks, and conditions the decoder on a cross-modal prompt built by aligning CLIP text embeddings with hierarchical visual features. The paper reports state-of-the-art Dice scores on kidney, pancreas, liver, and colon tumor benchmarks and on BTCV multi-organ segmentation, plus zero-shot results on AMOS CT and MRI after training only on BTCV. The central assertion is that volumetric adaptation and text prompts together make SAM a competitive and generalizable 3D medical segmenter.","feed_headline":"Text-guided 3D SAM beats prior models on tumor and organ benchmarks","feed_subtitle":"A CT-trained model reports top tumor and organ scores, plus zero-shot transfer to MRI.","key_machinery":"The mechanism that carries the argument is a cross-modal reference prompt built by hierarchical cross-attention. The visual encoder is SAM's ViT with patch embedding decomposed into $1 \\times k \\times k$ and $k \\times 1 \\times 1$ 3D convolutions and positional embeddings extended by a learnable depth axis; lightweight adapters with depthwise 3D convolutions let the model adjust to volumes without fine-tuning all weights. Features from the four encoder stages are each projected by an adapter, then cross-attended with a sentence-level text embedding produced by the CLIP text encoder and aligned by an MLP; the concatenated outputs form the prompt for the mask decoder. The decoder replaces 2D convolutions with 3D ones, upsamples progressively to full resolution, and fuses the original image at the end.","core_discovery":"The central claim is that a SAM-based model can be adapted to 3D medical imaging by (1) converting patch embedding and attention to accept 3D inputs while freezing most pre-trained weights, (2) inserting a 3D adapter with depthwise convolutions, and (3) conditioning the mask decoder on a cross-modal prompt generated by aligning CLIP text embeddings with hierarchical features from four encoder stages. In the paper's experiments, this design achieves 95.53% Dice on KiTS21 kidney tumor, 82.42% on MSD pancreas tumor, 80.10% on LiTS liver tumor, 70.14% on MSD colon cancer, and an average 88.3% Dice on BTCV 13-organ segmentation. After training only on BTCV CT, the same model reports zero-shot mean Dice of 85.7% on AMOS22 CT and 63.2% on AMOS22 MRI, rising to 84.1% with five fine-tuning scans. The ablation study attributes a 16-point Dice drop to removing the text prompt.","pith_inferences":["The hand-designed prompts in Table II include dataset-specific numbers (spacing ranges, dimensions, slice counts). If the model exploits these statistics as a dataset-identity signal, the reported state-of-the-art and zero-shot gains may not reflect general segmentation ability; replacing the numbers with purely anatomical descriptions would test this.","The 16-point ablation gap between prompted and unprompted runs could come partly from the prompt acting as a distribution selector. A cross-dataset experiment that swaps prompts between tasks (e.g., using the colon-cancer prompt on liver volumes) would reveal whether the text contributes semantic guidance or dataset identification.","The same three-stage recipe (frozen 2D encoder, lightweight 3D adapter, cross-modal prompt) is general enough to apply to other interactive segmentation foundation models, letting the community check the claimed gains without rebuilding the pipeline."],"forward_implications":["The method reports state-of-the-art Dice on four tumor benchmarks, so a text-conditioned 3D SAM is a viable alternative to specialized segmentation networks on those tasks.","Training once on BTCV CT and evaluating on AMOS CT/MRI without retraining yields 85.7% and 63.2% Dice, indicating cross-dataset and cross-modality transfer from a single model.","With five AMOS MRI scans for fine-tuning, the reported 84.1% Dice suggests few-shot adaptation is enough to close much of the cross-modality gap.","Removing the text prompt drops BTCV Dice from 88.3% to 72.3%, making the text prompt a load-bearing component rather than a marginal addition."],"supporting_citations":[{"why":"Base SAM model whose 2D ViT and mask decoder are adapted to 3D; supplies all frozen pre-trained weights.","marker":"[4]"},{"why":"3DSAM-adapter, a promptable 3D adaptation baseline that RefSAM3D extends and compares against.","marker":"[13]"},{"why":"MA-SAM, the strongest prior adaptation baseline and the source of the evaluation protocol for BTCV-to-AMOS generalization.","marker":"[35]"},{"why":"CLIP text encoder used to convert text prompts into embeddings for cross-modal conditioning.","marker":"[21]"},{"why":"BTCV dataset used for multi-organ training and the zero-shot generalization source.","marker":"[48]"},{"why":"AMOS22 dataset used for zero-shot and five-shot cross-modality generalization tests.","marker":"[50]"},{"why":"KiTS21 dataset used as one of the four tumor benchmarks where the top Dice is reported.","marker":"[44]"},{"why":"nnU-Net, the classical baseline that the method must beat across all benchmarks.","marker":"[51]"}],"fun_headline_variants":["SAM goes 3D with text prompts, tops medical benchmarks","Text-guided SAM adapts to 3D scans, beats SOTA","RefSAM3D: 3D SAM with text reference for tumor and organ segmentation","Cross-modal SAM scores top Dice on 3D medical scans","Text-conditioned 3D SAM outperforms on organs and tumors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-written text prompts provide genuine anatomical semantics and not hidden dataset information; the prompts list spacing ranges, volume sizes, and slice counts that match the test sets, so if the model uses those to infer which dataset a volume comes from, the reported accuracy is inflated.","fun_headline_variants_meta":{"raw":{"variants":["SAM goes 3D with text prompts, tops medical benchmarks","Text-guided SAM adapts to 3D scans, beats SOTA","RefSAM3D: 3D SAM with text reference for tumor and organ segmentation","Cross-modal SAM scores top Dice on 3D medical scans","Text-conditioned 3D SAM outperforms on organs and tumors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2851,"prompt_tokens":939,"completion_tokens":1912,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":1818}},"tokens_in":555,"tokens_out":1912,"duration_ms":11603,"temperature":1.0,"reasoning_tokens":1818,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:32:23.891676+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RefSAM3D on the same benchmarks with text prompts reduced to organ names and modality only (no spacing, dimension, or slice-count statistics), keeping everything else fixed. If Dice on KiTS21, MSD, LiTS, and AMOS falls toward the no-prompt ablation level (72.3% on BTCV), the reported advantage is a prompt-leakage artifact; if it stays near the published numbers, the mechanism is semantic.","supporting_citations":[{"cited_title":"Segment anything,","cited_arxiv_id":null,"evidence_quote":"Base SAM model whose 2D ViT and mask decoder are adapted to 3D; supplies all frozen pre-trained weights."},{"cited_title":"3dsam- adapter: Holistic adaptation of sam from 2d to 3d for promptable tumor segmentation,","cited_arxiv_id":null,"evidence_quote":"3DSAM-adapter, a promptable 3D adaptation baseline that RefSAM3D extends and compares against."},{"cited_title":"Ma-sam: Modality-agnostic sam adaptation for 3d medical image segmentation,","cited_arxiv_id":null,"evidence_quote":"MA-SAM, the strongest prior adaptation baseline and the source of the evaluation protocol for BTCV-to-AMOS generalization."},{"cited_title":"Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge,","cited_arxiv_id":null,"evidence_quote":"BTCV dataset used for multi-organ training and the zero-shot generalization source."},{"cited_title":"Amos: A large-scale abdominal multi- organ benchmark for versatile medical image segmentation,","cited_arxiv_id":null,"evidence_quote":"AMOS22 dataset used for zero-shot and five-shot cross-modality generalization tests."},{"cited_title":"The kits21 challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase ct,","cited_arxiv_id":null,"evidence_quote":"KiTS21 dataset used as one of the four tumor benchmarks where the top Dice is reported."},{"cited_title":"nnu-net: Self-adapting framework for u-net-based medical im- age segmentation,","cited_arxiv_id":null,"evidence_quote":"nnU-Net, the classical baseline that the method must beat across all benchmarks."}],"review_version":1}