{"id":"be8df5a9-3949-42ac-bcae-2da5c1a31c01","arxiv_id":"2502.03501","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Proxy Prompt lets frozen SAM and SAM 2 segment new medical images and videos using a high-dimensional prompt auto-generated from a non-target image-mask pair.","lead":"This paper adds a trainable module to SAM and SAM 2 that automatically builds a segmentation prompt from an unrelated reference image and mask, then runs the frozen foundation model on a new medical image or video. The appeal for a generalist reader is the claim that a clinician can guide the model by pointing at a different patient's scan once, instead of drawing prompts on every new frame.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Support-prompt transfer is only demonstrated with support pairs drawn from the same training dataset as the target; no cross-dataset same-modality test exists, and the paper's own unrelated-support test collapses to 65.25% Dice.","rationale":"The reader's weakest assumption is the correct one: the prompt embedding only works when support and target share object semantics and modality. My reading agrees, and the authors' own Fig. 10 confirms the failure mode. The remaining gap is sharper than the reader states: no test exists for the realistic case of same modality but different dataset or scanner, so the scope of 'non-target data' is unverified. I considered alternative concerns. Missing comparisons with ProtoSAM, EviPrompt, and VRP-SAM undermine the literal SOTA wording but are an evaluation-completeness issue, not a mechanism risk. The very small STARE test set (16 training, 4 test) is a statistical fragility, but the cross-dataset support question is more directly load-bearing for the central transfer claim. The architecture (Eqs. 3-12) is internally consistent, the ablations are reasonably thorough, and there is no sign of fabrication. Because the unresolved concern is already reflected in the reader's CONDITIONAL verdict, the verdict should not change; it stays CONDITIONAL pending the cross-dataset support experiment.","tokens_in":21873,"tokens_out":8137,"duration_ms":70326,"concrete_test":"Run one cross-dataset same-modality transfer experiment on the fundus model trained under Strategy 2 (Sec. 7.2). Keep the REFUGE2 test set and frozen SAM fixed, but draw the support image-mask pairs from a fundus dataset not used in training, e.g. DRIVE or CHASE_DB1 for vessel masks and RIM-ONE for disc/cup masks. Compute mean Dice and compare with the same-dataset support results in Table 14 (REFUGE2-Disc 88.0, REFUGE2-Cup 85.5). If cross-dataset support drops by more than about 5 Dice points, the transfer claim must be narrowed to same-dataset or same-distribution support; if it stays within a few points, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a high-dimensional prompt embedding extracted from a non-target image-mask pair can guide a frozen SAM/SAM 2 decoder on the target. The load-bearing condition is that the support pair and target share the relevant object semantics; otherwise the selective-map conditioning in Eqs. 6-7 has nothing useful to transfer. The paper's evidence for this condition is narrow. In Sec. 7.1, support pairs at inference are randomly selected from the same training dataset, and during training, data within the same dataset are alternated as support and target. Thus every positive result in Tables 1-3 is within-distribution support. The only out-of-distribution test, Sec. 11.2 and Fig. 10, replaces an ultrasound support with a retinal scan, and Dice collapses from about 95.5% to 65.25%. That is an existence proof that the embedding prompt does not carry object semantics independently of support-target relevance. The realistic retrospective-data scenario, however, is not an unrelated modality but a different scanner, protocol, or population within the same modality and anatomy. No experiment tests that scenario. Consequently, the abstract's general 'non-target data' claim is not yet established; the method may require a support image drawn from essentially the same dataset distribution as the target, which is a materially narrower scope.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Proxy Prompt (PP), a plug-in module called the Proxy Prompt Generator (PPG) that converts a support image-mask pair drawn from non-target data into a high-dimensional embedding prompt for frozen SAM and SAM 2. The PPG consists of a Contextual Selective Module (CSM), which uses a Vision Mamba encoder, a bridge unit, and a selective map to extract relevant contextual information from the support pair, and a Contextual Colorization Module (CCM), which refines the contextual embedding through dual reverse cross-attention with target features. The method is evaluated on five public datasets (REFUGE2, STARE, FPA, PROMISE12, JNU-IFM) using a few-shot setup with 16 training images or videos, and the authors report state-of-the-art performance and results comparable to fully trained models, along with extensive ablations of the modules, prompt quality, training size, and support-pair quantity.","tokens_in":22007,"tokens_out":5780,"duration_ms":53666,"significance":"If the central claim is established, the method addresses a real clinical need by replacing manual per-target prompting with a pre-annotated support pair and by enabling task switching without retraining. The paper is strong in its breadth of experiments and internal consistency: the ablation tables are detailed, the stability analysis on video data with multiple support pairs is valuable, and the training strategy that alternates support and target roles within the same dataset is a sensible way to avoid obvious target-leakage or circularity. The low trainable-parameter count of the Vision Mamba based CSM is also an attractive feature. The main weakness is that the central generalization claim, namely that prompts from non-target data transfer broadly, is only demonstrated for support pairs drawn from the same training distribution as the target.","major_comments":[{"comment":"The evidence for the central claim that non-target data can guide target segmentation is entirely within-distribution. At inference, support pairs are randomly selected from the same training dataset (Sec. 7.1), and during training, support and target images are alternated within the same dataset. The only out-of-distribution test, Fig. 10 (SI5), changes both modality and anatomy, and the Dice score collapses from about 95.5% to 65.25%. The realistic retrospective-data scenario, namely a different scanner, protocol, or population within the same modality and anatomy, is never tested. This is load-bearing because the abstract promises guidance from non-target data generally. Please add same-modality cross-dataset support experiments (for example, a fundus support pair from REFUGE2 used for STARE targets, or an ultrasound support pair from one scan protocol used for FPA targets) and temper the abstract and conclusion to the demonstrated scope until such evidence is available.","section":"Sec. 7.1 and Sec. 11.2, Fig. 10"},{"comment":"The comparison protocol for point and box baselines gives those baselines near-ideal prompts computed from ground-truth masks (one random point inside the target mask, or the minimum bounding rectangle of the target), while the proposed method uses a support pair that contains no target-specific ground truth. This is conservative with respect to the proposed method and should be acknowledged explicitly in the main text. In addition, the 'Upper' rows in Tables 1 and 2 are taken from results reported in the original papers (BEAL, nnUnet, SegNet, DSD-FCN) under their own protocols, not re-run on the same test splits. To support the claim of being 'comparable with fully-trained models', either re-run the Upper methods on the same split or state clearly that the comparison is against literature-reported numbers.","section":"Sec. 6.1 and Table 1"},{"comment":"The selective-map equation has a shape inconsistency. The first term, Fsup^T * Fx, has shape (K*H*W) x (H*W), while the second term, Fsup^2, is not defined in a way that permits subtraction from that matrix. If Fsup^2 is intended to denote a broadcast row-wise squared norm as in STCN, the notation should make that explicit; if it is elementwise squaring, the operation is dimensionally invalid. Because Eq. (6) defines the core selective map, the formula needs to be unambiguous and must match the implementation.","section":"Eq. (6), Sec. 3.1"},{"comment":"The closest prompt-from-reference methods, including VRP-SAM, ProtoSAM, EviPrompt, and PerSAM, are discussed in the related work and Sec. 10 but do not appear in any quantitative comparison. Since the abstract claims state-of-the-art performance, the manuscript should include at least one quantitative comparison with these methods under the same few-shot protocol, or provide an explicit and justified statement of why such a comparison is not applicable.","section":"Sec. 2 and Tables 1-3"}],"minor_comments":[{"comment":"The standalone abstract says 'four public datasets' while the main-text abstract says 'five public datasets'; the experiments actually cover five datasets (REFUGE2, STARE, FPA, PROMISE12, JNU-IFM). Please reconcile the count.","section":"Abstract"},{"comment":"The text says 'To evaluate the impact of training set size K' but then defines M in {2, 4, 8, 16}; the notation for training-set size and inference support-set size is used inconsistently and should be fixed.","section":"Sec. 11.6"},{"comment":"The tensor notation for the multi-object case is confusing: V is written as R^{K x Cv x H x W}, but the text says N feature matrices are produced, and Eq. (4) introduces Fcat with both K and N. Please clarify the roles of K and N consistently in the equations and in Fig. 3.","section":"Eqs. (3)-(5), Sec. 3.1"},{"comment":"The main-text ablation refers to Fig. 13 for training-size results, but the same figure appears as Fig. 6 in the body; supplementary figures and main-text figures should have consistent numbering.","section":"Figure numbering"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid engineering contribution with careful ablations, but the central 'non-target data' claim is currently supported only for within-distribution support pairs. The missing same-modality cross-dataset experiment is the key load-bearing gap, and it is fixable within the scope of the manuscript. The selective-map equation also needs clarification. If the authors add the requested cross-dataset experiment and tighten the claims, the paper could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a practical, well-ablated method for turning a reference image-mask pair into a high-dimensional SAM/SAM2 prompt, and the architecture (Vision Mamba selective module + dual cross-attention colorizer) is genuinely different from the PerSAM/EviPrompt/ProtoSAM line. The experiments are extensive: five datasets, image, MRI, and video, plus ablations on module design, support quality, training size, and support quantity. If the contribution is judged as a plug-in that works with 16 images, the paper largely delivers.\n\nThe main soft spot is that the SOTA claim isn't fully demonstrated. The paper cites ProtoSAM, EviPrompt, and VRP-SAM but never compares against them; Table 1 is basically a comparison against SAM/MedSAM with point/box prompts and a few fine-tuned adapters. Those are valid baselines, and the oracle prompts given to the point/box baselines are standard (and if anything generous), but the most relevant reference-prompt methods are missing. The 'Upper' numbers from prior papers are also borrowed under different protocols, so 'comparable to fully-trained models' should be read loosely.\n\nSecond, the generalization evidence is narrower than the abstract implies. All successful tests use support pairs drawn from the same training distribution as the target. The combined fundus training (Strategy 2) does mix REFUGE2 and STARE, and since inference support pairs are randomly sampled from the combined set, there is some implicit cross-dataset same-modality transfer, but it is not analyzed separately. The authors do test an unrelated-modality support (retina image for ultrasound) and Dice collapses to 65.25%; they acknowledge this and argue clinicians will choose relevant supports. That is a fair defense, but the paper should state the scope as 'semantically relevant retrospective data' rather than 'non-target data' without qualification.\n\nThird, reproducibility and statistics are thin: no code, LoRA rank/position not specified in the main text, and the image/MRI results have no error bars or significance tests. The video stability experiments (five support pairs, std and max-min) are a nice exception.\n\nNone of this is fatal. The method is novel enough, the ablations are thoughtful, and the reasoning is coherent. I'd send it to reviewers, but I'd expect the SOTA claim to be tempered and a head-to-head with ProtoSAM/EviPrompt/VRP-SAM to be added. Also worth asking for a proper cross-dataset same-modality experiment, e.g., train on one fundus dataset and support from a different fundus dataset, to establish whether the requirement is 'same dataset' or 'same modality'. That would strengthen the clinical story considerably.","headline":"A useful and genuinely novel plug-in for reference-prompted SAM/SAM2, but the SOTA claim needs head-to-head comparisons and the generalization claim needs a proper cross-dataset test.","tokens_in":22684,"tokens_out":4273,"would_cite":true,"duration_ms":37070,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learned 'proxy prompt' from annotated non-target images can steer a frozen SAM or SAM 2 to segment medical images and video as well as manual point, box, or mask prompts, matching fully trained models with 16 labeled image-mask pairs.","keywords":["proxy prompt","SAM 2","medical image segmentation","few-shot segmentation","vision mamba","automatic prompting","cross-attention","ultrasound video segmentation"],"falsifier":"Give the trained generator a target image and, on a fixed support image, two masks denoting different objects (e.g., optic disc vs. vessels, or fetal head vs. pubic symphysis): if the output does not switch cleanly with the mask, the prompt is not carrying user intent. The sharper probe is already half-run by the authors — a support image from a different modality drops Dice to 65.25% — so a systematic sweep over modality, anatomy, and scanner mismatch would show exactly where the 'non-target data' claim fails and whether 16 training images are enough to survive any of those shifts.","tokens_in":21558,"feed_emoji":"🩺","tokens_out":16666,"duration_ms":126381,"temperature":0.7,"pith_summary":"SAM and SAM 2 segment whatever a user points at, but in a busy clinic pointing at every image or frame is exactly the bottleneck. This paper claims that bottleneck can be removed: a 'proxy prompt' — a high-dimensional embedding computed from an annotated image of another subject, not the one being examined — can replace manual point, box, or mask prompts entirely, with SAM's and SAM 2's original weights left frozen. The authors build a Proxy Prompt Generator that selects the most target-relevant context from the support image-mask pair (Contextual Selective Module) and then 'colorizes' it with the user's object choice via dual cross-attention (Contextual Colorization Module). Trained on only 16 image-mask pairs, the method reports 85.9% average Dice across five medical segmentation objects, above both the few-shot baselines (best 80.7%) and a fully trained upper-bound model (83.3%); on ultrasound video it reports 80.9% average Dice with 0.3% standard deviation across five different support pairs. If these results hold, clinicians could pre-load a prompt once and then switch targets or cases by swapping an annotated support pair, with no per-case prompting and no retraining.","feed_headline":"16 labeled images let frozen SAM match fully trained segmenters","feed_subtitle":"Proxy prompts built from non-target annotated data drive SAM and SAM 2 with no per-image clicks.","key_machinery":"The load-bearing object is the proxy prompt $P \\in \\mathbb{R}^{N \\times ((H \\times W)/16) \\times C}$: $N$ high-dimensional embeddings, one per requested object, computed from a support set of image-mask pairs and fed into the frozen decoder. The selection machinery is the Selective Map (Eq. 6), a channel-normalized correlation\n$$\\mathrm{Selective} = \\frac{2\\left(\\hat{F}_{\\mathrm{sup}}^{T}\\hat{F}_x\\right) - \\hat{F}_{\\mathrm{sup}}^{2}}{\\sqrt{C}},$$\ncomputed between flattened support features $\\hat{F}_{\\mathrm{sup}}$ and target features $\\hat{F}_x$; after softmax normalization it weights the aggregated support features $A_{\\mathrm{agg}}$ to yield the contextual embedding $E_{ctx}$ (Eq. 7), so only support content relevant to the target survives. The interpretation machinery is the Contextual Colorization Module: four blocks in which target features read from the context by cross-attention and the context then reads back from the refined target features by reversed cross-attention, so the embedding absorbs the user's object choice from the support mask along with target appearance — conceptually, 'coloring' the target with the support's annotation. A Vision Mamba encoder supplies the first selection step, its input-dependent state-space parameters serving as the input-driven filter; a bridge unit with CBAM channel-spatial attention supplies the second step, communicating features across objects before the Selective Map applies the final target-relevance filter.","core_discovery":"The paper's claim, stated sympathetically, is that prompting a segment-anything model is a representation problem rather than a user-interface problem: the information a user would put into a point, box, or mask can instead be learned and delivered as an embedding, and that embedding can be produced from data that never touches the target. The same mechanism is claimed to work for images (SAM) and video (SAM 2), because both models accept prompt inputs through a prompt encoder or memory-attention channel; the Proxy Prompt Generator feeds the refined embedding directly into the SAM decoder (Eq. 12) and into SAM 2's memory attention. The experimental claim is that this yields state-of-the-art few-shot performance — 85.9% average Dice with 16 training image-mask pairs on REFUGE2 (disc/cup), STARE (vessels), and FPA (fetal head and pubic symphysis), versus 80.7% for the best prior few-shot method and 83.3% for fully trained models — that it transfers to 3D MRI (87.4% vs. 88.1% fully supervised), and that on real-time ultrasound video it is both accurate (80.9%) and stable (0.3% standard deviation across support pairs vs. 10.9% for the comparison method). The paper further claims the design is model-agnostic: it improved SAM-ViT-B, SAM-ViT-H, and MedSAM alike, and can be reattached as newer foundation models appear.","pith_inferences":["The paper trains one generator per modality rather than per object, so the natural next test is whether the support-pair machinery transfers across modalities; the authors' own experiment with an unrelated support (65.25% Dice) marks a boundary worth mapping systematically, e.g., MRI-supported ultrasound or fundus-supported endoscopy.","The finding that MedSAM gains less than SAM (and loses ground on FPA, its pretraining domain) implies the proxy prompt is most effective on models without a strong learned prompt prior; attaching this module to foundation models heavily trained on box or point prompts should be expected to yield diminishing returns.","Because the prompt enters through SAM's standard prompt channel and SAM 2's memory attention, the same generator could in principle steer any promptable segmenter with a comparable interface — a generalization the paper gestures at with its 'plug-and-play' framing but does not demonstrate.","The near-zero variance across support pairs on video (0.3% std) hints that the Selective Map actively discards support-specific appearance; a testable extension is whether stability survives when the support image is a different subject, a different scanner, or a compressed frame, conditions under which manual point prompts typically degrade."],"forward_implications":["A single annotated support pair replaces per-image prompting: with the proxy prompt, the same frozen model segments every image or video frame of a study, which is what makes real-time ultrasound guidance feasible.","Task switching without retraining: because the support mask encodes the object choice, clinicians can move from segmenting optic disc/cup to vessels, or from fetal head to pubic symphysis, by changing the support pair rather than training a new model.","Few-shot medical segmentation reaches full-data parity: 16 image-mask pairs (58 MRI slices) suffice for average Dice scores comparable to or better than models trained on the complete datasets (85.9% vs. 83.3% on images; 87.4% vs. 88.1% on MRI).","The prompting strategy is transferable across foundation models: applying the same generator to SAM-ViT-B, SAM-ViT-H, and MedSAM improved all three, with the largest gains on models without a conflicting prompt prior.","Video segmentation becomes pre-loadable: because the prompt derives from non-target data, it can be prepared before the exam starts, then applied as memory prompts frame after frame."],"supporting_citations":[{"why":"Supplies SAM, the frozen image segmenter whose decoder receives the proxy prompt.","marker":"[16]"},{"why":"Supplies SAM 2, the frozen video segmenter whose memory attention receives the prompt embedding.","marker":"[25]"},{"why":"Provides the Vision Mamba encoder used as the first input-driven selection step in the Contextual Selective Module.","marker":"[44]"},{"why":"Provides the correlation computation on which the Selective Map is based.","marker":"[5]"},{"why":"Provides the channel-spatial attention (CBAM) used in the Bridge Unit for inter-object feature communication.","marker":"[35]"},{"why":"Provides the low-rank adaptation (LoRA) used to fine-tune the frozen encoders and decoders with only 16 training images.","marker":"[12]"},{"why":"The train-free medical baseline (MedSAM) whose box-prompt performance the method must match on seen and unseen datasets.","marker":"[22]"},{"why":"The AutoSAM few-shot baseline that the method's 85.9% average Dice is compared against (80.7%).","marker":"[27]"}],"fun_headline_variants":["16 annotated images let frozen SAM rival fully trained models","Proxy Prompt automates SAM and SAM 2 prompting for medical imaging","Frozen SAM matches fully-supervised Dice with just 16 training masks","Auto-interactive prompts from non-target data: few-shot SAM rival"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a support image-mask pair from the same modality and roughly the same anatomy carries all the object information a manual point or box would carry, and that this information survives compression into a high-dimensional embedding — the paper trains only on 16 image-mask pairs, tests only within matching modalities, and documents the premise failing for an unrelated support (a retinal scan guiding an ultrasound target, 65.25% Dice).","fun_headline_variants_meta":{"raw":{"variants":["16 annotated images let frozen SAM rival fully trained models","Proxy Prompt automates SAM and SAM 2 prompting for medical imaging","Frozen SAM matches fully-supervised Dice with just 16 training masks","Auto-interactive prompts from non-target data: few-shot SAM rival"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000384,"raw_usage":{"total_tokens":2070,"prompt_tokens":1024,"completion_tokens":1046,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":973}},"tokens_in":640,"tokens_out":1046,"duration_ms":10072,"temperature":1.0,"reasoning_tokens":973,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:04:34.626856+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the trained generator a target image and, on a fixed support image, two masks denoting different objects (e.g., optic disc vs. vessels, or fetal head vs. pubic symphysis): if the output does not switch cleanly with the mask, the prompt is not carrying user intent. The sharper probe is already half-run by the authors — a support image from a different modality drops Dice to 65.25% — so a systematic sweep over modality, anatomy, and scanner mismatch would show exactly where the 'non-target data' claim fails and whether 16 training images are enough to survive any of those shifts.","supporting_citations":[{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"Supplies SAM, the frozen image segmenter whose decoder receives the proxy prompt."},{"cited_title":"Rethink- ing space-time networks with improved memory coverage for efficient video object segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the correlation computation on which the Selective Map is based."},{"cited_title":"Cbam: Convolutional block attention module","cited_arxiv_id":null,"evidence_quote":"Provides the channel-spatial attention (CBAM) used in the Bridge Unit for inter-object feature communication."}],"review_version":1}