{"id":"f9eec72e-df89-40a9-b21d-e74ff80a0189","arxiv_id":"2608.08844","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A SAM3 plus Qwen two-stage pipeline performs instance-level surgical instrument segmentation without pixel-level mask supervision, though it does not match fully supervised performance.","lead":"This paper combines the Segment Anything Model 3, prompted with the single word 'tool', with a vision-language model that names each detected surgical tool. The reported pipeline avoids pixel-level mask annotations and spatial clicks, but it still trails fully supervised systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Qwen fine-tuning labels are unspecified; the 'No Tool' category in §4.2 implies ground-truth mask overlap, which would falsify the mask-annotation-free claim.","rationale":"The reader's weakest assumption correctly identifies the core gap: the paper never states how SAM3-generated masks receive instrument labels during Qwen fine-tuning, and the 'No Tool' category in Section 4.2 is strong evidence that ground-truth knowledge was used somewhere in the labeling process. If ground-truth masks were used to label SAM3 masks, then the pipeline is not mask-annotation-free; it is a mask-proposal method plus a classifier trained with mask-derived supervision. This directly undermines the abstract and Section 3.3's claim to 'bypass pixel-level data annotation.' The concern is not merely about consensus or style; it is about whether the central contribution exists as stated. The binary 'tool'-prompt segmentation result and the oracle classifier analysis are useful and partially independent, but neither establishes the instance-level claim. The paper could be revised into a credible annotation-efficient pipeline if the label source were disclosed and the claims narrowed, but as written the central assertion is unsupported. Therefore the REJECT verdict stands without modification.","tokens_in":10154,"tokens_out":3382,"duration_ms":38014,"concrete_test":"Request the authors' exact Qwen fine-tuning data-construction protocol or code, and check the label source for each SAM3 training mask. Specifically, determine whether any training label is computed as the argmax IoU between a SAM3 mask and a ground-truth pixel mask, and whether 'No Tool' negatives are identified by overlap with ground-truth masks. If either is true, the pipeline uses pixel-level masks as supervision and the central claim fails. If labels come instead from manual review or frame-level lists with no pixel-mask overlap computation, the claim would need to be restated and re-evaluated, but at least the mechanism would be disclosed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the two-stage pipeline achieves instance-level segmentation without ground truth masks and without manual interaction. Stage 2 requires fine-tuning Qwen on masked regions, and Section 3.3 only says the input regions come from SAM3 predictions, not how each training region obtains its class label. Section 4.2 then introduces a 'No Tool' category during fine-tuning because SAM3 can mis-segment tissue. Constructing this category requires knowing which SAM3 masks are false positives, an annotation that cannot be derived from unlabeled SAM3 masks alone. The plausible ways to obtain these labels are: (1) matching each SAM3 mask to ground-truth masks by overlap, which uses pixel-level annotations and directly contradicts the abstract; (2) manual review of SAM3 masks, which is undisclosed and raises the 'manual interaction' question; or (3) some frame-level instrument-presence list, which the paper never describes and which would still be ambiguous for false positives and multiple same-class instruments. Because the entire novelty rests on bypassing pixel-level annotations, the undisclosed label source is the single most load-bearing weakness. The credible binary segmentation result with the 'tool' prompt and the oracle analysis do not rescue the instance-level claim, since they do not establish that Qwen can be trained without mask-derived supervision.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a two-stage approach to surgical instrument segmentation from endoscopic images. Stage 1 uses the pretrained SAM3 model with a fixed text prompt \"tool\" to generate binary masks, and Stage 2 uses a Qwen 2.5-7B vision-language model fine-tuned on the resulting masked regions to assign each mask an instrument class or a \"No Tool\" label. The authors claim this pipeline achieves instance-level segmentation without pixel-level ground truth masks and without manual spatial interaction. Experiments on EndoVis 2017, EndoVis 2018, and GraSP report binary IoU/Dice near or above a vanilla UNet baseline, instance-level metrics below fully supervised methods but far above a SAM3 instrument-name baseline, and an oracle analysis that isolates classification errors from mask generation errors.","tokens_in":10375,"tokens_out":9472,"duration_ms":100827,"significance":"The paper's strongest empirical contribution is the observation that a generic natural-language prompt (\"tool\") substantially outperforms instrument-name prompts for SAM3 on endoscopic images (Table 2), and the clean separation of mask localization from classification through the oracle analysis (Tables 3-4). The two-stage design is simple and the evaluation on three public benchmarks is clearly reported. If the mask-annotation-free claim were supported, this would be a meaningful step toward reducing annotation cost for surgical instrument segmentation. However, the central claim is currently unsupported because the paper does not explain how the training labels for Qwen are obtained without using mask-level ground truth.","major_comments":[{"comment":"The label source for Qwen fine-tuning is not described. Section 3.3 states that Qwen is fine-tuned on SAM3-predicted masked regions and asserts that this \"maintains a fully mask annotation-free training pipeline,\" but it never specifies how each masked region is assigned its class label during training. A supervised fine-tuning setup requires labels for every training region, including the \"No Tool\" category introduced in Section 4.2. Please state the exact labeling protocol, including the source of labels, the treatment of masks that overlap multiple instruments, and how \"No Tool\" negatives are identified. If any step uses pixel-level ground truth masks (for example, by matching each SAM3 mask to the ground truth with maximum IoU) or manual review of candidate masks, the abstract and Section 3.3 must be revised because the stated claim would be false.","section":"§3.3, §4.2"},{"comment":"The construction of the \"No Tool\" category implies access to mask-level supervision. Section 4.2 introduces this category because \"SAM3 can mis-segment tissue regions as instruments.\" Constructing it requires knowing, for each SAM3 mask, whether it is a true instrument or a false positive; this knowledge is exactly the pixel-level annotation the paper claims to avoid. The manuscript does not describe any mask-free mechanism for obtaining these labels. This is not a minor implementation detail: the \"No Tool\" filter is part of the proposed pipeline, and its presence in the training data implies either ground truth mask overlap, manual inspection, or an undisclosed heuristic. Please provide the mechanism and, ideally, an ablation without the \"No Tool\" category to measure how much of the reported performance depends on it.","section":"§4.2"},{"comment":"The evaluation of Qwen on ground-truth masked regions replaces every \"No Tool\" prediction with a randomly chosen valid instrument class and averages over five trials. This makes the reported accuracy and macro F1 numbers in Table 1 dependent on an arbitrary stochastic procedure and does not reflect how the classifier is actually used in the full pipeline. Please report the confusion matrix including the \"No Tool\" column, and either treat \"No Tool\" as background in the downstream segmentation evaluation or justify the random-replacement procedure with a sensitivity analysis.","section":"§4.2, Table 1"}],"minor_comments":[{"comment":"The direct SAM3 baseline is given the ground-truth list of instrument names present in each frame to construct prompts, so the comparison is not strictly apples-to-apples with the proposed pipeline; please state this asymmetry explicitly when interpreting the large improvement over the baseline.","section":"§4.3"},{"comment":"The confidence threshold t=0.2 and the IoU merging threshold m=0.02 are fixed across datasets, and Figure 3 only investigates t on EndoVis 2018. Please clarify how these values were selected and whether the qualitative conclusions are robust to m.","section":"§4.4"},{"comment":"The UNet row on EndoVis 2018 is blank; adding the value would help the reader judge whether the binary segmentation result transfers across datasets.","section":"Table 2"},{"comment":"When Qwen predicts \"No Tool\" during full inference, the paper does not state whether the mask is discarded, assigned to background, or handled differently in constructing the final segmentation map S; please specify this in the method description.","section":"§3.3, §5"}],"recommendation":"major_revision","confidential_remarks":"The decisive question for the editor is whether the authors can supply a genuine mask-free label source for the \"No Tool\" class. If the answer is no, the central claim collapses and the paper should be rejected because the contribution reduces to a supervised classifier trained on mask-derived labels. If the answer is yes, the paper might be salvageable after major revision. The current manuscript does not provide enough information to distinguish these cases, so my recommendation of major revision is conditional on the existence of such a protocol and on the authors' willingness to disclose it fully."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper has a genuinely interesting empirical finding — the generic text prompt \"tool\" gets SAM3 to produce binary surgical masks that are near supervised UNet accuracy on EndoVis — but the headline claim of mask-annotation-free instance segmentation is not supported as written. The label source for the Qwen fine-tuning is undisclosed, and the \"No Tool\" category in §4.2 basically requires knowing which SAM3 masks are false positives, which you can't get without ground truth masks or manual review.\n\nWhat is new and good: The prompt-domain-gap observation is real. Instrument names fail badly (12-16 IoU), \"tool\" gets 74-80 IoU. That alone is a useful result. The oracle analysis is also well done: it separates mask quality from classification error and shows the ceiling is ~61-79 Ch_IoU, which tells you where the bottleneck is. The paper is honest that fully supervised methods are still better and discusses failure modes (fragmentation, merging) carefully.\n\nSoft spots, in order of severity. First, the Qwen training labels. Section 3.3 says masked regions come from SAM3 predictions, but never says what label each region gets. The \"No Tool\" category can only be constructed if you identify false-positive masks, which requires either overlap with ground truth or manual inspection. Either way the \"mask annotation-free\" claim needs qualification or evidence. This is the load-bearing flaw. Second, the eval trick: replacing \"No Tool\" predictions with random valid classes and averaging five trials is odd and possibly inflates metrics; if the classifier says \"not a tool\" you shouldn't silently relabel it. Third, no error bars on the main instance numbers (the binary table has std but the big table doesn't). Fourth, no code or data. The thresholds t=0.2 and m=0.02 are presumably tuned on the validation fold; that's not fatal but should be stated.\n\nNote what the stress-test says about the central claim: I think it holds. Unless the authors disclose the labeling procedure, the abstract's claim is unsupported. This is not a tiny detail; it's the main selling point.\n\nWho is this for: people working on annotation-light surgical vision will find the prompt-gap result and the oracle decomposition useful. It deserves a serious referee: the ambiguity is resolvable, and the binary result is worth publishing even if the instance claim fails. I'd send it to review with a request for the labeling details, not desk-reject.","headline":"Useful empirical study of SAM3 text prompting for surgical instruments, but the mask-annotation-free instance segmentation claim collapses on the undisclosed Qwen label source.","tokens_in":10942,"tokens_out":1710,"would_cite":false,"duration_ms":16946,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SAM3 with the generic prompt 'tool' plus a Qwen classifier gives instance-level surgical instrument segmentation without pixel-level masks or manual prompts.","keywords":["surgical instrument segmentation","Segment Anything Model 3","text-prompted segmentation","vision-language model","zero-shot segmentation","mask annotation-free","EndoVis","Qwen fine-tuning"],"falsifier":"Run the Qwen fine-tuning script with every ground-truth mask file removed from the workspace. If the training set still assigns correct instrument labels and a working \"No Tool\" set to SAM3-generated masks, the mask-annotation-free claim holds; the moment label assignment reads a ground-truth mask, for instance by maximum-overlap matching, the central claim fails.","tokens_in":9891,"feed_emoji":"🩺","tokens_out":8921,"duration_ms":89174,"temperature":0.7,"pith_summary":"Surgical instrument segmentation, a building block for computer-assisted interventions, normally demands pixel-level masks or manual point or box prompts; this paper claims a route that needs neither. The proposal is a two-stage pipeline: prompt SAM3 with the generic word \"tool\" to obtain binary instrument masks, then fine-tune the Qwen vision-language model on those masked regions to assign each mask an instrument label. On EndoVis 2017 and 2018, the pipeline produces instance-level segmentation without ground-truth masks or spatial interaction, clearly outperforming SAM3 prompted with instrument names, although it still trails fully supervised systems. The paper argues this is a viable direction toward annotation-free surgical tool segmentation, which matters because annotation cost is the main scalability bottleneck in the field.","feed_headline":"No mask labels needed: one text prompt segments surgical tools","feed_subtitle":"SAM3 prompted with \"tool\" plus a Qwen classifier beats instrument-name prompting on EndoVis 2017 and 2018.","key_machinery":"The machinery is a two-stage localize-then-identify loop. Stage one treats SAM3 as a zero-shot text-prompted proposal generator: a fixed prompt \"tool\" yields candidate binary masks, filtered by confidence threshold $t=0.2$ and merged by an IoU threshold $m=0.02$. Stage two extracts each mask's region by element-wise multiplication with the RGB frame and feeds it to Qwen 2.5-7B, adapted with rank-8 LoRA on SAM3-generated masked regions, to output one of seven instrument classes or \"No Tool\". The load-bearing operation is the decoupling: the generic prompt \"tool\" avoids the domain gap that breaks instrument-name prompts, and the downstream classifier supplies the semantic label, so the pipeline never needs a spatial prompt or a pixel mask.","core_discovery":"The central claim is that text-prompted segmentation can work in surgery if the text prompt is category-agnostic rather than instrument-specific. SAM3, prompted with \"tool\" and then thresholded and mask-merged, generates binary masks; Qwen, fine-tuned with LoRA on the masked image regions, names each mask's instrument, and a \"No Tool\" class discards false-positive masks. The result is instance-level segmentation maps built from the input image and a fixed text string, with no manual prompt and no pixel-level ground truth. The paper reports binary Dice near 82-85 (comparable to a supervised UNet), instance Ch_IoU of 40.56 and 61.53 across the two EndoVis datasets, and an oracle-classifier analysis showing that with a perfect classifier the same masks would reach Ch_IoU of 61.49 and 79.39, close to some supervised methods.","pith_inferences":["Editorial extension: the paper never says how each SAM3 mask gets its instrument label during fine-tuning; if the labels come from maximum-overlap matching to ground-truth masks, the honest claim shrinks to \"no pixel-level mask supervision for the segmentation stage,\" not \"no mask annotation at all.\"","Editorial extension: if \"No Tool\" examples are instead defined by low confidence or Qwen's own uncertainty, the same design becomes a self-training loop that could run on fully unlabelled surgical videos, an extension the paper allows but does not state.","Editorial extension: the oracle results imply that investing in better text-prompted or language-only classification is likely to raise instance IoU more than tuning SAM3's mask thresholds, because the oracle nearly closes the gap to supervised methods."],"forward_implications":["Binary instrument detection reaches Dice of 82.10 on EndoVis 2017 and 84.93 on EndoVis 2018 with the generic \"tool\" prompt, essentially matching a fully supervised UNet without training a segmentation model.","Instance-level Ch_IoU improves from 4.23 to 40.56 on EndoVis 2017 and from 7.79 to 61.53 on EndoVis 2018 compared with prompting SAM3 by instrument name.","A classifier fine-tuned on SAM3-generated masks rather than ground-truth masks loses only modest accuracy (e.g., 72.33 vs 76.67 on EndoVis 2018) and transfers across datasets with nearly equivalent performance on shared instrument categories.","An oracle classifier raises Ch_IoU to 61.49 on EndoVis 2017 and 79.39 on EndoVis 2018, showing mask quality from SAM3 is not the dominant bottleneck; classification quality is.","Mask errors from the first stage, such as fragmentation, overlap, and missed detections, propagate into the classification stage and account for the main failure modes, so future gains depend on improving SAM3 mask quality or adding a refinement stage."],"supporting_citations":[{"why":"SAM3 supplies the text-prompted zero-shot mask generator that forms stage one of the pipeline.","marker":"[8]"},{"why":"Qwen2.5-VL is the vision-language model fine-tuned in stage two to classify each masked region.","marker":"[7]"},{"why":"ISINet provides the EndoVis 2018 label splits and the Ch_IoU, ISI_IoU, and mc_IoU metrics used for instance evaluation.","marker":"[10]"},{"why":"Establishes the EndoVis 2017 four-fold protocol and serves as a supervised baseline the framework compares against.","marker":"[24]"},{"why":"Prior comparative evaluation of vision-language models that motivates the choice of Qwen over alternatives.","marker":"[19]"},{"why":"LoRA is the parameter-efficient adaptation method used to fine-tune Qwen on masked regions.","marker":"[12]"},{"why":"Provides the EndoVis 2017 dataset used for binary and instance segmentation evaluation.","marker":"[3]"},{"why":"Provides the EndoVis 2018 dataset and its train/test sequence split.","marker":"[2]"},{"why":"GraSP dataset is used to test Qwen's cross-dataset generalization on shared instrument categories.","marker":"[5]"}],"fun_headline_variants":["One word 'tool' prompts annotation-free surgical instrument segmentation","Text prompt 'tool' replaces mask labels for surgical tool segmentation","SAM3 plus 'tool' prompt: no masks, no manual clicks for surgery segmentation","Category-agnostic prompt beats instrument names for SAM3 surgery masks","Annotation-free surgery masks via 'tool' prompt and Qwen classifier"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework is only genuinely mask annotation-free if every label used to fine-tune Qwen, including the \"No Tool\" category, can be produced from SAM3 masks and text alone, without ever consulting a ground-truth mask.","fun_headline_variants_meta":{"raw":{"variants":["One word 'tool' prompts annotation-free surgical instrument segmentation","Text prompt 'tool' replaces mask labels for surgical tool segmentation","SAM3 plus 'tool' prompt: no masks, no manual clicks for surgery segmentation","Category-agnostic prompt beats instrument names for SAM3 surgery masks","Annotation-free surgery masks via 'tool' prompt and Qwen classifier"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1331,"prompt_tokens":975,"completion_tokens":356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":264}},"tokens_in":591,"tokens_out":356,"duration_ms":4170,"temperature":1.0,"reasoning_tokens":264,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:22:58.732357+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Qwen fine-tuning script with every ground-truth mask file removed from the workspace. If the training set still assigns correct instrument labels and a working \"No Tool\" set to SAM3-generated masks, the mask-annotation-free claim holds; the moment label assignment reads a ground-truth mask, for instance by maximum-overlap matching, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ISINet provides the EndoVis 2018 label splits and the Ch_IoU, ISI_IoU, and mc_IoU metrics used for instance evaluation."},{"cited_title":"arXiv preprint arXiv:2601.16895 (2026)","cited_arxiv_id":null,"evidence_quote":"Prior comparative evaluation of vision-language models that motivates the choice of Qwen over alternatives."},{"cited_title":"Medical Image Analysis p","cited_arxiv_id":null,"evidence_quote":"GraSP dataset is used to test Qwen's cross-dataset generalization on shared instrument categories."}],"review_version":1}