{"id":"802ffbe7-1fd5-4998-ab6e-a3f8bcfbc33e","arxiv_id":"2411.12199","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new robust text-promptable surgical instrument segmentation task and a framework that predicts instrument presence before generating masks, reducing false positives when prompts refer to absent instruments.","lead":"The authors introduce a new evaluation protocol, R-SIS, in which text-promptable surgical instrument segmentation is tested by prompting every instrument category without knowing which are present, and they build a system, RoSIS, that decides presence before emitting masks. The result is a benchmark and method that cuts false-positive masks when instruments are absent, a condition that previous promptable surgical segmentation methods did not handle.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core FPR claim in Table 5 is not independently checkable: the metric is undefined, the baselines are unreleased reimplementations, and the 0.5 threshold is unexamined.","rationale":"The reader's conditional verdict is reasonable. The proposed task and framework have plausible independent support: the ablations show each added component helps, and the cross-dataset results are internally consistent. However, the headline quantitative claim is the false-positive reduction in Table 5, and that table's evidential weight depends on a defined metric, faithful baselines, and a non-arbitrary operating point. The paper does not provide the first two, and the third is only asserted through Eq. (4). These are not internal contradictions, but they are unverified parts of an empirical claim. I agree with the reader that baseline fidelity is the weakest spot; I would add that the missing FPR definition is a separate, equally load-bearing gap that is cheaper to close. The correct outcome remains CONDITIONAL: the central idea is credible, but the evidence as written cannot be independently checked without code, metric definitions, and baseline validation.","tokens_in":15881,"tokens_out":6611,"duration_ms":73589,"concrete_test":"Release the R-SIS evaluation code with a precise FPR/P/R/F1 definition, and rerun Table 5 using the official LA VT and TP-SIS implementations (or the authors' exact reimplementation with a stated training recipe) under the same R-SIS protocol. If official baselines reproduce the reported FPRs to within roughly 0.05, the central gap is genuine; if official baselines yield materially lower FPR (e.g., LA VT below about 0.3), the substantial-false-positive-reduction claim is overstated. In the same release, include a threshold sweep for Eq. (4) over 0.1-0.9 to verify the reported F1 is not a single-threshold artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that RoSIS substantially reduces false-positive predictions compared to prior promptable segmentation models (§4.3.3, Table 5) — depends entirely on the numbers in Table 5. Two necessary conditions are not met in the manuscript. First, Section 4.1 specifies only Ch IoU, ISI IoU, and mc IoU; the FPR, precision, recall, and F1 in Table 5 are never defined. Without knowing whether FPR is per-pixel, per-class, or per-image, and how masks from the seven prompts are aggregated, the headline gap (0.0892 vs 0.6155 for LA VT on EndoVis2018) cannot be reproduced or audited. Second, Section 4.2 states that LA VT and TP-SIS are re-implemented, but no code, training recipe, hyperparameters, or comparison to the original checkpoints or reported numbers is provided. If the reimplemented baselines are undertuned, or if they were trained with the original positive-only prompt distribution while RoSIS is trained with negative prompts (§3.2), the FPR gap partly measures a training-data difference rather than the proposed existence-prediction mechanism. The fixed 0.5 threshold in Eq. (4) is also unexamined; a threshold sweep is needed to show the FPR benefit is not an artifact of one operating point.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new evaluation task, Robust text-promptable Surgical Instrument Segmentation (R-SIS), in which all candidate instrument categories are prompted without oracle information about which instruments are present, and a model must predict instrument existence and generate masks only for present instruments. The authors present RoSIS, a promptable segmentation framework with an existence-prediction branch, multi-modal fusion blocks, and an iterative refinement strategy using name, GPT-4, and location prompts. They evaluate on EndoVis2017 and EndoVis2018 and compare against vision-based and promptable baselines, with the central claim that RoSIS substantially reduces false-positive predictions compared to prior promptable models (e.g., Table 5: FPR 0.0892 vs. 0.6155 for LA VT on EndoVis2018) and that previous oracle-based evaluation protocols overstate practical reliability.","tokens_in":16180,"tokens_out":5397,"duration_ms":50402,"significance":"The R-SIS task is a timely and practically motivated reformulation of promptable surgical instrument segmentation, and the paper correctly identifies a real limitation of existing evaluations that assume prompt presence or use oracle information. The within-model ablation in Table 5 (RoSIS w/o exist vs. RoSIS) provides some evidence for the benefit of the existence-prediction branch, and the use of two public benchmarks is a strength. If the comparative results are reproducible, the paper would be a valuable reference for the community. However, the main comparative evidence is not yet auditable because the key metrics are undefined, the baselines are unreleased re-implementations, and the operating point is not justified.","major_comments":[{"comment":"Table 5 reports FPR, Precision, Recall, and F1 for the R-SIS setting, but Section 4.1 defines only Ch IoU, ISI IoU, and mc IoU. The manuscript never states whether FPR is per-pixel, per-image, or per-class; how the masks and existence decisions for the multiple prompts are aggregated; or how absent classes are counted in the denominator. Since Section 4.3.3's central claim of substantially reducing false-positive predictions rests entirely on this table, these definitions are necessary for the results to be reproduced or audited.","section":"§4.1, §4.3.3, Table 5"},{"comment":"The promptable baselines LA VT and TP-SIS are re-implemented by the authors, but no code, training recipe, hyperparameters, or validation against original checkpoints or reported numbers is provided. The R-SIS protocol changes the prompt distribution (prompts for absent classes are introduced in Section 3.2), and it is unclear whether the re-implemented baselines were trained with the same negative-prompt distribution as RoSIS or with the original positive-only distribution. If the latter, the FPR gap in Table 5 partly measures a training-set difference rather than the proposed existence-prediction mechanism. The authors should release the re-implementations or provide detailed training recipes and original-protocol verification to make the comparison trustworthy.","section":"§4.2, Tables 3–5, 8"},{"comment":"The existence decision uses a fixed threshold of 0.5 on the averaged existence probabilities, with no justification or sensitivity analysis. Since this threshold directly controls which masks survive and therefore determines the FPR and precision numbers in Table 5, the paper should report a threshold sweep or a precision-recall curve to show that the FPR benefit of RoSIS is not an artifact of a single operating point.","section":"§3.4, Eq. (4)"},{"comment":"The cross-dataset study does not specify how the two unique categories of each dataset are handled when training on one dataset and testing on the other. If the evaluation includes only the five shared classes, this should be stated; if it includes unique classes, the prompts, ground-truth mapping, and evaluation protocol need to be described. Without this, the Ch IoU/ISI IoU/mc IoU values in Table 8 are hard to interpret.","section":"§4.6, Table 8"}],"minor_comments":[{"comment":"The value of the mask-loss weight λ in Eq. (3) is not reported in the implementation details; please provide it for reproducibility.","section":"§3.3, Eq. (3)"},{"comment":"The construction of the 'RoSIS w/o exist' variant is not described; it is unclear whether it outputs masks for all prompts without any existence filtering or uses some other selection rule.","section":"Table 5"},{"comment":"The claim that S3Net 'selects the top 5 instances out of 7 instances during inference' is stated without a citation or supporting evidence; please provide a reference or explicit experimental basis.","section":"§4.3.2"},{"comment":"The definitions of Ch IoU, ISI IoU, and mc IoU are informal; please give precise formulas and state how the masks from multiple prompts are merged for the promptable methods.","section":"§4.1"},{"comment":"No variance or statistical significance is reported for any experimental table; for EndoVis2017, which uses 4-fold cross-validation, mean and standard deviation should be reported.","section":"Tables 3–8"},{"comment":"There is a typo 'groud truth' in the caption of Fig. 5, and the qualitative figure does not show failure cases of the proposed method.","section":"§4.5"}],"recommendation":"major_revision","confidential_remarks":"The paper's main novelty is the R-SIS evaluation protocol, and the comparative FPR claim is its primary evidence. That evidence currently rests on metrics that are not defined in the manuscript and on baselines that are not released or verified. These issues are fixable with a careful revision, but until then the central comparative claims are not independently checkable. I would also note that defining a new task and then evaluating one's own method on it is not inherently circular, but it places a higher burden on transparent reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the R-SIS protocol is a real contribution to surgical vision, and the existence-prediction branch does what it claims in the within-model ablation. But Table 5's FPR numbers are not reproducible as written, and the baselines are unreleased reimplementations. That makes the central cross-method claim weaker than the authors present it.\n\nThe task formulation is the main thing. Prompting all candidate classes and requiring the model to decline absent instruments is a stricter and more realistic protocol than what prior promptable SIS papers used, which relied on oracle presence information. The four-quadrant location prompt refinement is a sensible, simple idea, and the ablation study shows it helps. The authors also deserve credit for not overselling novelty: they cite RefSegformer and GRES as the source of the existence-prediction idea, and the paper is framed as an adaptation of robust referring segmentation to surgery.\n\nThe soft spots are mostly in the evaluation. Section 4.1 defines Ch IoU, ISI IoU, and mc IoU, but Table 5 reports FPR, precision, recall, and F1 without any definitions. Without knowing whether FPR is per-pixel, per-class, or per-image, the headline gap (0.0892 vs 0.6155 on EV18) cannot be audited. Second, LA VT and TP-SIS are re-implemented with no code, hyperparameters, or comparison to original checkpoints. If those re-implementations are undertuned, the false-positive gap is overstated. That concern is partially mitigated by the RoSIS w/o exist ablation, which shows the existence branch itself cuts FPR from 0.1804 to 0.0892 on EV18—that comparison is within their own controlled model and is the strongest evidence in the paper. But the claim that RoSIS beats prior art is only as good as the baselines. Third, the 0.5 threshold in Eq. (4) is fixed without justification or a sweep, and no error bars or significance tests are reported anywhere.\n\nThis is a paper for people building or evaluating promptable surgical segmentation systems, and for anyone thinking about evaluation protocols that drop oracle assumptions. It deserves a serious referee, though the revision needs to make the evidence checkable: define FPR, release baseline code, and add at least one threshold sensitivity analysis. The task is likely to outlive the specific model.","headline":"Useful R-SIS task and plausible existence-prediction mechanism, but the FPR evidence in Table 5 is not auditable and the baselines are unreleased re-implementations.","tokens_in":16683,"tokens_out":4692,"would_cite":true,"duration_ms":45868,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Promptable surgical instrument segmentation is currently evaluated with oracle knowledge of which instruments are present; this paper argues that removing that oracle (R-SIS) exposes large false-positive rates, and proposes a…","keywords":["surgical instrument segmentation","text-promptable segmentation","vision-language models","existence prediction","false-positive reduction","robust evaluation protocol","EndoVis datasets","referring image segmentation"],"falsifier":"Run the R-SIS protocol with the original released models or exact training recipes of LA VT, TP-SIS, and RefSegformer; if their false-positive rates approach RoSIS's 0.0892 on EndoVis2018, the paper's central claim is weakened. A simpler check is to sweep the 0.5 existence threshold in Eq. (4) and see whether RoSIS's advantage persists across thresholds.","tokens_in":15632,"feed_emoji":"🩺","tokens_out":5906,"duration_ms":57467,"temperature":0.7,"pith_summary":"The paper argues that current text-promptable surgical instrument segmentation is evaluated under unrealistic oracle conditions: prompts are issued only for instruments known to be present. It defines a new task, Robust text-promptable Surgical Instrument Segmentation (R-SIS), in which all candidate categories are prompted and the model must decide existence before producing a mask. Evaluating existing promptable methods under R-SIS produces large false-positive rates (for example, 0.6155 on EndoVis2018 for a strong baseline), whereas the proposed RoSIS model, which predicts existence probability and refines masks iteratively, reaches 0.0892 FPR on the same dataset while improving precision and recall. The point of this reformulation is that practical surgical systems do not know which instruments are in view; if the paper is right, prior promptable SIS accuracy claims overstate real-world reliability.","feed_headline":"Presence check before surgical masks cuts false positives sharply","feed_subtitle":"A new benchmark prompts every instrument name; RoSIS masks only those actually in view.","key_machinery":"The load-bearing mechanism is the existence-probability branch plus a two-iteration refinement protocol. The model computes an existence probability $p_c$ for each prompted class by fusing decoder features with raw BERT language features through multi-head cross-attention, and it gates mask generation on that probability. Equation (4) is the central identity: if the average existence probability from the name and descriptive prompts is below 0.5, the final mask is zero; otherwise the masks from those prompts are averaged, and a third location-guided mask is added only when its own existence probability also clears 0.5. The encoder inserts Multi-Modal Fusion Blocks and Selective Gate Blocks between Swin Transformer stages so language and vision features interact early, but the robustness gain is attributed mainly to presence verification rather than to architectural complexity.","core_discovery":"The paper's central claim is that false-positive masks in text-promptable surgical instrument segmentation are largely driven by the evaluation protocol, not by the segmentation backbone, and that adding an explicit existence-prediction branch before mask generation removes most of them. On EndoVis2018 under the R-SIS protocol, RoSIS reduces FPR from 0.6155 for LA VT and 0.4008 for TP-SIS to 0.0892, while raising F1 from 0.3364 and 0.4957 to 0.7611; on EndoVis2017, FPR drops from 0.7744 for LA VT and 0.3338 for TP-SIS to 0.2343, and F1 rises from 0.1483 and 0.3981 to 0.5269. The paper interprets this as showing that previous oracle-based promptable results are misleading and that R-SIS should be the benchmark for promptable segmentation in surgery.","pith_inferences":["An extension the paper leaves implicit: the R-SIS protocol could transfer to other promptable segmentation domains where false positives for absent classes are costly, such as natural-language-driven robotics or medical image analysis beyond surgery.","Because the existence threshold is fixed at 0.5, a calibration sweep could shift the precision-recall balance; rare instruments such as the clip applier, which scores 0.00 IoU in the EndoVis2018 table, may need class-specific thresholds to avoid being suppressed.","The cross-dataset experiments suggest location prompts learned from one surgical domain do not fully transfer; testing a location-agnostic refinement prompt would isolate whether the gains come from position cues or from the second inference pass itself."],"forward_implications":["Numbers reported under oracle-based promptable SIS protocols should be treated as upper bounds; R-SIS numbers are the ones that reflect deployment conditions.","Once the oracle is removed, a promptable model with existence prediction can be compared fairly with vision-only SIS models on the same benchmark.","Existence prediction alone, even without the rest of RoSIS, lowers FPR from 0.1804 to 0.0892 on EndoVis2018, so presence verification is the main robustness mechanism.","Iterative refinement with location prompts produces measurable gains: adding it to name prompts raises ISI IoU from 69.24 to 73.51 on EndoVis2018, and adding it to GPT-4 descriptive prompts raises ISI IoU from 71.37 to 76.16.","Future promptable surgical segmentation models should be evaluated by prompting all candidate classes and reporting FPR, precision, recall, and F1 alongside IoU."],"supporting_citations":[{"why":"Supplies the EndoVis2017 robotic instrument segmentation dataset used for R-SIS evaluation and cross-dataset tests.","marker":"[24]"},{"why":"Supplies the EndoVis2018 robotic scene segmentation dataset that drives the main comparative and ablation tables.","marker":"[25]"},{"why":"TP-SIS is the principal prior text-promptable surgical instrument segmentation method that assumes oracle presence information and is re-implemented as a baseline under R-SIS.","marker":"[9]"},{"why":"LA VT is the language-aware vision transformer baseline re-implemented under R-SIS and carries the largest false-positive rate in the comparison.","marker":"[19]"},{"why":"RefSegformer supplies the robust referring-image-segmentation idea of handling empty-target prompts, which motivates the existence-prediction branch and serves as a comparison baseline.","marker":"[15]"},{"why":"GRES introduces empty-target and multi-target referring segmentation scenarios, which frame the R-SIS task formulation.","marker":"[22]"},{"why":"GPT-4 generates the descriptive prompts used during training and inference to enrich text cues beyond class names.","marker":"[14]"},{"why":"The deformable attention pixel decoder is the architectural component that produces the multi-scale features from which masks are decoded.","marker":"[33]"}],"fun_headline_variants":["Existence check slashes false positives in prompt-driven surgical masks","New R-SIS benchmark cuts false positives in text-prompted surgery","Adding presence check halves false positives in surgical segmentation","Promptable surgical masks get a presence check, false positives drop","RoSIS: prompt every instrument, mask only visible ones, fewer errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the re-implemented versions of LA VT and TP-SIS are faithful, comparably tuned renditions of the originals, so the measured false-positive gap reflects the task protocol rather than undertuned baselines.","fun_headline_variants_meta":{"raw":{"variants":["Existence check slashes false positives in prompt-driven surgical masks","New R-SIS benchmark cuts false positives in text-prompted surgery","Adding presence check halves false positives in surgical segmentation","Promptable surgical masks get a presence check, false positives drop","RoSIS: prompt every instrument, mask only visible ones, fewer errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1302,"prompt_tokens":972,"completion_tokens":330,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":242}},"tokens_in":588,"tokens_out":330,"duration_ms":4643,"temperature":1.0,"reasoning_tokens":242,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:48:54.907588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the R-SIS protocol with the original released models or exact training recipes of LA VT, TP-SIS, and RefSegformer; if their false-positive rates approach RoSIS's 0.0892 on EndoVis2018, the paper's central claim is weakened. A simpler check is to sweep the 0.5 existence threshold in Eq. (4) and see whether RoSIS's advantage persists across thresholds.","supporting_citations":[{"cited_title":"”Text promptable surgical instrument segmentation with vision- language models.” Advances in Neural Information Processing Systems 36 (2023): 28611-28623","cited_arxiv_id":null,"evidence_quote":"TP-SIS is the principal prior text-promptable surgical instrument segmentation method that assumes oracle presence information and is re-implemented as a baseline under R-SIS."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LA VT is the language-aware vision transformer baseline re-implemented under R-SIS and carries the largest false-positive rate in the comparison."},{"cited_title":"”Towards robust referring image segmentation.” IEEE Trans- actions on Image Processing (2024)","cited_arxiv_id":null,"evidence_quote":"RefSegformer supplies the robust referring-image-segmentation idea of handling empty-target prompts, which motivates the existence-prediction branch and serves as a comparison baseline."},{"cited_title":"”Gres: Generalized referring expression segmentation.” Proceedings of the IEEE /CVF conference on computer vision and pattern recognition","cited_arxiv_id":null,"evidence_quote":"GRES introduces empty-target and multi-target referring segmentation scenarios, which frame the R-SIS task formulation."}],"review_version":1}