REVIEW 4 major objections 6 minor 35 references
Rethinking Text-Promptable Surgical Instrument Segmentation with Robust Framework
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Promptable surgical instrument segmentation is currently evaluated with oracle knowledge of which instruments are present; this paper argues that removing that oracle (R-SIS) exposes large false-positive rates, and proposes a…
desk verdict Useful R-SIS task and plausible existence-prediction mechanism, but the FPR evidence in Table 5 is not auditable and the baselines are unreleased re-implementations. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the existence-probability branch plus a two-iteration refinement protocol. The model computes an existence probability $p_c$ for each prompted class by fusing decoder features with raw BERT language features through multi-head cross-attention, and it gates mask generation on that probability. Equation (4) is the central identity: if the average existence probability from the name and descriptive prompts is below 0.5, the final mask is zero; otherwise the masks from those prompts are averaged, and a third location-guided mask is added only when its own existence probability also clears 0.5. The encoder inserts Multi-Modal Fusion Blocks and Selective Gate Blocks between Swin Transformer stages so language and vision features interact early, but the robustness gain is attributed mainly to presence verification rather than to architectural complexity.
What would settle it
Run the R-SIS protocol with the original released models or exact training recipes of LA VT, TP-SIS, and RefSegformer; if their false-positive rates approach RoSIS's 0.0892 on EndoVis2018, the paper's central claim is weakened. A simpler check is to sweep the 0.5 existence threshold in Eq. (4) and see whether RoSIS's advantage persists across thresholds.
Extended reading notes
Core claim
The paper's central claim is that false-positive masks in text-promptable surgical instrument segmentation are largely driven by the evaluation protocol, not by the segmentation backbone, and that adding an explicit existence-prediction branch before mask generation removes most of them. On EndoVis2018 under the R-SIS protocol, RoSIS reduces FPR from 0.6155 for LA VT and 0.4008 for TP-SIS to 0.0892, while raising F1 from 0.3364 and 0.4957 to 0.7611; on EndoVis2017, FPR drops from 0.7744 for LA VT and 0.3338 for TP-SIS to 0.2343, and F1 rises from 0.1483 and 0.3981 to 0.5269. The paper interprets this as showing that previous oracle-based promptable results are misleading and that R-SIS should be the benchmark for promptable segmentation in surgery.
Load-bearing premise
The load-bearing premise is that the re-implemented versions of LA VT and TP-SIS are faithful, comparably tuned renditions of the originals, so the measured false-positive gap reflects the task protocol rather than undertuned baselines.
Editorial extensions
If this is right
- Numbers reported under oracle-based promptable SIS protocols should be treated as upper bounds; R-SIS numbers are the ones that reflect deployment conditions.
- Once the oracle is removed, a promptable model with existence prediction can be compared fairly with vision-only SIS models on the same benchmark.
- Existence prediction alone, even without the rest of RoSIS, lowers FPR from 0.1804 to 0.0892 on EndoVis2018, so presence verification is the main robustness mechanism.
- Iterative refinement with location prompts produces measurable gains: adding it to name prompts raises ISI IoU from 69.24 to 73.51 on EndoVis2018, and adding it to GPT-4 descriptive prompts raises ISI IoU from 71.37 to 76.16.
- Future promptable surgical segmentation models should be evaluated by prompting all candidate classes and reporting FPR, precision, recall, and F1 alongside IoU.
Reading between the lines
- An extension the paper leaves implicit: the R-SIS protocol could transfer to other promptable segmentation domains where false positives for absent classes are costly, such as natural-language-driven robotics or medical image analysis beyond surgery.
- Because the existence threshold is fixed at 0.5, a calibration sweep could shift the precision-recall balance; rare instruments such as the clip applier, which scores 0.00 IoU in the EndoVis2018 table, may need class-specific thresholds to avoid being suppressed.
- The cross-dataset experiments suggest location prompts learned from one surgical domain do not fully transfer; testing a location-agnostic refinement prompt would isolate whether the gains come from position cues or from the second inference pass itself.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a new evaluation task, Robust text-promptable Surgical Instrument Segmentation (R-SIS), in which all candidate instrument categories are prompted without oracle information about which instruments are present, and a model must predict instrument existence and generate masks only for present instruments. The authors present RoSIS, a promptable segmentation framework with an existence-prediction branch, multi-modal fusion blocks, and an iterative refinement strategy using name, GPT-4, and location prompts. They evaluate on EndoVis2017 and EndoVis2018 and compare against vision-based and promptable baselines, with the central claim that RoSIS substantially reduces false-positive predictions compared to prior promptable models (e.g., Table 5: FPR 0.0892 vs. 0.6155 for LA VT on EndoVis2018) and that previous oracle-based evaluation protocols overstate practical reliability.
Significance. The R-SIS task is a timely and practically motivated reformulation of promptable surgical instrument segmentation, and the paper correctly identifies a real limitation of existing evaluations that assume prompt presence or use oracle information. The within-model ablation in Table 5 (RoSIS w/o exist vs. RoSIS) provides some evidence for the benefit of the existence-prediction branch, and the use of two public benchmarks is a strength. If the comparative results are reproducible, the paper would be a valuable reference for the community. However, the main comparative evidence is not yet auditable because the key metrics are undefined, the baselines are unreleased re-implementations, and the operating point is not justified.
major comments (4)
- [§4.1, §4.3.3, Table 5] Table 5 reports FPR, Precision, Recall, and F1 for the R-SIS setting, but Section 4.1 defines only Ch IoU, ISI IoU, and mc IoU. The manuscript never states whether FPR is per-pixel, per-image, or per-class; how the masks and existence decisions for the multiple prompts are aggregated; or how absent classes are counted in the denominator. Since Section 4.3.3's central claim of substantially reducing false-positive predictions rests entirely on this table, these definitions are necessary for the results to be reproduced or audited.
- [§4.2, Tables 3–5, 8] The promptable baselines LA VT and TP-SIS are re-implemented by the authors, but no code, training recipe, hyperparameters, or validation against original checkpoints or reported numbers is provided. The R-SIS protocol changes the prompt distribution (prompts for absent classes are introduced in Section 3.2), and it is unclear whether the re-implemented baselines were trained with the same negative-prompt distribution as RoSIS or with the original positive-only distribution. If the latter, the FPR gap in Table 5 partly measures a training-set difference rather than the proposed existence-prediction mechanism. The authors should release the re-implementations or provide detailed training recipes and original-protocol verification to make the comparison trustworthy.
- [§3.4, Eq. (4)] The existence decision uses a fixed threshold of 0.5 on the averaged existence probabilities, with no justification or sensitivity analysis. Since this threshold directly controls which masks survive and therefore determines the FPR and precision numbers in Table 5, the paper should report a threshold sweep or a precision-recall curve to show that the FPR benefit of RoSIS is not an artifact of a single operating point.
- [§4.6, Table 8] The cross-dataset study does not specify how the two unique categories of each dataset are handled when training on one dataset and testing on the other. If the evaluation includes only the five shared classes, this should be stated; if it includes unique classes, the prompts, ground-truth mapping, and evaluation protocol need to be described. Without this, the Ch IoU/ISI IoU/mc IoU values in Table 8 are hard to interpret.
minor comments (6)
- [§3.3, Eq. (3)] The value of the mask-loss weight λ in Eq. (3) is not reported in the implementation details; please provide it for reproducibility.
- [Table 5] The construction of the 'RoSIS w/o exist' variant is not described; it is unclear whether it outputs masks for all prompts without any existence filtering or uses some other selection rule.
- [§4.3.2] The claim that S3Net 'selects the top 5 instances out of 7 instances during inference' is stated without a citation or supporting evidence; please provide a reference or explicit experimental basis.
- [§4.1] The definitions of Ch IoU, ISI IoU, and mc IoU are informal; please give precise formulas and state how the masks from multiple prompts are merged for the promptable methods.
- [Tables 3–8] No variance or statistical significance is reported for any experimental table; for EndoVis2017, which uses 4-fold cross-validation, mean and standard deviation should be reported.
- [§4.5] There is a typo 'groud truth' in the caption of Fig. 5, and the qualitative figure does not show failure cases of the proposed method.
Circularity Check
No significant circularity: RoSIS is a supervised architecture evaluated on external benchmarks; the R-SIS protocol is a task definition, not a fitted prediction.
full rationale
The paper's load-bearing claims are empirical evaluations, not derivations from their own outputs. RoSIS's existence branch is trained with ground-truth presence labels via Eq. (3), and the 0.5 threshold in Eq. (4) is a fixed, untuned decision rule, so the FPR reduction in Table 5 is a measured outcome rather than a fitted quantity relabeled as a prediction. The comparison baselines (LA VT, RefSegformer, TP-SIS) are external methods evaluated under the newly defined R-SIS protocol of prompting all classes; the protocol itself is a definition, and observing that methods without an existence gate produce false positives is an empirical consequence, not a circular derivation. The clearest concerns are unstated FPR definitions, unreleased re-implementations, and possible training-distribution mismatch from the authors' negative-prompt construction, but these bear on reproducibility and fairness of comparison, not on circularity. The paper contains no load-bearing self-citations and no uniqueness theorem imported from the authors' prior work, and the main ablations are grounded in held-out EndoVis2017/2018 test folds.
Assumptions & free parameters
free parameters (5)
- Existence threshold =
0.5
- Loss weight lambda (mask loss weight) =
not reported
- Language token size =
20
- Number of location quadrants =
4
- Iteration count for inference =
2
assumptions (5)
- domain assumption Ground-truth instrument presence is a well-defined binary property that an existence classifier can learn from the dataset labels.
- ad hoc to paper A fixed four-quadrant discretization is sufficient to represent instrument locations for the location prompts.
- domain assumption Pre-trained Swin Transformer and BERT features transfer to endoscopic surgical images with only fine-tuning.
- ad hoc to paper The re-implemented baseline methods are faithful and comparably tuned.
- ad hoc to paper The average of two existence probabilities from name and GPT-4 prompts is a reliable presence signal.
Cite this review
Pith. "Pith review of Rethinking Text-Promptable Surgical Instrument Segmentation with Robust Framework." pith.science (2026). https://pith.science/paper/DKIY74XA
@misc{pith2026241112199,
author = {Pith},
title = {Pith review of: Rethinking Text-Promptable Surgical Instrument Segmentation with Robust Framework},
year = {2026},
howpublished = {\url{https://pith.science/paper/DKIY74XA}},
note = {Machine review of arXiv:2411.12199}
}
read the original abstract
Surgical instrument segmentation is an essential component of computer-assisted and robotic surgery systems. Vision-based segmentation models typically produce outputs limited to a predefined set of instrument categories, which restricts their applicability in interactive systems and robotic task automation. Promptable segmentation methods allow selective predictions based on textual prompts. However, they often rely on the assumption that the instruments present in the scene are already known, and prompts are generated accordingly, limiting their ability to generalize to unseen or dynamically emerging instruments. In practical surgical environments, where instrument existence information is not provided, this assumption does not hold consistently, resulting in false-positive segmentation. To address these limitations, we formulate a new task called Robust text-promptable Surgical Instrument Segmentation (R-SIS). Under this setting, prompts are issued for all candidate categories without access to instrument presence information. R-SIS requires distinguishing which prompts refer to visible instruments and generating masks only when such instruments are explicitly present in the scene. This setting reflects practical conditions where uncertainty in instrument presence is inherent. We evaluate existing segmentation methods under the R-SIS protocol using surgical video datasets and observe substantial false-positive predictions in the absence of ground-truth instruments. These findings demonstrate a mismatch between current evaluation protocols and real-world use cases, and support the need for benchmarks that explicitly account for prompt uncertainty and instrument absence.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Iglovikov Vladimir, Alexey Shvets. ”Ternausnet: U-net with vgg11 encoder pre-trained on imagenet for image segmentation.” arXiv preprint arXiv:1801.05746 (2018)
arXiv 2018
-
[2]
Gonz ´alez Cristina, Laura Bravo-S´anchez, Pablo Arbelaez. ”Isinet: an instance-based approach for surgical instrument segmentation.” Inter- national conference on medical image computing and computer-assisted intervention. Cham: Springer International Publishing, 2020
work page 2020
-
[3]
Zhao Zixu, Yueming Jin, Pheng–Ann Heng. ”Trasetr: track-to-segment transformer with contrastive query for instance-level instrument segmentation in robotic surgery.” 2022 International conference on robotics and automation (ICRA). IEEE, 2022
work page 2022
-
[4]
Ayobi Nicol ´as, P´erez-Rond´on Alejandra, Rodr´ıguez Santiago, Arbel´aez Pablo. ”Matis: Masked-attention transformers for surgical instrument segmentation.” 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI). IEEE, 2023
work page 2023
-
[5]
Britty Baby, Daksh Thapar, Mustafa Chasmai, Tamajit Banerjee, Kunal Dargan, Ashish Suri, Subhashis Banerjee, Chetan Arora. ”From forks to forceps: A new framework for instance segmentation of surgical instruments.” Proceedings of the IEEE /CVF winter conference on applications of computer vision. 2023
work page 2023
-
[6]
”Mask r-cnn.” Proceedings of the IEEE international conference on computer vision
Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick. ”Mask r-cnn.” Proceedings of the IEEE international conference on computer vision. 2017. 13
work page 2017
-
[7]
Ronneberger Olaf, Philipp Fischer, Thomas Brox. ”U-net: Convolutional networks for biomedical image segmentation.” Medical image com- puting and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer international publishing, 2015
work page 2015
-
[8]
Schwing, Alexander Kirillov, Rohit Girdhar
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, Rohit Girdhar. ”Masked-attention mask transformer for universal image segmentation.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022
work page 2022
Show all 35 references
-
[9]
”Text promptable surgical instrument segmentation with vision- language models.” Advances in Neural Information Processing Systems 36 (2023): 28611-28623
Zijian Zhou, Oluwatosin Alabi, Meng Wei, Tom Vercauteren, Miaojing Shi. ”Text promptable surgical instrument segmentation with vision- language models.” Advances in Neural Information Processing Systems 36 (2023): 28611-28623
2023
-
[10]
”Surgicalsam: E fficient class promptable surgical instrument seg- mentation.” Proceedings of the AAAI Conference on Artificial Intelligence
Wenxi Yue, Jing Zhang, Kun Hu, Yong Xia, Jiebo Luo, Zhiyong Wang. ”Surgicalsam: E fficient class promptable surgical instrument seg- mentation.” Proceedings of the AAAI Conference on Artificial Intelligence. V ol. 38. No. 7. 2024
2024
-
[11]
”Part to whole: Collaborative prompting for surgical instrument segmentation.” CoRR (2023)
Wenxi Yue, Jing Zhang, Kun Hu, Qiuxia Wu, Zongyuan Ge, Yong Xia, Jiebo Luo, Zhiyong Wang. ”Part to whole: Collaborative prompting for surgical instrument segmentation.” CoRR (2023)
2023
-
[12]
Berg, Wan-Yen Lo, Piotr Dollar, Ross Girshick
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, Ross Girshick. ”Segment anything.” Proceedings of the IEEE /CVF international conference on computer vision. 2023
2023
-
[13]
”Learning transferable visual models from natural language supervision.” Interna- tional conference on machine learning
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever. ”Learning transferable visual models from natural language supervision.” Interna- tional confer...
2021
-
[14]
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Al- tenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo- hammad Bavarian, J...
2023 arXiv
-
[15]
”Towards robust referring image segmentation.” IEEE Trans- actions on Image Processing (2024)
Jianzong Wu, Xiangtai Li, Xia Li, Henghui Ding, Yunhai Tong, Dacheng Tao. ”Towards robust referring image segmentation.” IEEE Trans- actions on Image Processing (2024)
2024
-
[16]
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, Tamara L. Berg. ”Mattnet: Modular attention network for referring expression comprehension.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2018
2018
-
[17]
”Vision-language transformer and query generation for referring segmentation.” Proceedings of the IEEE/CVF international conference on computer vision
Henghui Ding, Chang Liu, Suchen Wang, Xudong Jiang. ”Vision-language transformer and query generation for referring segmentation.” Proceedings of the IEEE/CVF international conference on computer vision. 2021
2021
-
[18]
”Encoder fusion network with co-attention embedding for referring image segmentation.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Guang Feng, Zhiwei Hu, Lihe Zhang, Huchuan Lu. ”Encoder fusion network with co-attention embedding for referring image segmentation.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021
2021
-
[19]
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, Philip H.S. Torr. ”Lavt: Language-aware vision transformer for referring image segmentation.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022
2022
-
[20]
”An image is worth 16x16 words: Transformers for image recogni- tion at scale.” arXiv preprint arXiv:2010.11929 (2020)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby. ”An image is worth 16x16 words: Transformers for image recogni- tion at ...
2020 arXiv
-
[21]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. ”Bert: Pre-training of deep bidirectional transformers for language un- derstanding.” Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human 14 langu...
2019
-
[22]
”Gres: Generalized referring expression segmentation.” Proceedings of the IEEE /CVF conference on computer vision and pattern recognition
Chang Liu, Henghui Ding, Xudong Jiang. ”Gres: Generalized referring expression segmentation.” Proceedings of the IEEE /CVF conference on computer vision and pattern recognition. 2023
2023
-
[23]
Shvets, Alexander Rakhlin, Alexandr A
Alexey A. Shvets, Alexander Rakhlin, Alexandr A. Kalinin, Vladimir I. Iglovikov. ”Automatic instrument segmentation in robot-assisted surgery using deep learning.” 2018 17th IEEE international conference on machine learning and applications (ICMLA). IEEE, 2018
2018
-
[24]
”2017 robotic instrument segmentation challenge.” arXiv preprint arXiv:1902.06426 (2019)
Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, Luis Herrera, Wenqi Li, Vladimir Iglovikov, Huoling Luo, Jian Yang, Danail Stoyanov, Lena Maier-Hein, Stefanie Speidel, Mahdi...
2019 arXiv
-
[25]
”2018 robotic scene segmentation challenge.” arXiv preprint arXiv:2001.11190 (2020)
Max Allan, Satoshi Kondo, Sebastian Bodenstedt, Stefan Leger, Rahim Kadkhodamohammadi, Imanol Luengo, Felix Fuentes, Evangello Flouty, Ahmed Mohammed, Marius Pedersen, Avinash Kori, Varghese Alex, Ganapathy Krishnamurthi, David Rauber, Robert Mendel, Christoph Palm, Sophia Ban...
2020 arXiv
-
[26]
”Swin transformer: Hierarchical vision transformer using shifted windows.” Proceedings of the IEEE/CVF international conference on computer vision
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo. ”Swin transformer: Hierarchical vision transformer using shifted windows.” Proceedings of the IEEE/CVF international conference on computer vision. 2021
2021
-
[27]
”Decoupled weight decay regularization.” arXiv preprint arXiv:1711.05101 (2017)
Ilya Loshchilov, Frank Hutter. ”Decoupled weight decay regularization.” arXiv preprint arXiv:1711.05101 (2017)
2017 arXiv
-
[28]
”Imagenet: A large-scale hierarchical image database.” 2009 IEEE conference on computer vision and pattern recognition
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei. ”Imagenet: A large-scale hierarchical image database.” 2009 IEEE conference on computer vision and pattern recognition. IEEE, 2009
2009
-
[29]
”Pytorch: An imperative style, high-performance deep learning library.” arXiv preprint arXiv:1912.01703 (2019)
Paszke, A. ”Pytorch: An imperative style, high-performance deep learning library.” arXiv preprint arXiv:1912.01703 (2019)
2019 arXiv
-
[30]
Yueming Jin, Keyun Cheng, Qi Dou, Pheng-Ann Heng . ”Incorporating temporal prior from motion flow for instrument segmentation in minimally invasive surgery video.” Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Con- ference, Shenzhen...
2019
-
[31]
Zixu Zhao, Yueming Jin, Xiaojie Gao, Qi Dou, Pheng-Ann Heng. ”Learning motion flows for semi-supervised instrument segmentation from robotic surgical video.” Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, Octo...
2020
-
[32]
”Feature pyramid networks for object detection.” Proceedings of the IEEE conference on computer vision and pattern recognition
Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie. ”Feature pyramid networks for object detection.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017
2017
-
[33]
”Deformable detr: Deformable transformers for end-to-end object detection.” arXiv preprint arXiv:2010.04159 (2020)
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai. ”Deformable detr: Deformable transformers for end-to-end object detection.” arXiv preprint arXiv:2010.04159 (2020)
2020 arXiv
-
[34]
Fuchs, K. H. ”Minimally invasive surgery.” Endoscopy 34.02 (2002): 154-159
2002
-
[35]
Harrell, and B
Andrew G. Harrell, and B. Todd Heniford. ”Minimally invasive abdominal surgery: lux et veritas past, present, and future.” The American journal of surgery 190.2 (2005): 239-243. 15
2005
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.