Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

With only a first-frame prompt, a surgical video segmentation model tracks instruments and tissues across unseen procedure types in real time, beating fine-tuned SAM2.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-03 21:05 UTC pith:D3RJXXV6

load-bearing objection A genuinely useful surgical iVOS benchmark plus a plausible model win, but the manuscript as posted has a naming inconsistency and an under-specified baseline-tuning protocol that need fixing before the numbers can be fully trusted. the 4 major comments →

arxiv 2511.16618 v2 pith:D3RJXXV6 submitted 2025-11-20 cs.CV eess.IVq-bio.TO

SurgSLOT: Segment Anything in Surgical Videos via Semantic Long-term Tracking

Haofeng Liu , Ziyue Wang , Sudhanshu Mishra , Mingqi Gao , Guanyi Qin , Chang Han Low , Alex Y. W. Kong , Zhu Zhuo
show 3 more authors
Huazhu Fu Joseph S. Ng Yueming Jin
This is my paper
classification cs.CV eess.IVq-bio.TO
keywords surgical video segmentationinteractive video object segmentationSAM2long-term trackingmasklet benchmarkzero-shot generalizationinstrument segmentationmemory management
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that interactive video object segmentation can be made to work in surgery as a true generalist: a single model, prompted once in the first frame, can track any instrument or tissue the user points to across procedures that last up to about half an hour, including procedure types never seen during training. To get there, the authors build SA-SV, the largest surgical masklet benchmark they know of — 61k frames, 1.6k temporally consistent masklets across eight procedure types — and augment SAM2 with three components: a diverse memory bank (DiveMem) that re-identifies targets after long disappearance; temporal semantic learning that teaches the model instrument categories through contrastive language supervision; and ambiguity-resilient learning that softens inconsistent tissue-boundary labels from multi-source datasets. On eight unseen test subsets, the model reports 80.42 average J&F with a 3-click prompt and 78.07 with a single click, at 68 FPS, exceeding fine-tuned SAM2 by 4.11 and 5.38 points. If the claim holds, the path to clinical deployment does not require retraining per hospital or per operation; the surgeon just clicks once and the model follows the target for the rest of the case.

Core claim

The paper's central discovery is that a video object segmentation model can be turned into a surgical generalist by fine-tuning on a large masklet-annotated benchmark spanning multiple procedure types (called SA-SV in the full text, iSurg in the abstract), then prompting once at the first frame. The proposed model (SAM2S in the full text, SurgSLOT in the abstract) adds three modules to SAM2: DiveMem, a diverse long-term memory that re-identifies the target after long disappearances; Temporal Semantic Learning, which aligns a learnable token with instrument category text features to give the target a stable semantic identity; and Ambiguity-Resilient Learning, which softens inconsistent tissue

What carries the argument

The argument rests on three mechanisms, all built into SAM2. DiveMem is the long-term tracking engine: during training it randomly samples frames across the whole video rather than only consecutive frames, so the model learns to bridge large temporal gaps; during inference it deposits high-confidence frames into a candidate buffer and then selects the frame whose image-encoder feature is least similar to the current long-term memory, keeping a diverse spatio-temporal record. Temporal Semantic Learning (TSL) gives the target a stable identity: a learnable classification token attends to memory features and current-frame features, then is pushed toward the text-encoder feature of the instrumen

Load-bearing premise

The load-bearing premise is that placing the second and third clicks at ground-truth error-region centers gives a fair comparison of 3-click interactive performance; if this protocol injects information only into the proposed method and not the baselines, the reported margins could shrink.

What would settle it

Re-run the 3-click evaluation with later clicks generated from the model's own predictions instead of from ground-truth error regions, or from a human user, and compare against fine-tuned SAM2. If the 4.11-point average J&F margin drops to near zero, the headline advantage is an artifact of the ground-truth-informed click placement.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A single SA-SV-trained model can be prompted at the first frame and track the target through roughly 30-minute procedures, including a procedure type (nephrectomy) and object categories never seen together in training.
  • Fine-tuning on SA-SV alone lifts vanilla SAM2 by a large margin on the benchmark subsets, suggesting the masklet dataset itself is a reusable resource for future surgical iVOS models.
  • Trainable diversity-based memory selection outperforms training-free long-term memory heuristics on long surgical videos, where targets disappear and reappear after camera motions.
  • The model maintains real-time inference (68 FPS at 512 resolution), making it usable intraoperatively rather than only offline.
  • With a single click the model still reaches 78.07 average J&F, showing the approach works with minimal user effort.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An obvious next experiment is to measure the 4.11-point margin under a protocol with no access to ground-truth error regions; if a surgeon's unguided clicks behave differently, the margin may change. A randomized user study would settle how much of the gain is from the model versus from the click-placement rule.
  • The language-aligned semantic token learned by TSL could be repurposed for referring-expression or text-query segmentation in surgery, enabling commands such as track the suction tip without a click, provided the text encoder is kept at inference time.
  • DiveMem's diversity criterion is computed in image-encoder space and is not specific to surgery; it could be transferred to other long-horizon tracking settings such as endoscopy or natural long videos, but the paper does not demonstrate that transfer.
  • The benchmark's zero-shot claim currently tests unseen procedure types and datasets from the same label taxonomy; a stronger test would add instruments or tissues introduced after deployment, which the current evaluation does not cover.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents SA-SV, claimed to be the largest surgical interactive video object segmentation (iVOS) benchmark, assembled and refined from 17 open-source datasets into 61k frames, 123k masks, and 1.6k masklets across eight procedure types. On top of this benchmark, it proposes SAM2S (also titled SurgSLOT), a SAM2 adaptation with three modules: DiveMem for diverse long-term memory, Temporal Semantic Learning (TSL) for instrument semantics via vision-language contrastive learning, and Ambiguity-Resilient Learning (ARL) for label softening against multi-source annotation noise. The paper reports that SAM2S achieves 80.42 average J&F under 3-click and 78.07 under 1-click settings at 68 FPS, outperforming vanilla and fine-tuned SAM2 by 17.10/24.31 and 4.11/5.38 points respectively, and shows zero-shot generalization to an unseen nephrectomy procedure type. Ablations attribute gains to each proposed module.

Significance. If the results hold, this is a substantial contribution to surgical iVOS: it provides a large multi-procedure masklet benchmark, a trainable memory/semantic adaptation of SAM2, real-time inference, and extensive cross-dataset evaluation including long videos. The planned release of code and dataset, the surgical-supervision-based masklet refinement, and the explicit 1-click/3-click evaluation are strengths. However, the central comparative claim currently rests on incompletely specified baseline fine-tuning, and the abstract claims results (SAM3 backbone, 81.0/82.8 J&F) that do not appear in the experimental section. These issues need to be resolved before the headline numbers can be fully credited.

major comments (4)
  1. [Abstract and §5.2] The abstract states: 'On the SAM2 and SAM3 backbones, SurgSLOT reaches 81.0 and 82.8 Macro Average J&F under cross-dataset evaluation, surpassing their fine-tuned counterparts by 5.1 and 5.3 points.' However, the full text only describes and evaluates SAM2S on a Hiera-B+ SAM2 backbone (§5.1), with best results 80.42 (3-click) and 78.07 (1-click). No SAM3 experiments, no 81.0/82.8 numbers, and no definition of 'Macro Average J&F' appear anywhere in the manuscript. In addition, the title uses 'SurgSLOT' while the text uses 'SAM2S'. This is an unsupported central claim and a naming inconsistency. Please add the SAM3 experiments or revise the abstract and title to match the presented evidence.
  2. [§5.1 (Model and Training Configuration), Tables 2–3 and Table 7] The training recipe is specified only for SAM2S: 30 epochs, LR 1e-5, mixed image–video training at 1:4 ratio, and 1:1 DiveMem/vanilla sampling. The baselines 'fine-tuned on SA-SV at 512 resolution' in Tables 2 and 3, and the SA-SV-only row in Table 7, are given no fine-tuning budget, learning rate, data sampling, loss weighting, or augmentation. Since the headline claim is that SAM2S beats fine-tuned SAM2 by 4.11 (3-click) and 5.38 (1-click) points, an undertrained or differently tuned comparator could produce this margin artifactually. Please provide the full fine-tuning configuration for every baseline, ideally with a matched training budget, or explicitly state that all baselines were trained with the same recipe as SAM2S excluding the proposed modules.
  3. [§5.1 (Prompting and Evaluation Protocol)] The 3-click protocol places the second and third clicks 'at error region centers', which requires ground-truth masks to compute. This conflicts with the 'prompt once, track throughout' claim unless all clicks are restricted to the first frame and the GT error map is used only to simulate user correction, not to guide tracking. Please clarify whether the error-region simulation is applied identically to every model within the first frame only, and confirm that GT is not used during propagation. Because different first-click behavior can lead to different GT-guided clicks, the 3-click comparison is not purely a test of autonomous long-term tracking; the 1-click results in Table 3 should be highlighted as the primary zero-shot/long-term evidence, and a sensitivity analysis of click placement would strengthen the paper.
  4. [Table 7] The ablation table is ambiguously formatted: rows are identified only by checkmarks, with no row labels or explicit statement of which module is added in each row. As written, row 5 appears to be SA-SV + DiveMem + TSL and row 6 all modules, but this is not stated in the text. In addition, no error bars or repeated runs are reported, and several per-dataset margins are very small (e.g., PolypGen 66.30 vs 66.29 in Table 2). This makes it difficult to assess whether the module-specific improvements (2.94, 3.14, 1.24 average J&F) are statistically meaningful. Please label the rows explicitly and add variance or significance information.
minor comments (4)
  1. [§5.2] The sentence 'This represents improvements of 24.31 and 17.10 points over vanilla SAM2, and 5.38 and 4.11 points over fine-tuned SAM2' mixes the 1-click and 3-click settings. State the pairing explicitly (e.g., 1-click: 24.31 over vanilla, 5.38 over fine-tuned; 3-click: 17.10 over vanilla, 4.11 over fine-tuned).
  2. [Table 1] The table lists EndoVis18 with combined mask/tissue counts, but the experiments split EV18 into EV18-I and EV18-T. Provide the split statistics so the reader can see the evaluation scale of each subset.
  3. [Table 5] The row labels 'SAM2 (SA-SV)' and 'SAM2S (SA-SV)' are confusing because both are fine-tuned on SA-SV; clarify that the first is the SAM2 baseline fine-tuned on SA-SV and the second is the full proposed model.
  4. [General] The paper does not include a limitations section. Given the benchmark construction effort, a brief discussion of annotation cost, remaining ambiguities, and potential failure cases would be useful for readers assessing clinical applicability.

Circularity Check

0 steps flagged

No circular derivation found: all claims are empirical benchmark and training results, not derived quantities.

full rationale

The paper makes no formal derivation whose outputs are equivalent to its inputs. SA-SV is a newly assembled benchmark from existing datasets, and SAM2S is an empirical method evaluated on held-out test subsets. The TSL module uses instrument semantic categories from the benchmark during training, but at inference the text encoder is omitted and the target is specified by a first-frame visual prompt, so the model does not predict the benchmark's category labels as its output; this is supervision, not circularity. The DiveMem filter selects memory frames based on the model's own IoU confidence and feature diversity, which is a control loop during inference rather than a fitted target. ARL applies uniform Gaussian label softening to ground-truth masks; it is a loss modification, not a redefinition of the evaluation metric. No uniqueness theorem or load-bearing self-citation is invoked: references to the authors' prior SurgicalSAM2 and ReSurgSAM2 appear only as baselines or related work, not as justification for the central claims. The reported gains over fine-tuned SAM2 could depend on an underspecified baseline fine-tuning protocol, but that is a question of experimental fairness and reproducibility, not of circular reasoning. Accordingly, the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

8 free parameters · 4 axioms · 0 invented entities

No new physical or mediating entities are introduced. The free parameters are all training/inference hyperparameters, and the axioms are domain assumptions about the benchmark, the prompt protocol, and semantic transfer. The benchmark itself is a contribution, not a postulated entity.

free parameters (8)
  • λ_arl = 20
    Weight for the ARL loss in Eq. 1; set without a reported hyperparameter search.
  • λ_tsl = 0.1
    Weight for the TSL loss in Eq. 1; hand-chosen.
  • σ = 1.0
    Gaussian kernel width for label softening in Eq. 4.
  • τ = 100
    Contrastive temperature in the TSL loss, Eq. 3, following OVSeg.
  • γ_iou = 0.95
    IoU confidence threshold for the DiveMem candidate buffer; manually set.
  • Δ = 5
    Candidate buffer size for DiveMem; manually set.
  • N_l = 4
    Long-term memory queue capacity; manually set.
  • DiveMem sampling schedule = 1:1 ratio of DiveMem vs vanilla sampling
    Training-time sampling strategy chosen by hand.
axioms (4)
  • domain assumption Evaluation clicks during the 'prompt once, track throughout' protocol are placed using ground-truth error regions (§5.1).
    The reported J&F numbers assume this protocol is a fair and valid operationalization of interactive prompting; if later clicks leak GT information disproportionately, the margins change.
  • domain assumption The manually corrected annotations in SA-SV represent correct object boundaries.
    §3 reports manual correction of over 20,000 masks in CholecSeg8k by a surgeon; the benchmark quality depends on this effort being accurate and unbiased, which is not independently verified.
  • domain assumption CLIP text embeddings provide useful semantic supervision for surgical instrument categories (§4.3).
    The TSL loss relies on the transfer of CLIP's natural-image language concepts to instrument names; no ablation shows that CLIP features are necessary versus a learned classifier.
  • domain assumption A model trained on seven procedure types transfers zero-shot to nephrectomy (EV17/EV18).
    The claim of zero-shot generalization across procedure types assumes that instrument appearance and tissue structure generalize, which is plausible but not guaranteed.

pith-pipeline@v1.3.0-alltime-deepseek · 14840 in / 11602 out tokens · 102029 ms · 2026-08-03T21:05:00.944341+00:00 · methodology

0 comments
read the original abstract

Surgical scene understanding demands temporally consistent tracking of instruments and tissues. For clinical use, such tracking should generalize to new centers and procedure types, yet retraining for each of them is costly and not scalable. Interactive video object segmentation offers a way toward this generalization: the target is specified at inference by a first-frame visual prompt, so a model can generalize to unseen categories and new scenarios without retraining. However, training such a generalizable model demands spatio-temporal masklet annotations at a scale and procedural diversity that existing surgical benchmarks lack. We fill this gap with iSurg, the largest surgical segmentation benchmark to our knowledge, spanning six procedure types with over 170k frames, 410k object masks, and 2.4k masklets, including an in-house clinical dataset of four 30-minute videos. Yet tracking over such long videos remains challenging: the target needs to be re-identified among visually similar objects after long absences, while long-term memory has to judge whether each stored frame genuinely depicts the target, both demanding a stable object-level semantic identity. To this end, we propose SurgSLOT, a surgical segmentation generalist that segments any prompted target through two coupled modules built on this semantic identity: Temporal Semantic Learning learns it for re-identification, and Semantic-driven Long-term Memory reuses it to select reliable memory frames, suppressing identity drift over long procedures. On the SAM2 and SAM3 backbones, SurgSLOT reaches 81.0 and 82.8 Macro Average J&F under cross-dataset evaluation, surpassing their fine-tuned counterparts by 5.1 and 5.3 points and transferring zero-shot to an unseen procedure type and unseen object categories, with the SAM2 version running in real time at 68 FPS.

Figures

Figures reproduced from arXiv: 2511.16618 by Alex Y. W. Kong, Chang Han Low, Guanyi Qin, Haofeng Liu, Huazhu Fu, Joseph S. Ng, Mingqi Gao, Sudhanshu Mishra, Yueming Jin, Zhu Zhuo, Ziyue Wang.

Figure 1
Figure 1. Figure 1: Overview of SA-SV benchmark and SAM2S framework. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of SAM2S for surgical video segmentation. DiveMem handles long-term tracking, TSL enhances semantic understand [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison between SAM2 (vanilla), SAM2 (FT), SAM2Long (FT), and SAM2S on RARP50. Frame indices indicate [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison on EndoVis18 (140s duration). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA

    cs.CV 2026-06 unverdicted novelty 6.0

    An RL framework uses digital twin representations with hierarchical uncertainty estimates and a novel clinical plausibility reward to train LLMs for surgical VideoQA, achieving SOTA on a new 2000-pair benchmark and tw...

Reference graph

Works this paper leans on

55 extracted references · 10 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Deep learning for surgical instrument recog- nition and segmentation in robotic-assisted surgeries: a sys- tematic review.Artificial Intelligence Review, 58(1):1, 2024

    Fatimaelzahraa Ali Ahmed, Mahmoud Yousef, Mariam Ali Ahmed, Hasan Omar Ali, Anns Mahboob, Hazrat Ali, Zubair Shah, Omar Aboumarzouk, Abdulla Al Ansari, and Shidin Balakrishnan. Deep learning for surgical instrument recog- nition and segmentation in robotic-assisted surgeries: a sys- tematic review.Artificial Intelligence Review, 58(1):1, 2024. 2

    2024

  2. [2]

    Cholecinstanceseg: A tool instance segmentation dataset for laparoscopic surgery.Scientific Data, 12(1):825, 2025

    Oluwatosin Alabi, Ko Ko Zayar Toe, Zijian Zhou, Charlie Budd, Nicholas Raison, Miaojing Shi, and Tom Vercauteren. Cholecinstanceseg: A tool instance segmentation dataset for laparoscopic surgery.Scientific Data, 12(1):825, 2025. 2, 3

    2025

  3. [3]

    Temporally constrained neural networks (tcnn): A framework for semi-supervised video semantic segmentation.arXiv preprint arXiv:2112.13815, 2021

    Deepak Alapatt, Pietro Mascagni, Armine Vardazaryan, Alain Garcia, Nariaki Okamoto, Didier Mutter, Jacques Marescaux, Guido Costamagna, Bernard Dallemagne, and Nicolas Padoy. Temporally constrained neural networks (tcnn): A framework for semi-supervised video semantic segmentation.arXiv preprint arXiv:2112.13815, 2021. 3

    Pith/arXiv arXiv 2021

  4. [4]

    A multi-centre polyp detection and segmentation dataset for generalisability assessment.Scientific Data, 10(1):75, 2023

    Sharib Ali, Debesh Jha, Noha Ghatwary, Stefano Realdon, Renato Cannizzaro, Osama E Salem, Dominique Lamarque, Christian Daul, Michael A Riegler, Kim V Anonsen, et al. A multi-centre polyp detection and segmentation dataset for generalisability assessment.Scientific Data, 10(1):75, 2023. 3

    2023

  5. [5]

    2017 robotic instrument segmentation challenge.arXiv preprint arXiv:1902.06426, 2019

    Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, et al. 2017 robotic instrument segmentation challenge.arXiv preprint arXiv:1902.06426, 2019. 2, 3

    Pith/arXiv arXiv 2017

  6. [6]

    2018 robotic scene segmentation challenge

    Max Allan, Satoshi Kondo, Sebastian Bodenstedt, Stefan Leger, Rahim Kadkhodamohammadi, Imanol Luengo, Fe- lix Fuentes, Evangello Flouty, Ahmed Mohammed, Marius Pedersen, et al. 2018 robotic scene segmentation challenge. arXiv preprint arXiv:2001.11190, 2020. 3

    Pith/arXiv arXiv 2018

  7. [7]

    Matis: Masked-attention transformers for surgical instrument segmentation

    Nicol ´as Ayobi, Alejandra P ´erez-Rond´on, Santiago Rodr´ıguez, and Pablo Arbel ´aes. Matis: Masked-attention transformers for surgical instrument segmentation. In2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pages 1–5, 2023. 3

    2023

  8. [8]

    Pixel-wise recognition for holistic surgical scene understanding.arXiv, 2024

    Nicol ´as Ayobi, Santiago Rodr´ıguez, Alejandra P´erez, Isabela Hern´andez, Nicol´as Aparicio, Eug´enie Dessevres, Sebasti´an Pe˜na, Jessica Santander, Juan Ignacio Caicedo, Nicol ´as Fern´andez, and Pablo Arbel ´aez. Pixel-wise recognition for holistic surgical scene understanding.arXiv, 2024. 3

    2024

  9. [9]

    Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs

    Jorge Bernal, F Javier S ´anchez, Gloria Fern´andez-Esparrach, Debora Gil, Cristina Rodr ´ıguez, and Fernando Vilari ˜no. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physi- cians.Computerized medical imaging and graphics, 43: 99–111, 2015. 3

    2015

  10. [10]

    The dresden surgical anatomy dataset for abdominal organ segmentation in surgi- cal data science.Scientific Data, 10(1):1–8, 2023

    Matthias Carstens, Franziska M Rinner, Sebastian Bodenst- edt, Alexander C Jenke, J ¨urgen Weitz, Marius Distler, Ste- fanie Speidel, and Fiona R Kolbinger. The dresden surgical anatomy dataset for abdominal organ segmentation in surgi- cal data science.Scientific Data, 10(1):1–8, 2023. 3

    2023

  11. [11]

    Xmem: Long- term video object segmentation with an atkinson-shiffrin memory model

    Ho Kei Cheng and Alexander G Schwing. Xmem: Long- term video object segmentation with an atkinson-shiffrin memory model. InEuropean conference on computer vision, pages 640–658. Springer, 2022. 2

    2022

  12. [12]

    Putting the object back into video object segmentation

    Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3151–3161, 2024. 2, 5

    2024

  13. [13]

    Mosev2: A more challenging dataset for video object segmentation in complex scenes.arXiv preprint arXiv:2508.05630, 2025

    Henghui Ding, Kaining Ying, Chang Liu, Shuting He, Xudong Jiang, Yu-Gang Jiang, Philip HS Torr, and Song Bai. Mosev2: A more challenging dataset for video object segmentation in complex scenes.arXiv preprint arXiv:2508.05630, 2025. 2, 3

    arXiv 2025

  14. [14]

    Sam2long: Enhancing sam 2 for long video segmentation with a training-free memory tree

    Shuangrui Ding, Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Yuwei Guo, Dahua Lin, and Jiaqi Wang. Sam2long: Enhancing sam 2 for long video segmentation with a training-free memory tree. InProceedings of the IEEE/CVF international conference on computer vision,

  15. [15]

    Patch-based adaptive weighting with segmenta- tion and scale (pawss) for visual tracking in surgical video

    Xiaofei Du, Maximilian Allan, Sebastian Bodenstedt, Lena Maier-Hein, Stefanie Speidel, Alessio Dore, and Danail Stoyanov. Patch-based adaptive weighting with segmenta- tion and scale (pawss) for visual tracking in surgical video. Medical image analysis, 57:120–135, 2019. 1

    2019

  16. [16]

    Spatio-temporal represen- tation decoupling and enhancement for federated instru- ment segmentation in surgical videos.arXiv preprint arXiv:2506.23759, 2025

    Zheng Fang, Xiaoming Qi, Chun-Mei Feng, Jialun Pei, Weixin Si, and Yueming Jin. Spatio-temporal represen- tation decoupling and enhancement for federated instru- ment segmentation in surgical videos.arXiv preprint arXiv:2506.23759, 2025. 3

    arXiv 2025

  17. [17]

    Deep learning for video object segmentation: a review.Artificial Intelligence Review, 56(1):457–531, 2023

    Mingqi Gao, Feng Zheng, James JQ Yu, Caifeng Shan, Guiguang Ding, and Jungong Han. Deep learning for video object segmentation: a review.Artificial Intelligence Review, 56(1):457–531, 2023. 2

    2023

  18. [18]

    Image compositing for segmentation of surgical tools without manual annotations.IEEE transactions on medical imaging, 40(5):1450–1460, 2021

    Luis C Garcia-Peraza-Herrera, Lucas Fidon, Claudia D’Ettorre, Danail Stoyanov, Tom Vercauteren, and Sebastien Ourselin. Image compositing for segmentation of surgical tools without manual annotations.IEEE transactions on medical imaging, 40(5):1450–1460, 2021. 3

    2021

  19. [19]

    Softseg: Advantages of soft versus binary training for image segmentation.Medical image analysis, 71:102038, 2021

    Charley Gros, Andreanne Lemay, and Julien Cohen-Adad. Softseg: Advantages of soft versus binary training for image segmentation.Medical image analysis, 71:102038, 2021. 5

    2021

  20. [20]

    Detection, segmentation, and 3d pose es- timation of surgical tools using convolutional neural net- works and algebraic geometry.Medical Image Analysis, 70: 101994, 2021

    Md Kamrul Hasan, Lilian Calvet, Navid Rabbani, and Adrien Bartoli. Detection, segmentation, and 3d pose es- timation of surgical tools using convolutional neural net- works and algebraic geometry.Medical Image Analysis, 70: 101994, 2021. 3

    2021

  21. [21]

    Interac- tive video object segmentation using global and local trans- fer modules

    Yuk Heo, Yeong Jun Koh, and Chang-Su Kim. Interac- tive video object segmentation using global and local trans- fer modules. InEuropean Conference on Computer Vision, pages 297–313. Springer, 2020. 2

    2020

  22. [22]

    Lvos: A benchmark for long-term video object segmentation

    Lingyi Hong, Wenchao Chen, Zhongying Liu, Wei Zhang, Pinxue Guo, Zhaoyu Chen, and Wenqiang Zhang. Lvos: A benchmark for long-term video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13480–13492, 2023. 2, 3

    2023

  23. [23]

    Lvos: A benchmark for large- scale long-term video object segmentation.IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2025

    Lingyi Hong, Zhongying Liu, Wenchao Chen, Chenzhi Tan, Yuang Feng, Xinyu Zhou, Pinxue Guo, Jinglun Li, Zhaoyu Chen, Shuyong Gao, et al. Lvos: A benchmark for large- scale long-term video object segmentation.IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2025. 3

    2025

  24. [24]

    Cholecseg8k: a semantic segmentation dataset for laparoscopic cholecystectomy based on cholec80.arXiv preprint arXiv:2012.12453, 2020

    W-Y Hong, C-L Kao, Y-H Kuo, J-R Wang, W-L Chang, and C-S Shih. Cholecseg8k: a semantic segmentation dataset for laparoscopic cholecystectomy based on cholec80.arXiv preprint arXiv:2012.12453, 2020. 2, 3

    Pith/arXiv arXiv 2012

  25. [25]

    Domain and content adaptive convolution based multi- source domain generalization for medical image segmenta- tion.IEEE Transactions on Medical Imaging, 42(1):233– 244, 2022

    Shishuai Hu, Zehui Liao, Jianpeng Zhang, and Yong Xia. Domain and content adaptive convolution based multi- source domain generalization for medical image segmenta- tion.IEEE Transactions on Medical Imaging, 42(1):233– 244, 2022. 2

    2022

  26. [26]

    Kvasir-seg: A segmented polyp dataset

    Debesh Jha, Pia H Smedsrud, Michael A Riegler, P ˚al Halvorsen, Thomas De Lange, Dag Johansen, and H˚avard D Johansen. Kvasir-seg: A segmented polyp dataset. InIn- ternational conference on multimedia modeling, pages 451–

  27. [27]

    Exploring intra-and inter-video relation for surgical semantic scene segmentation.IEEE Transactions on Medical Imaging, 41(11):2991–3002, 2022

    Yueming Jin, Yang Yu, Cheng Chen, Zixu Zhao, Pheng-Ann Heng, and Danail Stoyanov. Exploring intra-and inter-video relation for surgical semantic scene segmentation.IEEE Transactions on Medical Imaging, 41(11):2991–3002, 2022. 1

    2022

  28. [28]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InInternational Conference on Computer Vision, pages 4015–4026, 2023. 3

    2023

  29. [29]

    Ngoc Lan. P. et al. neounet: Towards accurate colon polyp segmentation and neoplasm detection, 15–28, 2021. 3

    2021

  30. [30]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 7061–7070, 2023. 5

    2023

  31. [31]

    Video object segmentation with adaptive feature bank and uncertain-region refinement.Advances in Neural Informa- tion Processing Systems, 33:3430–3441, 2020

    Yongqing Liang, Xin Li, Navid Jafari, and Jim Chen. Video object segmentation with adaptive feature bank and uncertain-region refinement.Advances in Neural Informa- tion Processing Systems, 33:3430–3441, 2020. 2

    2020

  32. [32]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. InPro- ceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 5

    2017

  33. [33]

    Surgical sam 2: Real-time segment anything in surgical video by efficient frame pruning

    Haofeng Liu, Erli Zhang, Junde Wu, Mingxuan Hong, and Yueming Jin. Surgical sam 2: Real-time segment anything in surgical video by efficient frame pruning. InAdvancements In Medical Foundation Models: Explainability, Robustness, Security, and Beyond, 2024. 2, 6

    2024

  34. [34]

    Resurgsam2: Referring segment anything in surgical video via credible long-term tracking.arXiv preprint arXiv:2505.08581, 2025

    Haofeng Liu, Mingqi Gao, Xuxiao Luo, Ziyue Wang, Guanyi Qin, Junde Wu, and Yueming Jin. Resurgsam2: Referring segment anything in surgical video via credible long-term tracking.arXiv preprint arXiv:2505.08581, 2025. 3

    Pith/arXiv arXiv 2025

  35. [35]

    Learning high-quality dynamic memory for video object segmentation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    Yong Liu, Ran Yu, Fei Yin, Xinyuan Zhao, Wei Zhao, Wei- hao Xia, Jiahao Wang, Yitong Wang, Yansong Tang, and Yu- jiu Yang. Learning high-quality dynamic memory for video object segmentation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. 2

    2025

  36. [36]

    Aditya Murali, Deepak Alapatt, Pietro Mascagni, Armine Vardazaryan, Alain Garcia, Nariaki Okamoto, Guido Costa- magna, Didier Mutter, Jacques Marescaux, Bernard Dalle- magne, et al. The endoscapes dataset for surgical scene segmentation, object detection, and critical view of safety assessment: Official splits and benchmark.arXiv preprint arXiv:2312.12429...

    Pith/arXiv arXiv 2023

  37. [37]

    Pyramid attention aggregation network for semantic segmentation of surgical instruments

    Zhen-Liang Ni, Gui-Bin Bian, Guan-An Wang, Xiao-Hu Zhou, Zeng-Guang Hou, Hua-Bin Chen, and Xiao-Liang Xie. Pyramid attention aggregation network for semantic segmentation of surgical instruments. InProceedings of the AAAI Conference on Artificial Intelligence, pages 11782– 11790, 2020. 1

    2020

  38. [38]

    Surginet: Pyramid attention aggregation and class-wise self- distillation for surgical instrument segmentation.Medical Image Analysis, 76:102310, 2022

    Zhen-Liang Ni, Xiao-Hu Zhou, Guan-An Wang, Wen- Qian Yue, Zhen Li, Gui-Bin Bian, and Zeng-Guang Hou. Surginet: Pyramid attention aggregation and class-wise self- distillation for surgical instrument segmentation.Medical Image Analysis, 76:102310, 2022. 2

    2022

  39. [39]

    Video object segmentation using space-time memory networks

    Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. InProceedings of the IEEE/CVF international conference on computer vision, pages 9226–9235, 2019. 2

    2019

  40. [40]

    Mvd-net: Semantic segmentation of cataract surgery using multi-view learning

    Mingyang Ou, Heng Li, Haofeng Liu, Xiaoxuan Wang, Chenlang Yi, Luoying Hao, Yan Hu, and Jiang Liu. Mvd-net: Semantic segmentation of cataract surgery using multi-view learning. InIEEE EMBC, pages 5035–5038, 2022. 1

    2022

  41. [41]

    Sar-rarp50: Segmentation of surgical instrumentation and action recognition on robot-assisted radical prostatectomy challenge.arXiv preprint arXiv:2401.00496, 2023

    Dimitrios Psychogyios, Emanuele Colleoni, Beatrice Van Amsterdam, Chih-Yang Li, Shu-Yu Huang, Yuchong Li, Fucang Jia, Baosheng Zou, Guotai Wang, Yang Liu, et al. Sar-rarp50: Segmentation of surgical instrumentation and action recognition on robot-assisted radical prostatectomy challenge.arXiv preprint arXiv:2401.00496, 2023. 3

    Pith/arXiv arXiv 2023

  42. [42]

    Structure matters: Revisiting boundary refinement in video object seg- mentation

    Guanyi Qin, Ziyue Wang, Daiyun Shen, Haofeng Liu, Han- tao Zhou, Junde Wu, Runze Hu, and Yueming Jin. Structure matters: Revisiting boundary refinement in video object seg- mentation. InProceedings of the IEEE/CVF International Conference on Computer Vision, 2025. 2

    2025

  43. [43]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, pages 8748–8763, 2021. 5

    2021

  44. [44]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024. 1, 2, 3, 5

    Pith/arXiv arXiv 2024

  45. [45]

    Towards real-time multiple surgical tool tracking.Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 9(3): 279–285, 2021

    Maria Robu, Abdolrahim Kadkhodamohammadi, Imanol Luengo, and Danail Stoyanov. Towards real-time multiple surgical tool tracking.Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 9(3): 279–285, 2021. 1

    2021

  46. [46]

    FA Davis, 2011

    Colleen J Rutherford.Differentiating surgical instruments. FA Davis, 2011. 3, 5

    2011

  47. [47]

    Towards holistic surgical scene understanding

    Natalia Valderrama, Paola Ruiz, Isabela Hern ´andez, Nicol´as Ayobi, Mathilde Verlyck, Jessica Santander, Juan Caicedo, Nicol´as Fern ´andez, and Pablo Arbel ´aez. Towards holistic surgical scene understanding. InMedical Image Computing and Computer Assisted Intervention – MICCAI 2022, pages 442–452, Cham, 2022. Springer Nature Switzerland. 3

    2022

  48. [48]

    A distractor-aware memory for visual object tracking with sam2

    Jovana Videnovic, Alan Lukezic, and Matej Kristan. A distractor-aware memory for visual object tracking with sam2. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 24255–24264, 2025. 2, 4, 5

    2025

  49. [49]

    Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy

    Ziyi Wang, Bo Lu, Yonghao Long, Fangxun Zhong, Tak- Hong Cheung, Qi Dou, and Yunhui Liu. Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy. InInternational Conference on Medical Image Computing and Computer- Assisted Intervention, pages 486–496. Springer, 2022. 2, 3

    2022

  50. [50]

    Scribbleprompt: fast and flexible interactive segmen- tation for any biomedical image

    Hallee E Wong, Marianne Rakic, John Guttag, and Adrian V Dalca. Scribbleprompt: fast and flexible interactive segmen- tation for any biomedical image. InEuropean Conference on Computer Vision, pages 207–229. Springer, 2024. 2

    2024

  51. [51]

    Samurai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory.arXiv preprint arXiv:2411.11922,

    Cheng-Yen Yang, Hsiang-Wei Huang, Wenhao Chai, Zhongyu Jiang, and Jenq-Neng Hwang. Samurai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory.arXiv preprint arXiv:2411.11922,

    Pith/arXiv arXiv

  52. [52]

    Decoupling features in hierar- chical propagation for video object segmentation.Advances in Neural Information Processing Systems, 35:36324–36336,

    Zongxin Yang and Yi Yang. Decoupling features in hierar- chical propagation for video object segmentation.Advances in Neural Information Processing Systems, 35:36324–36336,

  53. [53]

    Surgicalsam: Efficient class prompt- able surgical instrument segmentation

    Wenxi Yue, Jing Zhang, Kun Hu, Yong Xia, Jiebo Luo, and Zhiyong Wang. Surgicalsam: Efficient class prompt- able surgical instrument segmentation. InProceedings of the AAAI Conference on Artificial Intelligence, pages 6890– 6898, 2024. 1

    2024

  54. [54]

    Sur- gai3

    Sabrina Madad Zadeh, Tom Franc ¸ois, Aur ´elie Comptour, Michel Canis, Nicolas Bourdel, and Adrien Bartoli. Sur- gai3. 8k: a labeled dataset of gynecologic organs in la- paroscopy with application to automatic augmented reality surgical guidance.Journal of Minimally Invasive Gynecol- ogy, 30(5):397–405, 2023. 3

    2023

  55. [55]

    Medical sam 2: Seg- ment medical images as video via segment anything model 2.arXiv preprint arXiv:2408.00874, 2024

    Jiayuan Zhu, Yunli Qi, and Junde Wu. Medical sam 2: Seg- ment medical images as video via segment anything model 2.arXiv preprint arXiv:2408.00874, 2024. 6

    Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.