Pith. sign in

REVIEW 3 major objections 5 minor 46 references

Fusing semantic and spatial vision features produces surgical affordance maps that let a robot act autonomously on phantoms.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 19:35 UTC pith:CGQT72RQ

load-bearing objection First solid surgical affordance pipeline with real dVRK phantom closed-loop results; SAM pseudo-labels are the soft spot but do not sink the contribution. the 3 major comments →

arxiv 2607.04378 v1 pith:CGQT72RQ submitted 2026-07-05 cs.RO

SurgAM: Surgical Affordance Map Prediction with Multimodal Feature Fusion for Robot Autonomy

Lei Song , Yonghao Long , Mengya Xu , Jiayi Geng , Xiuyuan Chen , Qi Dou This is my paper
classification cs.RO
keywords surgical affordancerobot autonomyfeature fusionDINOv2Stable Diffusionhierarchical prompt learningda Vinci Research Kitvisual servoing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Surgical robots still struggle to turn what they see into where they should act. This paper claims that the missing link is an affordance map: a spatial probability field that marks the safe, functional tissue regions for three elementary actions (aspiration, clipping, retraction). The authors build those maps by adaptively combining the fine semantic features of a self-supervised vision transformer with the smooth spatial layout features of a generative diffusion model, guided by hierarchical surgical-context prompts and a scene-focused attention decoder. On a new multi-source dataset of nearly two thousand annotated sequences the method outperforms prior affordance baselines. When the same maps are fed to a visual-servoing controller on a physical surgical robot, the system completes lung retraction and vessel clipping on realistic phantoms with near-perfect success rates. If the approach generalizes, robots gain an explicit, interpretable bridge from camera images to safe manipulation targets without needing end-to-end action imitation or rigid pre-operative plans.

Core claim

An adaptive fusion of DINOv2 semantic features and Stable Diffusion spatial features, conditioned by hierarchical surgical prompts and decoded with scene-guided attention, yields surgical affordance maps that both score highest on a new three-action dataset and successfully drive autonomous retraction and clipping on physical phantoms.

What carries the argument

Adaptive feature fusion: a learnable scalar beta mixes multi-layer DINOv2 semantic features with multi-scale Stable Diffusion spatial features; the fused map is then refined by hierarchical text prompts and a global-scene-guided cross-attention decoder to produce the final affordance heatmap.

Load-bearing premise

That expanding a single pre-contact tool-tissue point with a general segmentation model yields a faithful ground-truth region of safe and functional affordance for the intended surgical action.

What would settle it

Collect expert-annotated safe/functional tissue masks independent of the SAM pipeline; if the model trained on the original SAM-derived labels shows a large drop in KLD/SIM/NSS or fails the same phantom tasks when evaluated against the new labels, the ground-truth assumption is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces surgical affordance map prediction as a bridge from visual scene understanding to autonomous action planning for three fundamental actions (aspiration, clipping, retraction). It proposes an adaptive multimodal fusion framework that combines multi-layer semantic features from a frozen DINOv2 encoder with multi-scale spatial features from a Stable Diffusion U-Net (Eqs. 1–3), hierarchical prompt learning that extends CoOp with surgical context and action tokens (Eqs. 4–5), and a scene-guided cross-modal attention decoder that uses the global CLS token to mask background (Eqs. 6–8). A new dataset of 1,915 sequences is constructed by expanding single pre-contact points via SAM into heatmaps; quantitative results claim SOTA over five recent baselines (Table I: KLD 1.362, SIM 0.367, NSS 1.642, CLA 0.895), progressive ablations isolate each module (Tables II–III), and the maps are shown to drive autonomous lung retraction (100 % of 50 trials) and prostate vessel clipping (98 % of 50 trials) on a dVRK platform with silicone phantoms.

Significance. If the results hold, this is a genuine first step toward interpretable, action-centric perception for surgical robots, moving beyond pure segmentation or end-to-end imitation. The dual-foundation fusion insight, hierarchical surgical prompting, and closed-loop phantom validation on a real dVRK platform are concrete engineering contributions that other groups can build upon. The public-dataset-derived affordance annotations and the explicit success/failure reporting on physical hardware strengthen reproducibility and practical relevance for the surgical robotics community.

major comments (3)
  1. §IV-A (Dataset construction): The entire training signal and all reported metrics rest on the assumption that a single human-annotated pre-contact point expanded by SAM yields a faithful ground-truth affordance region. No inter-annotator agreement, no surgeon-rated mask quality scores, and no sensitivity analysis of the SAM step are provided. If the masks systematically include non-actionable tissue or exclude the true functional zone, both the ranking in Table I and the claim that the maps “successfully enable autonomous surgical actions” become misaligned with clinical affordance. This is load-bearing and must be addressed (e.g., by multi-surgeon validation of a subset of masks or by reporting agreement metrics).
  2. Tables I–III and §IV-C/D: All quantitative claims (KLD, SIM, NSS, CLA) are reported as point estimates with no standard deviations, confidence intervals, or statistical significance tests across folds or random seeds. Given the class imbalance (904 retraction / 780 aspiration / 231 clipping) and the modest absolute gains over WorldAfford and OOAL, it is impossible to judge whether the improvements are robust. Error bars or a paired statistical test are required for the SOTA claim to be credible.
  3. §IV-E (Phantom experiments): The 100 % / 98 % success rates are encouraging, yet the silicone lung and prostate phantoms lack the continuous deformation, bleeding, specular highlights and lighting changes of real tissue. The single reported failure mode (temporary occlusion) does not stress-test the SAM-derived label fidelity under realistic conditions. A clearer discussion of this domain gap, or at least a qualitative comparison of predicted maps on real endoscopic frames versus phantom frames, is needed to support the “practical applicability” claim.
minor comments (5)
  1. Fig. 3 qualitative panels would benefit from a consistent color scale and an explicit indication of the ground-truth centroid so that over-activation versus under-activation can be judged at a glance.
  2. Eq. (3): the learnable scalar β is introduced without stating its final learned value or its variance across runs; reporting this would help readers understand the relative contribution of the two foundation models.
  3. The CLA metric is newly proposed; a short formal definition and a comparison against raw ACD on a fixed-resolution subset would improve clarity.
  4. Several baseline citations (Cross-View-AG, WorldAfford, OOAL) are recent; ensure the exact training protocol (same data split, same number of epochs) is stated so that the comparison remains fair.
  5. Minor typographical issues: “a f f” subscript formatting in Eq. (8), and occasional missing spaces after citations.

Circularity Check

1 steps flagged

No significant circularity: empirical SOTA metrics and phantom success rates are measured against held-out annotations and physical trials, not forced by construction or self-defined inputs.

specific steps
  1. self citation load bearing [Section IV-E (Deployment and Validation)]
    "we integrated our affordance prediction module with a visual servoing controller based on our VPPV framework [46] and deployed the overall framework on a physical surgical robot platform"

    Authors cite their own prior VPPV system for the controller that executes the maps. This is ordinary engineering reuse and is not load-bearing for the central claim (map quality or SOTA metrics); the maps are independently trained and scored. Mild self-reference only.

full rationale

The paper is a standard supervised ML contribution. Affordance maps are trained on a newly annotated dataset (pre-contact points expanded by SAM into heatmaps) and evaluated with independent metrics (KLD/SIM/NSS/CLA) on held-out frames plus physical dVRK trials. No equation reduces a claimed prediction to a fitted constant by construction; the adaptive fusion (Eq. 3), hierarchical prompts (Eq. 4), and scene-guided attention (Eqs. 6-8) are architectural choices whose value is shown by ablation, not tautological. The sole self-citation ([46] VPPV controller) is used only for the downstream servoing loop and does not define or force the affordance maps themselves. Ground-truth construction via SAM is a potential label-quality risk, not circularity. Score 1 reflects only the minor non-load-bearing self-reference.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 2 invented entities

The central empirical claim rests on (1) the assumption that SAM masks seeded by pre-contact points are valid affordance ground truth, (2) the complementary-feature hypothesis taken from prior vision work, and (3) a handful of learnable scalars and prompt tokens that are optimized on the new dataset. No new physical entities are postulated; the ‘surgical affordance map’ is a task definition rather than a novel ontological object.

free parameters (3)
  • β (adaptive fusion weight) = initialized 0.5, learned
    Learnable scalar initialized at 0.5 that balances DINOv2 vs. Stable Diffusion features; optimized end-to-end on the surgical dataset.
  • α_i (layer aggregation weights)
    Learnable coefficients that sum to 1 for the last j=4 DINOv2 layers; fitted during training.
  • learnable prompt vectors {v_1 … v_p}
    Context Optimization-style continuous tokens prepended to surgical context and action tokens; optimized on the training set.
axioms (3)
  • domain assumption Features extracted from a frozen DINOv2 ViT and from intermediate U-Net layers of a frozen Stable Diffusion model are complementary (semantic precision vs. spatial coherence) for dense prediction tasks.
    Invoked in §III-B and justified by citation to Zhang et al. (2023); treated as given rather than re-proven.
  • ad hoc to paper A single human-annotated pre-contact point expanded by SAM yields a valid ground-truth affordance region for aspiration, clipping or retraction.
    Core of the dataset construction pipeline in §IV-A; no independent clinical validation of the resulting masks is provided.
  • ad hoc to paper Global CLS token from the fused visual features supplies a reliable attention mask that suppresses non-surgical background.
    Used to form the scene-guided mask M in Eq. (6); assumed without ablation against alternative global descriptors.
invented entities (2)
  • surgical affordance map (for aspiration / clipping / retraction) no independent evidence
    purpose: Defines the prediction target that bridges scene understanding and robot action planning.
    Task definition introduced in the abstract and §I; independent evidence is limited to the authors’ own phantom trials.
  • hierarchical surgical prompt (context + action tokens) no independent evidence
    purpose: Injects procedural context into the CLIP text encoder so the same action word can mean different spatial targets.
    Extension of CoOp introduced in §III-C; no external validation that the learned vectors transfer across institutions.

pith-pipeline@v1.1.0-grok45 · 17028 in / 3028 out tokens · 37566 ms · 2026-07-11T19:35:57.406620+00:00 · methodology

0 comments
read the original abstract

Surgical automation is being increasingly studied, yet bridging visual scene understanding with autonomous action planning remains a fundamental challenge. While much research effort has been made on scene perception (e.g., tool recognition and scene segmentation), understanding and predicting actionable possibilities for surgical automation is still underexplored. In this paper, we introduce surgical affordance prediction, which identifies actionable regions for fundamental surgical actions from visual data. Specifically, a novel adaptive feature fusion framework is proposed that leverages the complementary strengths of a self-supervised vision transformer encoder for its superior semantic understanding and a large-scale generative model encoder for its spatially-aware capability. Furthermore, we introduce a hierarchical prompt learning mechanism to adapt to varying procedural contexts. Finally, a scene-guided attention decoder is proposed to focus on critical surgical areas while suppressing background distractions. To validate the effectiveness, we established a new dataset, derived from publicly available surgical datasets with affordance annotations for three basic surgical actions: aspiration, clipping, and retraction. Extensive experiments demonstrate that our approach achieves state-of-the-art performance. Moreover, we validate our framework's applicability for downstream automation on a realistic lung and prostate phantom, and results show that the predicted affordance maps successfully enable autonomous surgical actions.

Figures

Figures reproduced from arXiv: 2607.04378 by Jiayi Geng, Lei Song, Mengya Xu, Qi Dou, Xiuyuan Chen, Yonghao Long.

Figure 1
Figure 1. Figure 1: Overall concept for surgical scene affordance map prediction: [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Framework of surgical scene affordance map prediction. The architecture features two parallel streams on the left for multimodal feature [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison of affordance prediction methods across different surgical tasks (aspiration, clipping, retraction) and datasets showing [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Experimental setup for autonomous task validation using dVRK. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 6 linked inside Pith

  1. [1]

    Will your next surgeon be a robot? autonomy and ai in robotic surgery,

    S. Schmidgall, J. D. Opfermann, J. W. Kim, and A. Krieger, “Will your next surgeon be a robot? autonomy and ai in robotic surgery,” Science Robotics, vol. 10, no. 104, p. eadt0187, 2025

    2025

  2. [2]

    Concepts and trends in autonomy for robot-assisted surgery,

    P. Fiorini, K. Y . Goldberg, Y . Liu, and R. H. Taylor, “Concepts and trends in autonomy for robot-assisted surgery,”Proceedings of the IEEE, vol. 110, no. 7, pp. 993–1011, 2022

    2022

  3. [3]

    Autonomy in surgical robotics,

    A. Attanasio, B. Scaglioni, E. De Momi, P. Fiorini, and P. Valdastri, “Autonomy in surgical robotics,”Annual Review of Control, Robotics, and Autonomous Systems, vol. 4, no. 1, pp. 651–679, 2021

    2021

  4. [4]

    EndoNet: a deep architecture for recognition tasks on laparoscopic videos,

    A. P. Twinanda, S. Shehata, D. Mutter, J. Marescaux, E. De Momi, and N. Padoy, “EndoNet: a deep architecture for recognition tasks on laparoscopic videos,”IEEE Transactions on Medical Imaging, vol. 36, no. 1, pp. 86–97, 2017

    2017

  5. [5]

    Pixel- wise recognition for holistic surgical scene understanding,

    N. Ayobi, S. Rodr ´ıguez, A. P´erez, I. Hern ´andez, N. Aparicio, E. Des- sevres, S. Pe˜na, J. Santander, J. I. Caicedo, N. Fern´andez,et al., “Pixel- wise recognition for holistic surgical scene understanding,”Medical Image Analysis, p. 103726, 2025

    2025

  6. [6]

    Medical robotics—regulatory, ethical, and legal considerations for increasing levels of autonomy,

    G.-Z. Yang, J. Cambias, K. Cleary, E. Daimler, J. Drake, P. E. Dupont, N. Hata, P. Kazanzides, S. Martel, R. V . Patel,et al., “Medical robotics—regulatory, ethical, and legal considerations for increasing levels of autonomy,” p. eaam8638, 2017

    2017

  7. [7]

    Surgical robot transformer (srt): Imitation learning for surgical tasks,

    J. W. Kim, T. Z. Zhao, S. Schmidgall, A. Deguet, M. Kobilarov, C. Finn, and A. Krieger, “Surgical robot transformer (srt): Imitation learning for surgical tasks,”arXiv preprint arXiv:2407.12998, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Sufia-bc: Generating high quality demonstration data for visuomotor policy learning in surgical subtasks,

    M. Moghani, N. Nelson, M. Ghanem, A. Diaz-Pinto, K. Hari, M. Az- izian, K. Goldberg, S. Huver, and A. Garg, “Sufia-bc: Generating high quality demonstration data for visuomotor policy learning in surgical subtasks,” in2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 4534–4541

    2025

  9. [9]

    Clinical appli- cations of artificial intelligence in robotic surgery,

    J. E. Knudsen, U. Ghaffar, R. Ma, and A. J. Hung, “Clinical appli- cations of artificial intelligence in robotic surgery,”Journal of robotic surgery, vol. 18, no. 1, p. 102, 2024

    2024

  10. [10]

    J. J. Gibson,The ecological approach to visual perception: classic edition. Psychology press, 2014

    2014

  11. [11]

    Af- fordances in robotic tasks–a survey,

    P. Ard ´on, `E. Pairet, K. S. Lohan, S. Ramamoorthy, and R. Petrick, “Af- fordances in robotic tasks–a survey,”arXiv preprint arXiv:2004.07400, 2020

    Pith/arXiv arXiv 2004

  12. [12]

    Robotic grasp detection using structure prior attention and multiscale features,

    L. Chen, M. Niu, J. Yang, Y . Qian, Z. Li, K. Wang, T. Yan, and P. Huang, “Robotic grasp detection using structure prior attention and multiscale features,”IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2024

    2024

  13. [13]

    Position- aware pushing and grasping synergy with deep reinforcement learning in clutter,

    M. Zhao, G. Zuo, S. Yu, D. Gong, Z. Wang, and O. Sie, “Position- aware pushing and grasping synergy with deep reinforcement learning in clutter,”CAAI Transactions on Intelligence Technology, vol. 9, no. 3, pp. 738–755, 2024

    2024

  14. [14]

    Au- tomatic instrument segmentation in robot-assisted surgery using deep learning,

    A. A. Shvets, A. Rakhlin, A. A. Kalinin, and V . I. Iglovikov, “Au- tomatic instrument segmentation in robot-assisted surgery using deep learning,” in2018 17th IEEE international conference on machine learning and applications (ICMLA). IEEE, 2018, pp. 624–628

    2018

  15. [15]

    Automated laparoscopic colorectal surgery workflow recognition using artificial intelligence: experimental research,

    D. Kitaguchi, N. Takeshita, H. Matsuzaki, T. Oda, M. Watanabe, K. Mori, E. Kobayashi, and M. Ito, “Automated laparoscopic colorectal surgery workflow recognition using artificial intelligence: experimental research,”International journal of surgery, vol. 79, pp. 88–94, 2020

    2020

  16. [16]

    Tecno: Surgical phase recognition with multi- stage temporal convolutional networks,

    T. Czempiel, M. Paschali, M. Keicher, W. Simson, H. Feussner, S. T. Kim, and N. Navab, “Tecno: Surgical phase recognition with multi- stage temporal convolutional networks,” inMedical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd Interna- tional Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III 23. Springer, 2020, pp. 343–352

    2020

  17. [17]

    Automating multi-throw multilateral surgical suturing with a mechan- ical needle guide and sequential convex optimization,

    S. Sen, A. Garg, D. V . Gealy, S. McKinley, Y . Jen, and K. Goldberg, “Automating multi-throw multilateral surgical suturing with a mechan- ical needle guide and sequential convex optimization,” in2016 IEEE international conference on robotics and automation (ICRA). IEEE, 2016, pp. 4178–4185

    2016

  18. [18]

    Autonomy for surgical robots: Concepts and paradigms,

    T. Haidegger, “Autonomy for surgical robots: Concepts and paradigms,”IEEE Transactions on Medical Robotics and Bionics, vol. 1, no. 2, pp. 65–76, 2019

    2019

  19. [19]

    Supervised autonomous robotic soft tissue surgery,

    A. Shademan, R. S. Decker, J. D. Opfermann, S. Leonard, A. Krieger, and P. C. Kim, “Supervised autonomous robotic soft tissue surgery,” Science translational medicine, vol. 8, no. 337, pp. 337ra64–337ra64, 2016

    2016

  20. [20]

    Srt-h: A hierarchical framework for autonomous surgery via language-conditioned imitation learning,

    J. W. Kim, J.-T. Chen, P. Hansen, L. X. Shi, A. Goldenberg, S. Schmidgall, P. M. Scheikl, A. Deguet, B. M. White, D. R. Tsai, et al., “Srt-h: A hierarchical framework for autonomous surgery via language-conditioned imitation learning,”Science robotics, vol. 10, no. 104, p. eadt5254, 2025

    2025

  21. [21]

    Tracking everything in robotic-assisted surgery,

    B. Zhan, W. Zhao, Y . Fang, B. Du, F. Vasconcelos, D. Stoyanov, D. S. Elson, and B. Huang, “Tracking everything in robotic-assisted surgery,” in2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 1–7

    2025

  22. [22]

    Affordancenet: An end-to-end deep learning approach for object affordance detection,

    T.-T. Do, A. Nguyen, and I. Reid, “Affordancenet: An end-to-end deep learning approach for object affordance detection,” in2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018, pp. 5882–5889

    2018

  23. [23]

    Affor- dancellm: Grounding affordance from vision language models,

    S. Qian, W. Chen, M. Bai, X. Zhou, Z. Tu, and L. E. Li, “Affor- dancellm: Grounding affordance from vision language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 7587–7597

    2024

  24. [24]

    Actionclip: Adapting language-image pretrained models for video action recognition,

    M. Wang, J. Xing, J. Mei, Y . Liu, and Y . Jiang, “Actionclip: Adapting language-image pretrained models for video action recognition,”IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  25. [25]

    On the opportunities and risks of foundation models,

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill,et al., “On the opportunities and risks of foundation models,”arXiv preprint arXiv:2108.07258, 2021

    Pith/arXiv arXiv 2021

  26. [26]

    A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence,

    J. Zhang, C. Herrmann, J. Hur, L. Polania Cabrera, V . Jampani, D. Sun, and M.-H. Yang, “A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence,”Advances in Neural Information Processing Systems, vol. 36, pp. 45 533–45 547, 2023

    2023

  27. [27]

    High-resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695

    2022

  28. [28]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khali- dov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby,et al., “Dinov2: Learning robust visual features without supervision,”arXiv preprint arXiv:2304.07193, 2023

    Pith/arXiv arXiv 2023

  29. [29]

    A review of deep learning-based information fusion techniques for multimodal medical image classification,

    Y . Li, M. E. H. Daho, P.-H. Conze, R. Zeghlache, H. Le Boit ´e, R. Tadayoni, B. Cochener, M. Lamard, and G. Quellec, “A review of deep learning-based information fusion techniques for multimodal medical image classification,”Computers in Biology and Medicine, vol. 177, p. 108635, 2024

    2024

  30. [30]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,et al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  31. [31]

    Learning to prompt for vision-language models,

    K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models,”International Journal of Computer Vision, vol. 130, no. 9, pp. 2337–2348, 2022

    2022

  32. [32]

    One-shot open affordance learning with foundation models,

    G. Li, D. Sun, L. Sevilla-Lara, and V . Jampani, “One-shot open affordance learning with foundation models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 3086–3096

    2024

  33. [33]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo,et al., “Segment anything,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 4015–4026

    2023

  34. [34]

    Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy,

    Z. Wang, B. Lu, Y . Long, F. Zhong, T.-H. Cheung, Q. Dou, and Y . Liu, “Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2022, pp. 486–496

    2022

  35. [35]

    Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos,

    C. I. Nwoye, T. Yu, C. Gonzalez, B. Seeliger, P. Mascagni, D. Mutter, J. Marescaux, and N. Padoy, “Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos,” Medical Image Analysis, vol. 78, p. 102433, 2022

    2022

  36. [36]

    Heidelberg colorectal data set for surgical data science in the sensor operating room,

    L. Maier-Hein, M. Wagner, T. Ross, A. Reinke, S. Bodenstedt, P. M. Full, H. Hempe, D. Mindroc-Filimon, P. Scholz, T. N. Tran,et al., “Heidelberg colorectal data set for surgical data science in the sensor operating room,”Scientific data, vol. 8, no. 1, p. 101, 2021

    2021

  37. [37]

    Challenges in multi-centric generalization: phase and step recognition in roux-en-y gastric bypass surgery,

    J. L. Lavanchy, S. Ramesh, D. Dall’Alba, C. Gonzalez, P. Fiorini, B. P. M¨uller-Stich, P. C. Nett, J. Marescaux, D. Mutter, and N. Padoy, “Challenges in multi-centric generalization: phase and step recognition in roux-en-y gastric bypass surgery,”International journal of computer assisted radiology and surgery, vol. 19, no. 11, pp. 2249–2257, 2024

    2024

  38. [38]

    Video retrieval in laparoscopic video recordings with dynamic content descriptors,

    K. Schoeffmann, H. Husslein, S. Kletz, S. Petscharnig, B. Muenzer, and C. Beecks, “Video retrieval in laparoscopic video recordings with dynamic content descriptors,”Multimedia Tools and Applications, vol. 77, no. 13, pp. 16 813–16 832, 2018

    2018

  39. [39]

    2018 robotic scene segmentation challenge,

    M. Allan, S. Kondo, S. Bodenstedt, S. Leger, R. Kadkhodamoham- madi, I. Luengo, F. Fuentes, E. Flouty, A. Mohammed, M. Pedersen, et al., “2018 robotic scene segmentation challenge,”arXiv preprint arXiv:2001.11190, 2020

    Pith/arXiv arXiv 2018

  40. [40]

    The saras endoscopic surgeon action detection (esad) dataset: Challenges and methods,

    V . S. Bawa, G. Singh, F. KapingA, I. Skarga-Bandurova, E. Oleari, A. Leporini, C. Landolfo, P. Zhao, X. Xiang, G. Luo,et al., “The saras endoscopic surgeon action detection (esad) dataset: Challenges and methods,”arXiv preprint arXiv:2104.03178, 2021

    Pith/arXiv arXiv 2021

  41. [41]

    Locate: Localize and transfer object parts for weakly supervised affordance grounding,

    G. Li, V . Jampani, D. Sun, and L. Sevilla-Lara, “Locate: Localize and transfer object parts for weakly supervised affordance grounding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10 922–10 931

    2023

  42. [42]

    Understanding 3d object interaction from a single image,

    S. Qian and D. F. Fouhey, “Understanding 3d object interaction from a single image,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 21 753–21 763

    2023

  43. [43]

    Learning affordance grounding from exocentric images,

    H. Luo, W. Zhai, J. Zhang, Y . Cao, and D. Tao, “Learning affordance grounding from exocentric images,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 2252–2261

    2022

  44. [44]

    Grounded affordance from exocentric view,

    ——, “Grounded affordance from exocentric view,”International Journal of Computer Vision, vol. 132, no. 6, pp. 1945–1969, 2024

    1945

  45. [45]

    Worldafford: Affordance grounding based on natural language instructions,

    C. Chen, Y . Cong, and Z. Kan, “Worldafford: Affordance grounding based on natural language instructions,” in2024 IEEE 36th Interna- tional Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 2024, pp. 822–828

    2024

  46. [46]

    Surgical embodied intelligence for generalized task autonomy in laparoscopic robot-assisted surgery,

    Y . Long, A. Lin, D. H. C. Kwok, L. Zhang, Z. Yang, K. Shi, L. Song, J. Fu, H. Lin, W. Wei,et al., “Surgical embodied intelligence for generalized task autonomy in laparoscopic robot-assisted surgery,” Science Robotics, vol. 10, no. 104, p. eadt3093, 2025

    2025