Computer Vision and Pattern Recognition · arXiv:2611.11437 v1 · submitted 8 Nov 2026 · 84 references
How to "train" a surgeon? Benchmarking generalist agents in surgical scene understanding
The paper says
“Evaluating general-purpose multimodal large language models (MLLMs) for surgical scene understanding remains limited due to scarce surgical data and fragmented evaluation protocols. We address these challenges through a two-stage evaluation framework for surgical scene understanding.”
more of the abstract
In Stage I, we introduce SurgKnowBench, a diagnostic benchmark that harmonizes public surgical datasets across procedures and tasks and evaluates proprietary and open-source MLLMs to map the boundary of pretrained surgical competence. SurgKnowBench reveals that current models retain partial visual priors but remain weak on relational and procedural understanding and workflow assessment. Building on this failure frontier and motivated by the scaffolded nature of surgical training, we introduce SurgAgentGym, an agentic environment where general-purpose MLLM agents can use bounded knowledge search over SurgWiki and general-purpose visual grounding for surgical tasks. SurgAgentGym evaluates whether generalist agents can improve surgical understanding at inference time through tool-mediated evidence use, without additional surgical training. Our results show that agentic inference narrows the gap mainly for perception-heavy tasks, while fine-grained localization and expert-style assessment remain persistent bottlenecks. This opens a path toward future studies of agentic surgical understanding.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 10 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. A thorough and honest benchmark for surgical MLLM evaluation; the agentic stage is less conclusive due to tool failures but the diagnosis of current model limitations is solid. This line is wrong
3 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 84 references say.
The paper says, at Table 3, Panel B (matched tool comparison); §4.2
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The matched tool comparison includes only 5 of the 16 backbones (InternVL2.5-8B, Qwen2.5-VL-7B-Instruct, Qwen3-VL-8B-Instruct, Gemma-3-27B-it, Lingshu-32B) with no stated selection criteria. These five do not, for example, include the best-performing model (Qwen3-VL-32B-Instruct) or any proprietary model. The claim that 'tool access benefits perception-heavy tasks' would be stronger if the matched comparison covered a broader, more representative set of backbones or if the selection rationale (e.g., spanning scale tiers and medical specialization) were stated.
more of objection 1
As it stands, the generalizability of the matched-comparison results to the full model set is unclear.
The paper says, at §4.2 / Table 3, Panel A; Conclusion and Abstract
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The across-stage comparison (Panel A) contrasts Stage I direct answers with Stage II agent answers, but the two stages differ not only in tool availability but also in prompts, system instructions, and potentially reasoning mode. The paper acknowledges this by including the matched comparison (Panel B), but the main narrative (Abstract: 'agentic inference narrows the gap mainly for perception-heavy tasks'; Conclusion: 'Agentic inference improves some tasks') primarily references the across-stage changes.
more of objection 2
To avoid conflating confounds, the paper should more explicitly state which claim relies on which comparison and note that the Panel A deltas are a composite of tool access, prompt engineering, and reasoning-mode changes, whereas the pure tool contribution is isolated in Panel B.
The paper says, at Appendix D.3.2 / §3.1 (SurgWiki construction)
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The knowledge base includes case records from the same datasets used for evaluation. The paper states that only training-side annotations are included and that evaluation labels are excluded from SurgWiki. This separation is critical to avoid label leakage. The paper should explicitly confirm that no episode in the SurgKnowBench evaluation set shares a video identifier or frame timestamp with a …
more of objection 3
… case-record chunk in SurgWiki, and that the procedure for excluding evaluation labels has been verified (e.g., by checking that the hard-set episodes' ground-truth labels do not appear in any case-record text). A brief description of this verification would resolve a potential concern about information leakage.
6 minor objections
The paper says, at Fig. 2 (line plots)
Pith says: minor objection 1 of 6
The figure panels, particularly the 'Skill Assessment' subplot, overlay multiple model lines that are hard to distinguish in grayscale or print. Consider adding distinct marker shapes or a more detailed legend that maps colors to model names, as the current line-only encoding is difficult to read.
The paper says, at §2.3, paragraph on CVS assessment
Pith says: minor objection 2 of 6
The paper states 'overstating the surgical skills' in reference to GPT-4o-mini and Lingshu-32B. Consider rephrasing to 'overstating surgical competence' or 'over-crediting' (the term used later) for terminological consistency with the rest of the section.
show 4 more
The paper says, at §3.2 / Appendix D.1 (perceive tool constraints)
Pith says: minor objection 3 of 6
The tool specification says 'repeating the same (frame, granularity) is redundant' and that duplicate calls are rejected. The paper does not specify how duplicate rejection is communicated to the agent — whether via an error message, a silent no-op, or a warning in the observation. Clarifying this would help reproducibility.
The paper says, at Appendix C.4, Eq. (1)
Pith says: minor objection 4 of 6
The PanelError definition uses range-normalized agreement for ordinal tasks. It should state explicitly whether the average is over criteria within an episode (macro) or over all criterion-episode pairs, as the choice affects the difficulty distribution.
The paper says, at Appendix D.3.5 (retrieval quality estimate)
Pith says: minor objection 5 of 6
The expert–expert linearly-weighted κ is 0.460, indicating only moderate agreement. The paper notes that the LLM judge agrees with each expert at least as well as the experts agree with each other. While this is factually stated, it would be helpful to discuss the implications of moderate expert agreement for the reliability of the retrieval-quality ground truth, which the paper uses as a gold standard for the AUROC calculation.
The paper says, at Throughout
Pith says: minor objection 6 of 6
Several tables in the appendix (e.g., Table 8 onward) use 'Bold/ underline mark the best / second-best per column among models' but the formatting is not visible in plain-text PDF. Ensure that the formatting (bold and underline) renders correctly in the camera-ready version.
What would settle it
Re-run the matched tool comparison (Tab. 3 Panel B) on SURGKNOWBENCH-Hard using SurgSAM-2 instead of SAM2 as the perceive() backbone, with all other settings identical. If the task-specific pattern (perception gains, workflow mixed) persists, the central claim is reinforced; if it changes substantially, the original conclusion is tool-dependent.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 10 Nov 2026
- Reader holds up
- Skeptic disagrees
- Editor wrote the verdict above
- Referee minor revision: 3 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This paper does what a good benchmark paper should: it harmonizes 11 surgical datasets into a single evaluation protocol, tests 17 MLLMs across 7 tasks, and honestly reports that zero-shot performance is weak on relational, procedural, and assessment tasks. The Stage I diagnostic is the strongest contribution here—it gives the community a clear picture of where general-purpose models stand on surgical scene understanding.
The Stage II agentic environment (SurgAgentGym) is a reasonable next step, but its conclusions are softer. The perceive() tool, built on SAM2 with relaxed thresholds, returns no usable proposals in 90-100% of action-recognition trajectories for GPT-4o-mini and Claude-Haiku. That means the agentic evaluation for those models on that task is essentially just the retrieve() tool, which also shows limited precision. The paper acknowledges this, which I appreciate, but it means the headline claim that 'agentic inference narrows the gap mainly for perception-heavy tasks' rests on a handful of open-source models where perceive() works passably. The matched tool comparison (Panel B) helps, but it covers only five backbones.
Other concerns are minor. The hard-set construction uses a panel of open-source MLLMs, which could introduce blind spots, though the reported flip rate of 0.32% suggests stability. The retrieval quality estimate relies on an LLM judge calibrated on only 50 human-annotated calls, with moderate expert-expert agreement (κ=0.46). And the knowledge base includes case records from the same datasets as evaluation episodes (training splits), which is fine in principle but not checked for near-duplicate visual patterns.
These are not fatal. The paper is honestly written, the experimental design is sound, and the central takeaway—that current models cannot handle relational or procedural surgical tasks even with tool support—is well-evidenced. The limitations are clearly stated.
Verdict: this deserves peer review. A minor revision should re-center the claims about agentic inference, noting that the perception tool's failure for proprietary models weakens the cross-model generalizability of the Stage II results. I'd bring this to a reading group and would cite it for the benchmark.
The strongest objection
The paper conducts an honest, thorough evaluation with appropriate controls (matched comparison, ablation, trajectory analysis) and clearly acknowledges its limitations, including the perceive() tool's failures and the knowledge base's bounded coverage. The central claim—that tool-mediated generalist agents improve perception-heavy tasks but not fine-grained localization or expert assessment—is well-supported by the evidence, and the authors do not overstate it. I find no internal inconsistency or unwarranted inference that would threaten the conclusion.
The reader identified the perceive() tool's quality as the weakest assumption. While this is a real challenge, the paper acknowledges it explicitly, quantifies its impact (90-100% no-proposal rates for some models), and addresses it through matched comparisons that separate tool access from the agent setting. The central claim does not depend on the tool being perfect; it depends on the pattern of task-specific improvement, which holds even with the acknowledged tool failures. Therefore, I find no load-bearing concern that would alter the verdict.
The referee's report
If the empirical findings hold, this paper makes several valuable contributions. (1) SurgKnowBench provides a harmonized, multi-procedure, multi-task evaluation suite that will help standardize future surgical scene understanding research; the dataset aggregation and unit conversion are done carefully with explicit metadata. (2) The two-stage evaluation design (direct inference then agentic inference) is a useful methodological contribution for disentangling pretrained competence from tool-mediated reasoning. (3) The trajectory analysis, including the failure-mode decomposition (wrong queries, ignoring retrieved evidence, robustness to noisy evidence), gives concrete targets for future work on agentic surgical AI. (4) The paper is unusually honest about failures: it reports that perceive() returns no usable proposals in 90-100% of trajectories for proprietary models on action recognition, that the best action-recognition score (27.6%) remains below the majority baseline (30.0%), and that tool access does not help subjective rubric-based assessment. This candor strengthens the paper's credibility.
Pith's reading of the paper (a machine's summary, not the authors')
This paper argues that current general-purpose multimodal large language models (MLLMs) have only superficial understanding of surgical scenes, and that equipping them with visual grounding and knowledge retrieval tools does not fix their core deficiencies in fine-grained spatial localization and expert-style assessment. The authors introduce a two-stage benchmark: SurgKnowBench tests 17 MLLMs zero-shot across 7 surgical tasks, revealing that models can recognize instruments but fail at relational, procedural, and safety judgments. In the second stage, SurgAgentGym tests whether tool-mediated agents can improve at inference time; they find gains in perception-heavy tasks like tool detection and action recognition, but the highest action-recognition score (27.6% macro-F1) is still below the majority baseline (30.0%). Agent trajectory analysis shows that agents frequently formulate wrong queries, retrieve irrelevant evidence, or ignore useful evidence, making epistemic control the binding constraint. The paper concludes that generalist agents cannot achieve surgical competence through tool use alone without better metacognitive capabilities.
What carries the argument. The argument is carried by a two-stage evaluation framework: Stage I uses SurgKnowBench, a diagnostic benchmark that harmonizes 11 public surgical datasets (376,602 samples across 7 procedures and 7 tasks) into a unified QA format to map the boundary of pretrained MLLM surgical competence. Stage II uses SurgAgentGym, an agentic environment where MLLM agents wield a perceive() tool (SAM2-based class-agnostic region proposals) and a retrieve() tool (hybrid lexical-dense search over SurgWiki, a curated multimodal knowledge base of 74,315 chunks from textbooks, guidelines, tutorial videos, and case records). The key evaluative mechanism is trajectory-level analysis that tracks whether the agent recognizes an information need, selects appropriate tools, retrieves relevant evidence, and grounds it in the surgical input, exposing three failure modes: poor query formulation, low robustness to imperfect evidence, and inability to integrate retrieved evidence when it is relevant.
Core claim. The central discovery is that general-purpose MLLMs, while retaining partial visual priors for surgical instruments, are unable to reason about surgical interactions (triplet detection below 5% macro-F1), procedural phase (accuracy below majority baseline on most datasets), or expert-level workflow assessment (CVS and skill assessment show near-chance or negative agreement with expert rubrics). When deployed as agents with access to a regulated surgical knowledge base and a general-purpose visual grounding tool, they improve on perception-heavy tasks (tool detection, action recognition) but remain unable to localize tools precisely or make rubric-consistent judgments. The binding failure is not the absence of domain knowledge but the agents' inability to formulate effective search queries, reconcile imperfect evidence, and ground retrieved knowledge in the surgical scene. This work establishes that epistemic control—the metacognitive ability to seek, evaluate, and integrate evidence—is the missing ingredient for generalist surgical understanding, and that current tool-mediated inference is insufficient to supply it.
The premise it rests on. The paper assumes that the SAM2-based perceive() tool with general-purpose thresholds (pred_iou_thresh=0.70, stability_score_thresh=0.90) provides region proposals usable enough for agents to improve surgical understanding, but this tool returns no usable region proposals in 90-100% of action-recognition trajectories for GPT-4o-mini and Claude-Haiku-4.5, and the retrieval relevance score (AUROC 0.817 against experts) still leaves many calls with low-quality results that depend heavily on agent query formulation.
If the paper is right (Pith's inferences, not the authors')
Any attempt to build surgical superintelligence from general-purpose MLLMs alone must first solve the epistemic control problem; simply adding more domain data at inference time via retrieval will not suffice if the agent cannot formulate good queries or integrate conflicting evidence.
Surgical AI benchmarks that only measure final-answer accuracy may be misleading; trajectory-level analysis is necessary to distinguish cases where the agent retrieved the right evidence but ignored it from cases where retrieval failed outright.
The finding that tool access can actually degrade performance in proprietary models (GPT-4o-mini and Claude-Haiku-4.5 declined on action recognition) suggests that tool-use policies must be learned, not just given, and that naive tool access can backfire.
Persistent failure on fine-grained localization and expert-style assessment points to fundamental limitations in current vision encoders and reasoning chains that no amount of external knowledge can bypass without architectural changes.
The fact that retrieval hurt performance in the medical-domain model Lingshu-32B suggests that domain-specific pretraining may already embed knowledge that is more reliable than retrieved evidence; a hybrid strategy that selectively invokes retrieval only when model confidence is low could outperform full tool access.
The reiteration of the same three failure modes (query quality, robustness, grounding) across diverse backbones hints that epistemic control is a general limitation of transformer-based autoregressive generation as a reasoning substrate for open-world medical tasks, not an artifact of any single model family.
If the trajectory-level analysis is replicated, it could inform a new class of training objectives that explicitly reward the agent for improving its query quality and evidence-use consistency, rather than just final answer accuracy.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Evaluating general-purpose multimodal large language models (MLLMs) for surgical scene understanding remains limited due to scarce surgical data and fragmented evaluation protocols. We address these challenges through a two-stage evaluation framework for surgical scene understanding. In Stage I, we introduce SurgKnowBench, a diagnostic benchmark that harmonizes public surgical datasets across procedures and tasks and evaluates proprietary and open-source MLLMs to map the boundary of pretrained surgical competence. SurgKnowBench reveals that current models retain partial visual priors but remain weak on relational and procedural understanding and workflow assessment. Building on this failure frontier and motivated by the scaffolded nature of surgical training, we introduce SurgAgentGym, an agentic environment where general-purpose MLLM agents can use bounded knowledge search over SurgWiki and general-purpose visual grounding for surgical tasks. SurgAgentGym evaluates whether generalist agents can improve surgical understanding at inference time through tool-mediated evidence use, without additional surgical training. Our results show that agentic inference narrows the gap mainly for perception-heavy tasks, while fine-grained localization and expert-style assessment remain persistent bottlenecks. This opens a path toward future studies of agentic surgical understanding.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-10
The authors' main claim, as the classifier picked it out 1.00
“Our results show that agentic inference narrows the gap mainly for perception-heavy tasks, while fine-grained localization and expert-style assessment remain persistent bottlenecks.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.92 · human participants 0.10 · controlled comparison 0.50 · randomized assignment 0.01 · states quantitative result 0.05 · compares with prior results 0.75 · mentions code or data availability 0.08
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Evaluating general-purpose multimodal large language models (MLLMs) for surgical scene understanding remains limited due to scarce surgical data and fragmented evaluation protocols. | gap 0.61 | 0.06 | 0.02 | asserted 1.00 |
| We address these challenges through a two-stage evaluation framework for surgical scene understanding. | contribution 0.78 | 0.83 | 0.03 | asserted 0.98 |
| In Stage I, we introduce SurgKnowBench, a diagnostic benchmark that harmonizes public surgical datasets across procedures and tasks and evaluates proprietary and open-source MLLMs to map the boundary of pretrained surgical competence. | method 0.55 | 0.93 | 0.04 | asserted 0.99 |
| SurgKnowBench reveals that current models retain partial visual priors but remain weak on relational and procedural understanding and workflow assessment. | result 0.98 | 0.92 | 0.17 | asserted 1.00 |
| Building on this failure frontier and motivated by the scaffolded nature of surgical training, we introduce SurgAgentGym, an agentic environment where general-purpose MLLM agents can use bounded knowledge search over SurgWiki and general-purpose visual grounding for surgical tasks. | contribution 0.69 | 0.82 | 0.04 | asserted 0.95 |
| SurgAgentGym evaluates whether generalist agents can improve surgical understanding at inference time through tool-mediated evidence use, without additional surgical training. | method 0.56 | 0.53 | 0.07 | asserted 0.39 |
| Our results show that agentic inference narrows the gap mainly for perception-heavy tasks, while fine-grained localization and expert-style assessment remain persistent bottlenecks. | result 0.94 | 0.96 | 0.70 | asserted 1.00 |
| This opens a path toward future studies of agentic surgical understanding. | future work 0.99 | 0.09 | 0.05 | hedged possibility 0.72 |
What kind of paper the classifier reports this is: dataset or benchmark 0.86 · evidence from observation or experiment 0.95
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-10: 1 request, 6,259 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (8)
Figures are the authors', reproduced from the paper.
References (84)
- [1] Chinedu Innocent Nwoye, Tong Yu, Cristians Gonzalez, Barbara Seeliger, Pietro Mascagni, Didier Mutter, Jacques Marescaux, and Nicolas Padoy. Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos.Medical Image Analysis, 78: 102433... on Pithdoi:10.1016/j.media.2022.102433
- [2] Martin Wagner, Beat-Peter Müller-Stich, Anna Kisilenko, Duc Tran, Patrick Heger, Lars Mündermann, David M. Lubotsky, Benjamin Müller, Tornike Davitashvili, Manuela Capek, et al. Comparative validation of machine learning algorithms for surgical workflow and skill analysis with... on Pithdoi:10.1016/j.media.2023.102770
- [3] Natalia Valderrama, Paola Ruiz Puentes, Isabela Hernández, Nicolás Ayobi, Mathilde Verlyck, Jessica Santander, Juan Caicedo, Nicolás Fernández, and Pablo Arbeláez. Towards holistic surgical scene understanding. InInternational conference on medical image computing and computer... on Pith
- [4] Zhitao Zeng, Zhu Zhuo, Xiaojun Jia, Erli Zhang, Junde Wu, Jiaan Zhang, Yuxuan Wang, Chang Han Low, Jian Jiang, Zilong Zheng, Xiaochun Cao, Yutong Ban, Qi Dou, Yang Liu, and Yueming Jin. SurgVLM: A large vision-language model and systematic evaluation benchmark for surgical int... on PithPith's reviewarXiv
- [5] Andru P. Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel de Mathelin, and Nicolas Padoy. EndoNet: A deep architecture for recognition tasks on laparoscopic videos. IEEE Transactions on Medical Imaging, 36(1):86–97, 2017. doi: 10.1109/TMI.2016.2593957 on Pithdoi:10.1109/tmi.2016.2593957
- [6] Matthias Carstens, Shubha Vasisht, Zheyuan Zhang, Iulia Barbur, Annika Reinke, Lena Maier- Hein, Daniel A Hashimoto, and Fiona R Kolbinger. Artificial intelligence for surgical scene understanding: a systematic review and reporting quality meta-analysis.npj Digital Medicine, 9... on Pithdoi:10.1038/s41746-025-02227-4
- [7] Lena Maier-Hein, Matthias Eisenmann, Duygu Sarikaya, Keno März, Toby Collins, Anand Malpani, Johannes Fallert, Hubertus Feussner, Stamatia Giannarou, Pietro Mascagni, et al. Surgical data science–from concepts toward clinical translation.Medical image analysis, 76: 102306, 2022 on Pith
- [8] Manuel Sebastián Ríos, María Alejandra Molina-Rodriguez, Daniella Londoño, Camilo Andrés Guillén, Sebastián Sierra, Felipe Zapata, and Luis Felipe Giraldo. Cholec80-CVS: An open dataset with an evaluation of Strasberg’s critical view of safety for AI.Scientific Data, 10(1): 19... on Pithdoi:10.1038/s41597-023-02073-7
- [9] Joel L. Lavanchy, Sanat Ramesh, Diego Dall’Alba, Cristians Gonzalez, Paolo Fiorini, Beat P. Müller-Stich, Philipp C. Nett, Jacques Marescaux, Didier Mutter, and Nicolas Padoy. Challenges 11 in multi-centric generalization: phase and step recognition in Roux-en-Y gastric bypass... on Pithdoi:10.1007/s11548-024-03166-3
- [10] Ziyi Wang, Bo Lu, Yonghao Long, Fangxun Zhong, Tak-Hong Cheung, Qi Dou, and Yunhui Liu. AutoLaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy. InMedical Image Computing and Computer Assisted Intervention (MICCAI),... on Pithdoi:10.1007/978-3-031-16449-1_46
- [11] Lena Maier-Hein, Martin Wagner, Tobias Ross, Annika Reinke, Sebastian Bodenstedt, Peter M. Full, Hellena Hempe, Diana Mindroc-Filimon, Patrick Scholz, Thuy Nuong Tran, et al. Heidel- berg colorectal data set for surgical data science in the sensor operating room.Scientific Dat... on Pithdoi:10.1038/s41597-021-00882-2
- [12] Nicolás Ayobi, Santiago Rodríguez, Alejandra Pérez, Isabela Hernández, Nicolás Aparicio, Eugénie Dessevres, Sebastián Peña, Jessica Santander, Juan Ignacio Caicedo, Nicolás Fernández, and Pablo Arbeláez. Pixel-wise recognition for holistic surgical scene understanding.Medical ... on Pithdoi:10.1016/j.media.2025.103726
- [13] Adrito Das, Danyal Z Khan, Dimitrios Psychogyios, Yitong Zhang, John G Hanrahan, Francisco Vasconcelos, You Pang, Zhen Chen, Jinlin Wu, Xiaoyang Zou, Guoyan Zheng, Abdul Qayyum, Moona Mazher, Imran Razzak, Tianbin Li, Jin Ye, Junjun He, Szymon Płotka, Joanna Kaleta, Amine Yaml... on Pith
- [14] Klaus Schoeffmann, Mario Taschwer, Stephanie Sarny, Bernd Münzer, Manfred Jürgen Primus, and Doris Putzgruber-Adamitsch. Cataract-101: video dataset of 101 cataract surgeries. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys), pages 421–425, 2018. doi: 10.114... on Pithdoi:10.1145/3204949.3208137
- [15] Hassan Al Hajj, Mathieu Lamard, Pierre-Henri Conze, Soumali Roychowdhury, Xiaowei Hu, Gabija Maršalkait˙e, Odysseas Zisimopoulos, Muneer Ahmad Dedmari, Fenqiang Zhao, Jonas Prellberg, et al. CATARACTS: Challenge on automatic tool annotation for cataRACT surgery. Medical Image ... on Pithdoi:10.1016/j.media.2018.11.008
- [16] OpenAI. GPT-4o mini: advancing cost-efficient intelligence, July 2024. URLhttps://openai. com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ on Pith
- [17] Anthropic. Claude Haiku 4.5. https://www.anthropic.com/news/claude-haiku-4-5 , on Pith
- [18] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. InternVL 2.5: Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2024. URL https: //arxiv.org... on PithPith's reviewarXiv
- [19] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han... on Pith
- [20] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junya... on PithPith's reviewarXiv
show 64 more
- [21] URLhttps://arxiv.org/abs/2502.13923 on PithPith's reviewarXiv
- [22] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. InternVL3: Exploring advanced training and test- time recipes for open-source multimodal models, 2025. URL https://arxiv.org/abs/2504. 10479 on Pith
- [23] Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume La... on PithPith's reviewarXiv
- [24] Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Hardy Chen, Xidong Wang, Zhenyang Cai, Ke Ji, Xiang Wan, et al. Towards injecting medical visual knowledge into multimodal LLMs at scale. InProceedings of the 2024 Conference on Empirical Methods in Natural Langu... on Pith
- [25] Gemma Team. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786 on PithPith's reviewarXiv
- [26] Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, Justin Chen, Fereshteh Mahvar, Liron Yang, et al. MedGemma technical report, 2025. URL https://arxiv.org/ abs/2507.05201 on PithPith's reviewarXiv
- [27] LingshuTeam. Lingshu: A generalist foundation model for unified multimodal medical under- standing and reasoning, 2025. URLhttps://arxiv.org/abs/2506.07044 on PithPith's reviewarXiv
- [28] FT Drake, KD Horvath, AB Goldin, and KW Gow. The general surgery chief resident operative experience: 23 years of national ACGME case logs.JAMA Surgery, 148(9):841–847, 2013. doi: 10.1001/jamasurg.2013.2919 on Pithdoi:10.1001/jamasurg.2013.2919
- [29] David R. Krathwohl. A revision of Bloom’s taxonomy: An overview.Theory Into Practice, 41(4):212–218, 2002. ISSN 00405841, 15430421. URL http://www.jstor.org/stable/ 1477405 on Pith
- [30] Kaiwen Wei, Rui Shan, Dongsheng Zou, Jianzhong Yang, Bi Zhao, Junnan Zhu, and Jiang Zhong. MIRAGE: Scaling test-time inference with parallel graph-retrieval-augmented reasoning chains, 2026. URLhttps://arxiv.org/abs/2508.18260 on PitharXiv
- [31] Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval-augmented generation for medicine. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Computational Linguistics: ACL 2024, pages 6233–6251, Bangkok, Thailand, ... on Pithdoi:10.18653/v1/2024
- [32] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. SAM 2: Segment anything in images and videos. InInternational Conference on Learning Representations, volume 2025, pages 280... on Pith
- [33] B Huo, N Marfo, P Sylla, E Calabrese, S Kumar, BJ Slater, DS Walsh, and W V osburg. Clinical artificial intelligence: teaching a large language model to generate recommendations that align with guidelines for the surgical management of GERD.Surgical Endoscopy, 38(10):5668–5677, on Pith
- [34] OpenAI. GPT-5 System Card.https://openai.com/index/gpt-5-system-card/, 2025. 13 on Pith
- [35] Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, Luis Herrera, Wenqi Li, Vladimir Iglovikov, Huoling Luo, Jian Yang, Danail Stoyanov, Lena Maier-Hein, Stefanie Speidel, and M... on PithPith's reviewarXiv
- [36] Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set- of-Mark prompting unleashes extraordinary visual grounding in GPT-4V.arXiv preprint arXiv:2310.11441, 2023 on PithPith's reviewarXiv
- [37] Jiajie Li, Garrett Skinner, Gene Yang, Brian R. Quaranto, Steven D. Schwaitzberg, Peter C. W. Kim, and Jinjun Xiong. LLaV A-Surg: Towards multimodal surgical assistant via structured surgical video learning.arXiv preprint arXiv:2408.07981, 2024. URL https: //arxiv.org/abs/2408.07981 on PithPith's reviewarXiv
- [38] Kun Yuan, Manasi Kattel, Joël L Lavanchy, Nassir Navab, Vinkle Srivastav, and Nicolas Padoy. Advancing surgical VQA with scene graph knowledge.International Journal of Computer As- sisted Radiology and Surgery, 19(7):1409–1417, July 2024. doi: 10.1007/s11548-024-03141-y. URLht... on Pithdoi:10.1007/s11548-024-03141-y
- [39] Kun Yuan, Vinkle Srivastav, Nassir Navab, and Nicolas Padoy. HecVL: Hierarchical video- language pretraining for zero-shot surgical phase recognition. InMedical Image Computing and Computer Assisted Intervention – MICCAI 2024, volume 15006 ofLecture Notes in Computer Science, ... on Pithdoi:10.1007/978-3-031-72089-5_
- [40] Alejandra Perez, Anita Rau, Lee White, Busisiwe Mlambo, Chinedu Nwoye, Muhammad Ab- dullah Jamal, and Omid Mohareri. Sureon: A benchmark and vision-language-model for surgical reasoning.arXiv preprint arXiv:2603.06570, 2026 on PithPith's reviewarXiv
- [41] Yaoqian Li, Xikai Yang, Dunyuan Xu, Yang Yu, Litao Zhao, Xiaowei Hu, Jinpeng Li, and Pheng-Ann Heng. SurgPub-Video: A comprehensive surgical video framework for enhanced surgical intelligence in vision-language model. InProceedings of the AAAI Conference on Artificial Intellig... on Pithdoi:10.48448/ppqb-tp30
- [42] Jianhui Wei, Zikai Xiao, Danyu Sun, Luqi Gong, Zongxin Yang, Zuozhu Liu, and Jian Wu. SurgBench: A unified large-scale benchmark for surgical video analysis, 2025. URL https: //arxiv.org/abs/2506.07603 on PithPith's reviewarXiv
- [43] Simon Pezold, Jérôme A. Kurylec, Jan S. Liechti, Beat Peter Müller, and Joël L. Lavanchy. Leveraging generic foundation models for multimodal surgical data analysis. InML-CDS Workshop, MICCAI, 2025. arXiv:2509.06831 on PithPith's reviewarXiv
- [44] Jinlin Wu, Felix Holm, Chuxi Chen, An Wang, Yaxin Hu, Xiaofan Ye, Zelin Zang, Miao Xu, Lihua Zhou, Huai Liao, Danny T. M. Chan, Ming Feng, Wai S. Poon, Hongliang Ren, Dong Yi, Nassir Navab, Gaofeng Meng, Jiebo Luo, Hongbin Liu, and Zhen Lei. UniSurg: A video-native foundation ... on PithPith's reviewarXiv
- [45] Zhitao Zeng, Mengya Xu, Jian Jiang, Pengfei Guo, Yunqiu Xu, Zhu Zhuo, Chang Han Low, Yufan He, Dong Yang, Chenxi Lin, Yiming Gu, Jiaxin Guo, Yutong Ban, Daguang Xu, Qi Dou, and Yueming Jin. SurgΣ: A spectrum of large-scale multimodal data and foundation models for surgical int... on PitharXiv
- [46] Puxun Tu, Ce Zheng, Xiaoling Xie, Jiao Lv, Meng Xie, Jinming Guo, Shengjie Yin, Kunliang Qiu, Yue Wei, Chongyang Wang, Jingfeng Cai, Wei Mi, Yafu Wang, Xiao Zhang, Danba Jiachu, Kun Peng, Wanqi Zhang, Fengfeng Tong, Huiying Chu, Peiquan Zhao, Mingzhi Zhang, and Xiaojun Chen. A... on Pithdoi:10.1038/s41551-026-01622-w
- [47] Samuel Schmidgall, Joseph Cho, Cyril Zakka, and William Hiesinger. GP-VLS: A general- purpose vision language model for surgery, 2024. URL https://arxiv.org/abs/2407. 19305 on Pith
- [48] Kanggil Park, Yongjun Jeon, Soyoung Lim, Seonmin Park, Jongmin Shin, Jung Yong Kim, Sehyeon An, Jinsoo Rhu, Jongman Kim, Gyu-Seong Choi, Namkee Oh, and Kyu-Hwan Jung. A generalizable foundation model for intraoperative understanding across surgical procedures. arXiv preprint a... on PitharXiv
- [49] Yixing Jiang, Kameron C. Black, Gloria Geng, Danny Park, James Zou, Andrew Y . Ng, and Jonathan H. Chen. MedAgentBench: A virtual EHR environment to benchmark medical LLM agents.NEJM AI, 2(9):AIdbp2500144, 2025. doi: 10.1056/AIdbp2500144. URL https://ai.nejm.org/doi/full/10.10... on Pithdoi:10.1056/aidbp2500144
- [50] Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, and Yixin Wang. MMedAgent: Learning to use medical tools with multi-modal agent. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the As... on Pithdoi:10.18653/v1/
- [51] Samuel Schmidgall, Rojin Ziaei, Carl Harris, Ji Woong Kim, Eduardo Pontes Reis, Jeffrey Jopling, and Michael Moor. AgentClinic: a multimodal benchmark for tool-using clinical AI agents.npj Digital Medicine, 2026. ISSN 2398-6352. doi: 10.1038/s41746-026-02674-7. URL https://doi... on Pithdoi:10.1038/s41746-026-02674-7
- [52] Chang Han Low, Zhu Zhuo, Ziyue Wang, Jialang Xu, Haofeng Liu, Nazir Sirajudeen, Matthew Boal, Philip J. Edwards, Danail Stoyanov, Nader Francis, Jiehui Zhong, Di Gu, Evangelos B. Mazomenos, and Yueming Jin. CARES: Collaborative agentic reasoning for error detection in surgery,... on PithPith's reviewarXiv
- [53] Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. MinerU: An open-source solution for precise document content extraction.arXiv preprint arXiv:2409.18839, 2024 on PithPith's reviewarXiv
- [54] Chang Han Low, Ziyue Wang, Tianyi Zhang, Zhu Zhuo, Zhitao Zeng, Evangelos B. Mazomenos, and Yueming Jin. SurgRAW: Multi-agent workflow with chain of thought reasoning for robotic surgical video analysis.IEEE Robotics and Automation Letters, 11(4):4857–4864, 2026. doi: 10.1109/... on Pithdoi:10.1109/lra.2026.3665443
- [55] Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: BM25 and beyond.Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009 on Pith
- [56] Gordon V . Cormack, Charles L A Clarke, and Stefan Buettcher. Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. InSIGIR ’09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages ... on Pith
- [57] Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureanu, Bo Wang, Sedigheh Eslami, Scott Martens, Maximilian Werk, Nan Wang, et al. jina-embeddings- v4: Universal embeddings for multimodal multilingual retrieval. InProceedings of the 5th Workshop on... on Pith
- [58] Norman Williams, P Ronan O’Connell, and Andrew McCaskie.Bailey & Love’s Short Practice of Surgery, 27th Edition. CRC Press, 2018 on Pith
- [59] Susan Standring.Gray’s Anatomy: The Anatomical Basis of Clinical Practice, 41st Edition. Elsevier Health Sciences, 2015 on Pith
- [60] Jina AI. jina-reranker-v2-base-multilingual. https://jina.ai/models/ jina-reranker-v2-base-multilingual/ , 2024. Model documentation. Released 2024-06-25. Accessed: 2026-05-06 on Pith
- [61] Society of American Gastrointestinal and Endoscopic Surgeons (SAGES). SAGES guidelines for the clinical application of laparoscopic biliary tract surgery. https://www.sages.org/ publications/guidelines/, 2023 on Pith
- [62] Nicola de’Angelis, Fausto Catena, Riccardo Memeo, Federico Coccolini, Aleix Martínez-Pérez, Oreste M Romeo, Belinda De Simone, Salomone Di Saverio, Raffaele Brustia, Rami Rhaiem, et al. 2020 WSES guidelines for the detection and management of bile duct injury during cholecyste... on Pith
- [63] Vanderbilt University Medical Center. Open Manual of Surgery in Resource-Limited Settings. https://www.vumc.org/global-surgical-atlas/about. Accessed: 2026-05-04. 15 on Pith
- [64] Shivali Patel, V Reddy, KR Shumway, and JF Araujo. StatPearls.Treasure Island (FL): StatPearls Publishing, 2025 on Pith
- [65] Alejandra Perez, Chinedu Nwoye, Ramtin Raji Kermani, Omid Mohareri, and Muhammad Ab- dullah Jamal. SurgLaVi: Large-scale hierarchical dataset for surgical vision-language represen- tation learning.Medical Image Analysis, 110:103982, 2026 on Pith
- [66] Go Wakabayashi, Yukio Iwashita, Taizo Hibi, Tadahiro Takada, Steven M Strasberg, Horacio J Asbun, Itaru Endo, Akiko Umezawa, Koji Asai, Kenji Suzuki, et al. Tokyo Guidelines 2018: surgical management of acute cholecystitis: safe steps in laparoscopic cholecystectomy for acute ... on Pith
- [67] Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C-Pack: Packed resources for general Chinese embeddings. InProceedings of the 47th Interna- tional ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page... on Pithdoi:10.1145/3626772.3657878
- [68] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training, on Pith
- [69] Zhitong Wang, Cheng Gao, Chaojun Xiao, Yufei Huang, Shuzheng Si, Kangyang Luo, Yuzhuo Bai, Wenhao Li, Tangjian Duan, Chuancheng Lv, Guoshan Lu, Gang Chen, Fanchao Qi, and Maosong Sun. Document segmentation matters for retrieval-augmented generation. In Wanxiang Che, Joyce Nabe... on Pith
- [70] Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. Nomic Embed: Training a reproducible long context text embedder, 2025. URL https://arxiv.org/abs/ 2402.01613 on PithPith's reviewarXiv
- [71] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International ... on Pithdoi:10.18653/v1/d19-1410
- [72] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MPNet: masked and permuted pre-training for language understanding. InProceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY , USA, 2020. Curran Associates Inc on Pith
- [73] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning, 2023. URL https: //arxiv.org/abs/2308.03281 on PithPith's reviewarXiv
- [74] Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, and Han Xiao. jina-embeddings-v3: Multilingual embeddings with task LoRA, 2024. URL https://arxiv. org/abs/2409.10173 on PithPith's reviewarXiv
- [75] OpenAI. GPT-4o system card, 2024. URLhttps://arxiv.org/abs/2410.21276 on PithPith's reviewarXiv
- [76] Rodrigo Nogueira and Kyunghyun Cho. Passage Re-ranking with BERT.arXiv e-prints, art. arXiv:1901.04085, January 2019. doi: 10.48550/arXiv.1901.04085 on PithPith's reviewarXivdoi:10.48550/arxiv.1901.04085
- [77] Qiao Jin, Won Kim, Qingyu Chen, Donald C Comeau, Lana Yeganova, W John Wilbur, and Zhiyong Lu. MedCPT: Contrastive pre-trained transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval.Bioinformatics, 39(11):btad651, 11 2023. 16 ISSN 1367-... on Pithdoi:10.1093/bioinformatics/btad651
- [78] Lixin Fang, Lei Mou, Yuanyuan Gu, Yan Hu, Bang Chen, Xu Chen, Yang Wang, Jiang Liu, and Yitian Zhao. Global–local multi-stage temporal convolutional network for cataract surgery phase recognition.BioMedical Engineering OnLine, 21(1):82, 2022. doi: 10.1186/ s12938-022-01048-w on Pith
- [79] Hao Ding, Zhongpai Gao, Benjamin Planche, Tianyu Luan, Abhishek Sharma, Meng Zheng, Ange Lou, Terrence Chen, Mathias Unberath, and Ziyan Wu. Neural finite-state machines for surgical phase recognition.arXiv preprint arXiv:2411.18018, 2025 on PithPith's reviewarXiv
- [80] Chinedu Innocent Nwoye, Deepak Alapatt, Tong Yu, Armine Vardazaryan, Fangfang Xia, Zix- uan Zhao, Tong Xia, Fucang Jia, Yuxuan Yang, Hao Wang, Derong Yu, Guoyan Zheng, Xiaotian Duan, Neil Getty, Ricardo Sanchez-Matilla, Maria Robu, Li Zhang, Huabin Chen, Jiacheng Wang, Lianshe... on Pith
- [81] Saurav Sharma, Chinedu Innocent Nwoye, Didier Mutter, and Nicolas Padoy. Surgi- cal action triplet detection by mixed supervised learning of instrument-tissue interactions. InMedical Image Computing and Computer Assisted Intervention — MICCAI 2023, vol- ume 14228 ofLecture Not... on Pithdoi:10.1007/978-3-031-43996-4_48
- [2022] URLhttps://arxiv.org/abs/2212.03533 on PithPith's reviewarXiv
- [2024] doi: 10.1007/s00464-024-11155-5 on Pithdoi:10.1007/s00464-024-11155-5
- [2025] Model card; accessed 2026-05 on Pith
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment