Specialized Foundation Models for Intelligent Operating Rooms

Chantal Pellegrini; David Bani-Harouni; Ege \"Ozsoy; Kun Yuan; Matthias Keicher; Nassir Navab

REVIEW 3 major objections 5 minor 30 references

Specialized Foundation Models for Intelligent Operating Rooms

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ORQA model beats generalist chatbots at reading the OR

desk verdict A useful surgical QA benchmark undermined by a missing control: the model trains on the same benchmark it tests on, so the claim that specialized training is required is not yet established. read the letter →

arxiv 2505.12890 v2 pith:AXNULDYM submitted 2025-05-19 cs.CV

Ege Özsoy , Chantal Pellegrini , David Bani-Harouni , Kun Yuan , Matthias Keicher , Nassir Navab This is my paper · ORCID

classification cs.CV

keywords operatingroomunderstandingmultimodalquestionansweringsurgicalscenevision-languagefoundationmodelsdomain-specializedtrainingknowledgedistillationgraphssafety

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ORQA, a benchmark and a family of models for holistic operating-room (OR) understanding, and argues that this kind of understanding cannot be achieved by general-purpose vision-language models alone. The evidence is a head-to-head comparison: on the ORQA test set, the specialized model scores 0.559, while GPT-4o, GPT-4.1, Gemini 2.5 Flash, and Qwen2-VL score between 0.149 and 0.249, hovering around a trivial statistical baseline of 0.191. The authors' claim is that training a multimodal language model on one million question-answer pairs generated from four public OR datasets yields reliable spatial, temporal, and safety-related reasoning about surgical scenes, and that distilled smaller versions remain accurate enough for real-time use. If true, this points to domain-specialized training as a necessary ingredient for intelligent OR systems.

What carries the argument

The central object is the ORQA benchmark and model, which together recast 23 surgical scene-understanding tasks as a single question-answering framework. Questions are generated by template from the annotations of four public OR datasets, then diversity-sampled to one million training pairs, with the same procedure yielding 10,000 validation and 10,000 test pairs. The model is a multimodal LLM built on the Qwen2-VL backbone, with encoders for RGB images, point clouds, audio, speech transcripts, robot logs, and tracker data, plus an optional memory of prior scene graphs for temporal reasoning; the ORQA Score metric gives task-specific partial credit, and performance is compared against generalist vision-language models in a zero-shot setting.

What would settle it

Fine-tune a comparable generalist vision-language model on the same one million ORQA training pairs and evaluate it on the ORQA test set; if it reaches ORQA's 0.559 score without the OR-specific encoders or scene-graph machinery, the claim that domain-specialized surgical training is required would collapse.

Watch

Extended reading notes

Core claim

ORQA (Operating Room Question Answering) establishes that a domain-specialized multimodal foundation model can read an operating room in a way that generalist chatbots cannot. Trained on one million curated QA pairs drawn from MVOR, 4D-OR, EgoSurgery, and MM-OR, the model reaches an ORQA Score of 0.559 (95% CI 0.551–0.567), far above the best generalist baseline, GPT-4.1 at 0.249, and above the 0.191 trivial baseline that guesses the most frequent answer per task. The model integrates RGB images, point clouds, audio, speech, robot logs, and tracking data through modality-specific encoders feeding a language model, with optional scene-graph reasoning, and the paper reports strong performance across all 23 tasks, including tool localization, 3D distance estimation, sterility breach detection, gaze-object detection, and next-robot-step prediction. The paper further shows that knowledge-distilled variants, down to 278M parameters, keep scores near 0.525 while running more than three times faster, supporting local real-time deployment.

Load-bearing premise

The automatically generated QA pairs, derived from the annotation schemas of the four datasets, are a valid measure of clinically meaningful surgical scene understanding.

Editorial extensions

If this is right

Generalist vision-language models, as currently trained, cannot be relied on for surgical scene understanding even with careful prompting, so OR-specific training data is needed for this domain.
ORQA provides a single, interpretable evaluation format for comparing OR perception models across modalities, datasets, and task types.
The distilled ORQA variants make local, real-time, and privacy-preserving intraoperative assistance feasible on resource-constrained hardware.
Leave-one-out experiments suggest that adapting ORQA to a new OR dataset requires only light fine-tuning, not full retraining.
The QA-template framework is extensible to new modalities and clinical tasks, allowing the benchmark and model to grow with new OR technologies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the QA pairs are generated from each dataset's own annotation schemas, the benchmark may reward reproducing those labels rather than genuine clinical reasoning; a generalist model fine-tuned on the same pairs might close much of the reported gap.
The ORQA Score's partial-credit rules and BLEU-based OCR scoring can inflate apparent competence, so the headline margin may narrow under stricter exact-match evaluation.
A natural next experiment is a prospective test on real, unannotated OR videos from a new hospital; the paper's leave-one-out results suggest zero-shot transfer to unseen annotation conventions remains moderate.
The 23 tasks are largely perception and state recognition, so the path from ORQA's scene understanding to improved patient outcomes or surgical safety is not directly demonstrated by this paper.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Short version: ORQA is a real benchmark-and-model contribution to surgical AI, but the headline margin over generalist VLMs is confounded by the fact that the model trains on the same benchmark it is tested on. Worth a careful referee, not a desk reject.

What's genuinely new: The paper unifies four public OR datasets (MVOR, 4D-OR, EgoSurgery, MM-OR) into one multimodal QA benchmark spanning 23 tasks, generating 100M QA pairs and curating 1M for training plus 10k each for validation and test. That unification alone is useful—the field is fragmented by dataset- and task-specific models. The model is an honest assembly of proven parts: Qwen2-VL backbone, Point Transformer V3, CLAP, and the authors' own MM2SG scene-graph architecture. The distilled variants (3.3x speedup for ~0.03 score drop) are a practical plus. The leave-one-out experiments are the right instinct and show that adaptation to a new dataset works better than zero-shot, even if zero-shot transfer is only moderate.

The soft spots, in proportion. The main one is experimental control. ORQA trains on 1M QA pairs from the same four datasets and annotation schemas that define the test set; the generalist baselines are all zero-shot. There is no fine-tuned generalist baseline—no Qwen2-VL-2B trained on those same pairs. Without that, the margin between 0.559 and 0.249 could be mostly task fine-tuning on the benchmark itself, not ORQA's multimodal architecture or domain specialization. The paper also never says the test timepoints are disjoint from training timepoints, which is a leakage risk. The ORQA Score is a reasonable composite but uses hand-picked thresholds, so it is not a neutral yardstick.

None of this kills the contribution. The benchmark is a credible evaluation substrate even if the model claim needs qualification. The authors are candid about moderate cross-dataset transfer, which actually supports the worry about schema memorization. The heavy self-citation is justified: the model builds directly on their MM2SG and 4D-OR work.

Who this is for: anyone working in surgical scene understanding or OR monitoring. Read it for the benchmark; treat the model results as promising but confounded.

Recommendation: send to peer review, but require a fine-tuned generalist baseline, a clear statement on train/test timepoint disjointness, and release of benchmark and code with the revision. That would move it from conditionally acceptable to solid.

Referee Report

3 major / 5 minor

Summary. The paper introduces ORQA, a multimodal question-answering benchmark and model for operating-room scene understanding. The benchmark is generated from four publicly available OR datasets (MVOR, 4D-OR, EgoSurgery, MM-OR) by applying automated templates to their annotations, yielding 23 tasks and, after diversity sampling, 1 million training QA pairs plus 10,000 validation and 10,000 test pairs. The ORQA model extends Qwen2-VL with encoders for RGB, point clouds, audio, speech, robot logs, and tracking data, and is trained on the 1M QA pairs. The paper reports that ORQA achieves an ORQA Score of 0.559 (95% CI 0.551–0.567), far above zero-shot GPT-4o, GPT-4.1, Gemini 2.5 Flash, and Qwen2-VL (2B/7B), and introduces distilled variants with lower parameter counts and higher throughput. The central claim is that domain-specialized multimodal training on surgical annotation data is required for reliable OR scene understanding, whereas generalist vision-language models are inadequate.

Significance. If the central comparison were clean, ORQA would be a valuable contribution: it unifies four disparate surgical datasets into a single QA benchmark, provides a reproducible evaluation protocol, and demonstrates that a moderately sized multimodal model can handle a wide range of OR perception tasks with structured outputs. The planned public release of the benchmark, model, and evaluation code is a clear strength, as is the inclusion of leave-one-out transfer experiments and distilled deployment variants. However, the significance is substantially contingent on resolving the evaluation asymmetry: the benchmark serves simultaneously as training and test data, and the strongest result—the large margin over generalists—may reflect template memorization or task fine-tuning rather than domain-specific architectural design. As written, the paper does not yet establish that domain-specialized training is *required*.

major comments (3)

[Question–Answer Generation and Sampling; Evaluation Procedure] The paper does not describe any guarantee that the 10,000 test QA pairs are disjoint from the 1 million training QA pairs at the level of timepoints, clips, or recordings. The training pairs are generated from the training splits of the four datasets, and the test pairs are generated "analogously," but no temporal or scene-level split is specified. Because all QA pairs are produced by the same automated templates from the same annotation schemas, an ORQA model trained on 1M template-generated pairs could score highly by reproducing dataset-specific label distributions and template answer priors rather than by general OR understanding. The authors should state the exact split construction (e.g., disjoint clips, disjoint procedure recordings, or disjoint timepoints) and, ideally, report performance when training and test are separated by recording session.
[Evaluation Procedure; Figure 2] The comparison to generalist models is not controlled. The ORQA model is initialized from Qwen2-VL-2B, but Qwen2-VL-2B is evaluated only zero-shot. No vision-language model of any family is fine-tuned on the same 1M QA pairs (or an equivalent surgical QA set) and then evaluated on the ORQA test set. Without such a fine-tuned generalist baseline, the large margin in Figure 2 could be due to task fine-tuning and template exposure alone, rather than to ORQA's specialized multi-encoder architecture or to a fundamental advantage of domain-specialized training. Adding a Qwen2-VL-2B (and possibly a 7B) model fine-tuned on the ORQA trainingQA pairs under the same training budget is essential to support the claim that domain-specialized multimodal training is required.
[Leave-one-out generalization experiments; Figure 6] The paper's own leave-one-out results weaken the generalization claim. When each dataset is held out during training, zero-shot ORQA performance is described as "moderate" and is far below the full-model score, and only after 100–1000 fine-tuning steps does performance rise. This pattern is consistent with the model having learned dataset- and annotation-schema-specific regularities rather than a holistic, transferable understanding of OR scenes. The authors should temper the claim of "broad generalizability" and instead frame ORQA as a strong within-distribution benchmark model requiring adaptation to new annotation conventions. At minimum, the discussion of Figure 6 should be reconciled with the more general language used elsewhere (e.g., the results section paragraph beginning "Taken together") about generalization to unseen settings.

minor comments (5)

[ORQA Score in Evaluation Procedure] The hand-chosen thresholds in the ORQA Score (e.g., 10%/25% relative distance bands, 0.125–0.75 IoU bands, 0.5 for off-by-one counting) are presented without sensitivity analysis. Since all headline comparisons rely on this scalar metric, a brief robustness check (e.g., varying the thresholds and confirming the ranking of models is unchanged) would strengthen the conclusions.
[Evaluation Procedure and results] The trivial statistical baseline achieves an ORQA Score of 0.191, yet GPT-4o scores 0.180 and Qwen2-VL-2B scores 0.149, below that baseline. This is not impossible, but it is surprising for models with strong vision-language capabilities and deserves a sentence of interpretation to rule out systematic prompt-format mismatch or metric artifacts.
[Question–Answer Generation and Sampling] The paper states that over 100 million QA pairs are generated but only 1 million are kept. It would be helpful to report the distribution of kept pairs across the four datasets and 23 tasks, since Figure 5 reports per-task scores but the reader cannot judge whether performance is driven by a few dominant tasks.
[Figure 3 table] The table in Figure 3 lists ORQA Score confidence intervals but no confidence intervals for throughput (T/s); reporting variance across runs would make the efficiency claims more precise. Also, the table headers "Base," "Temp," and "Dist-*" are not defined in the caption beyond the surrounding text.
[General] The paper repeatedly uses "broad generalizability" and "holistic OR understanding" in the results and discussion, but the evidence is mostly within-distribution (trained and tested on the same four datasets). Suggest using more measured phrasing such as "strong within-benchmark performance" and "moderate transfer to unseen annotation schemas."

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: ORQA's benchmark score is an empirical in-distribution result, and the main weaknesses are experimental-design concerns rather than construction-level circularity.

full rationale

The paper contains no derivation in which a predicted quantity is equivalent by construction to a fitted parameter or to a cited prior result. ORQA is trained on 1M QA pairs sampled from the ORQA benchmark and evaluated on a test split created with the same automated templates; while this makes the headline score an in-distribution measurement, it is not a forced result — the zero-shot generalists score 0.149–0.249 against a 0.191 trivial baseline, whereas ORQA reaches 0.559, so the benchmark can discriminate and the outcome is empirical. The leave-one-out experiments in Figure 6 and the authors' admission that 'zero-shot performance remains moderate, underscoring the challenge of generalizing to unseen annotation conventions' frame the main score as benchmark-specific rather than universal OR competence, but this is a validity caveat, not a circular reduction. Self-citations (MM-OR, 4D-OR, Labrad-OR, ORacle) point to public, externally usable datasets and architectures; none is invoked as a uniqueness theorem or as the sole justification forcing the central result. The missing fine-tuned generalist baseline (Qwen2-VL trained on the same 1M QA pairs) weakens the claim that domain-specialized training is required, but an omitted control is a correctness risk, not circularity. No equation in the paper maps an output back to an input by definition.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on treating the four datasets' annotations as ground truth and treating template-generated QA as a valid measure of OR understanding. The evaluation metric thresholds and distillation temperature are hand-chosen. No new physical entities are introduced.

free parameters (3)

Distillation temperature T = not reported
In Eq. (1), T softens teacher logits; its value affects all distilled model scores and is not stated.
ORQA Score tolerance thresholds = 10% and 25% distance errors; IoU thresholds 0.75, 0.5, 0.25, 0.125
Hand-chosen scoring rules in the Evaluation Procedure determine the headline scores and are not validated against clinical utility.
Diversity sampling inverse-frequency weights = not specified
The 100M to 1M QA pair curation uses inverse question and answer frequency, but the exact weighting and sampling ratios are not given; this affects the training distribution and results.

assumptions (4)

domain assumption The annotations in MVOR, 4D-OR, EgoSurgery, and MM-OR are accurate ground truth for the 23 tasks.
Every QA pair is generated from these annotations; label errors propagate into training and evaluation (Question-Answer Generation and Sampling).
ad hoc to paper Template-generated QA pairs are a valid proxy for clinically meaningful OR understanding.
The benchmark's validity rests on this; no clinician agreement study or real-world task validation is provided.
domain assumption The validation and test QA pairs are disjoint from training pairs at the timepoint level, not just the QA-string level.
The paper says it creates train and test QA pairs but does not state that the underlying timepoints are split, so score inflation from near-duplicate questions is possible.
domain assumption Pretrained encoders (CLIP, Point Transformer V3, CLAP, Qwen2-VL) provide useful representations for OR data.
The model relies on these pretrained components transferring to surgical scenes (Model Architecture).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Specialized Foundation Models for Intelligent Operating Rooms." pith.science (2026). https://pith.science/paper/AXNULDYM

@misc{pith2026250512890,
  author       = {Pith},
  title        = {Pith review of: Specialized Foundation Models for Intelligent Operating Rooms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AXNULDYM}},
  note         = {Machine review of arXiv:2505.12890}
}

read the original abstract

Surgical procedures unfold in complex environments demanding coordination between surgical teams, tools, imaging and increasingly, intelligent robotic systems. Ensuring safety and efficiency in ORs of the future requires intelligent systems, like surgical robots, smart instruments and digital copilots, capable of understanding complex activities and hazards of surgeries. Yet, existing computational approaches, lack the breadth, and generalization needed for comprehensive OR understanding. We introduce ORQA, a multimodal foundation model unifying visual, auditory, and structured data for holistic surgical understanding. ORQA's question-answering framework empowers diverse tasks, serving as an intelligence core for a broad spectrum of surgical technologies. We benchmark ORQA against generalist vision-language models, including ChatGPT and Gemini, and show that while they struggle to perceive surgical scenes, ORQA delivers substantially stronger, consistent performance. Recognizing the extensive range of deployment settings across clinical practice, we design, and release a family of smaller ORQA models tailored to different computational requirements. This work establishes a foundation for the next wave of intelligent surgical solutions, enabling surgical teams and medical technology providers to create smarter and safer operating rooms.

Figures

Figures reproduced from arXiv: 2505.12890 by the authors.

**Figure 1.** Overview of the ORQA benchmark and model. Four OR datasets are unified into a multimodal QA benchmark. Our model encodes samples via modality-specific encoders and processes them with an LLM, optionally leveraging scene graphs (SG) for reasoning, and outputs structured or free-text responses. The increasing integration of digital tools, robotic assistance, and sensor-rich platforms into the OR has dramatically incr… view at source ↗

**Figure 2.** Comparison of the domain-specialized ORQA model with state-of-theart generalist vision-language foundation models on our benchmark test set. We report the ORQA Score along with 95% confidence intervals. ChatGPT and Similar Generalist Models Struggle to Grasp the Operating Room To assess whether existing foundation models can generalize to the OR setting, we evaluate three proprietary state-of-the-art vision-languag… view at source ↗

**Figure 3.** Performance and efficiency trade-offs across ORQA model variants and ablations. The bar chart shows model performance with 95% confidence intervals. The table below details architectural specifications and inference throughput (tokens per second). Distilled variants offer smaller sizes and higher throughput with moderate performance degradation. net sample, all models except Gemini 2.5 Flash and ORQA predict boundi… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Qualitative examples from (left to right) MM-OR, EgoSurgery and a zero-shot image from the internet. For all non ORQA models, we explicitly prompt with possible answers per question and the expected output formats, and manually parse their answers. The ground truth ord…

**Figure 5.** Figure 5: Performance of ORQA across all 23 clinical tasks in the ORQA benchmark. The ORQA Score, reported per task, captures accuracy across diverse challenges, including spatial reasoning, workflow prediction, and safety monitoring. Results demonstrate robust performance and…

**Figure 6.** Figure 6: Leave-one-out generalization experiments. We evaluate how well ORQA and its base model, Qwen2-VL-2B, generalize to unseen datasets by systematically leaving out one dataset during training and evaluating on it. For each held-out dataset, we report four configurations:…

**Figure 7.** Figure 7: Creation of Smaller ORQA Variants through Knowledge Distillation. The original ORQA model, teacher, supervises progressively smaller students, enabling efficient deployment in resource-constrained surgical environments. performance and improved multimodal compatibilit…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 20 canonical work pages

[1]

Carts: Causality-driven robot tool segmentation from vision and kinematics data

Hao Ding, Jintan Zhang, Peter Kazanzides, Jie Ying Wu, and Mathias Unberath. Carts: Causality-driven robot tool segmentation from vision and kinematics data. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VII, pages 387–398. Springer, 2022

work page 2022
[2]

Clap learning audio concepts from natural language supervision

Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

work page 2023
[3]

Egosurgery-phase: a dataset of surgical phase recognition from egocentric open surgery videos

Ryo Fujii, Masashi Hatano, Hideo Saito, and Hiroki Kajita. Egosurgery-phase: a dataset of surgical phase recognition from egocentric open surgery videos. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 187–196. Springer, 2024

work page 2024
[4]

Tri-modal confluence with temporal dynamics for scene graph generation in operating rooms

Diandian Guo, Manxi Lin, Jialun Pei, He Tang, Yueming Jin, and Pheng-Ann Heng. Tri-modal confluence with temporal dynamics for scene graph generation in operating rooms. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 714–724. Springer, 2024

work page 2024
[5]

Pitvqa: Image-grounded text embedding llm for visual question answering in pituitary surgery

Runlong He, Mengya Xu, Adrito Das, Danyal Z Khan, Sophia Bano, Hani J Marcus, Danail Stoyanov, Matthew J Clarkson, and Mobarakol Islam. Pitvqa: Image-grounded text embedding llm for visual question answering in pituitary surgery. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 488–498. Springer, 2024

work page 2024
[6]

Distilling the knowledge in a neural network

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015

arXiv 2015
[7]

Lora: Low-rank adaptation of large language models

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021

arXiv 2021
[8]

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Yueming Jin, Huaxia Li, Qi Dou, Hao Chen, Jing Qin, Chi-Wing Fu, and Pheng- Ann Heng. Multi-task recurrent convolutional network with correlation loss for surgical video analysis. Medical image analysis, 59:101572, 2020

work page 2020

Show all 30 references

[9]

Work-system interventions in robotic- assisted surgery: a systematic review exploring the gap between challenges and solutions

Falisha Kanji, Ken Catchpole, Eunice Choi, Myrtede Alfred, Kate Cohen, Daniel Shouhed, Jennifer Anger, and Tara Cohen. Work-system interventions in robotic- assisted surgery: a systematic review exploring the gap between challenges and solutions. Surgical endoscopy, 35:1976–1989, 2021

1976
[10]

Surgical process modelling: a review

Florent Lalys and Pierre Jannin. Surgical process modelling: a review. Inter- national Journal of Computer Assisted Radiology and Surgery, Springer Verlag , 9:495–511, 2014

2014
[11]

Llava-surg: towards multimodal surgical as- sistant via structured surgical video learning

Jiajie Li, Garrett Skinner, Gene Yang, Brian R Quaranto, Steven D Schwaitzberg, Peter CW Kim, and Jinjun Xiong. Llava-surg: towards multimodal surgical as- sistant via structured surgical video learning. arXiv preprint arXiv:2408.07981 , 2024. 17

2024 arXiv
[12]

A multimodal generative ai copilot for human pathology

Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Melissa Zhao, Aaron K Chow, Kenji Ikemura, et al. A multimodal generative ai copilot for human pathology. Nature, 634(8033):466–473, 2024

2024
[13]

Surgical data science for next- generation interventions

Lena Maier-Hein, Swaroop S Vedula, Stefanie Speidel, Nassir Navab, Ron Kiki- nis, Adrian Park, Matthias Eisenmann, , et al. Surgical data science for next- generation interventions. Nature Biomedical Engineering, 1(9):691–696, 2017

2017
[14]

Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos

Chinedu Innocent Nwoye, Tong Yu, Cristians Gonzalez, Barbara Seeliger, Pietro Mascagni, Didier Mutter, Jacques Marescaux, and Nicolas Padoy. Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos. Medical Image Analysis, 78, 2022

2022
[15]

Labrad-or: Lightweight memory scene graphs for accurate bimodal reasoning in dynamic operating rooms

Ege ¨Ozsoy, Tobias Czempiel, Felix Holm, Chantal Pellegrini, and Nassir Navab. Labrad-or: Lightweight memory scene graphs for accurate bimodal reasoning in dynamic operating rooms. In International Conference on Medical Image Com- puting and Computer-Assisted Intervention . Sp...

2023
[16]

4d-or: Semantic scene graphs for or domain modeling

Ege ¨Ozsoy, Evin Pınar ¨Ornek, Ulrich Eck, Tobias Czempiel, Federico Tombari, and Nassir Navab. 4d-or: Semantic scene graphs for or domain modeling. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 1...

2022
[17]

Mm-or: A large multimodal operating room dataset for semantic understanding of high-intensity surgical environments

Ege ¨Ozsoy, Chantal Pellegrini, Tobias Czempiel, Felix Tristram, Kun Yuan, David Bani-Harouni, Ulrich Eck, Benjamin Busam, Matthias Keicher, and Nassir Navab. Mm-or: A large multimodal operating room dataset for semantic understanding of high-intensity surgical environments. I...

2025
[18]

Oracle: Large vision-language models for knowledge-guided holistic or domain modeling

Ege ¨Ozsoy, Chantal Pellegrini, Matthias Keicher, and Nassir Navab. Oracle: Large vision-language models for knowledge-guided holistic or domain modeling. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 455–465. Springer, 2024

2024
[19]

S 2 former-or: Single-stage bi-modal transformer for scene graph generation in or

Jialun Pei, Diandian Guo, Jingyang Zhang, Manxi Lin, Yueming Jin, and Pheng- Ann Heng. S 2 former-or: Single-stage bi-modal transformer for scene graph generation in or. IEEE Transactions on Medical Imaging , 2024

2024
[20]

Radialog: A large vision-language model for radiology report generation and conversational assistance

Chantal Pellegrini, Ege ¨Ozsoy, Benjamin Busam, Nassir Navab, and Matthias Keicher. Radialog: A large vision-language model for radiology report generation and conversational assistance. arXiv preprint arXiv:2311.18681 , 2023

2023 arXiv
[21]

Development, deployment and scal- ing of operating room-ready artificial intelligence for real-time surgical decision support

Sergey Protserov, Jaryd Hunter, Haochi Zhang, Pouria Mashouri, Caterina Masino, Michael Brudno, and Amin Madani. Development, deployment and scal- ing of operating room-ready artificial intelligence for real-time surgical decision support. NPJ Digital Medicine , 7(1):231, 2024

2024
[22]

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

2021
[23]

Robust speech recognition via large-scale weak supervision

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning , pages 28492–28518. PMLR, 2023. 18

2023
[24]

Mvor: A multi-view rgb-d operating room dataset for 2d and 3d human pose estimation

Vinkle Kumar Srivastav, Thibaut Issenhuth, Abdolrahim Kadkhodamohammadi, Michel de Mathelin, Afshin Gangi, and Nicolas Padoy. Mvor: A multi-view rgb-d operating room dataset for 2d and 3d human pose estimation. In MICCAI 2018 Satellite Workshop, Granada, Spain, september 16-20...

2018
[25]

Endonet: a deep architecture for recognition tasks on laparoscopic videos

Andru P Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy. Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging, 36(1):86–97, 2016

2016
[26]

Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 , 2024

2024 arXiv
[27]

Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data

Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. arXiv preprint arXiv:2308.02463 , 2023

2023 arXiv
[28]

Point transformer v3: Simpler faster stronger

Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler faster stronger. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4840–4851, 2024

2024
[29]

Teleor: Real-time telemedicine system for full-scene operating room

Yixuan Wu, Kaiyuan Hu, Qian Shao, Jintai Chen, Danny Z Chen, and Jian Wu. Teleor: Real-time telemedicine system for full-scene operating room. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 628–638. Springer, 2024

2024
[30]

Advancing surgical vqa with scene graph knowledge

Kun Yuan, Manasi Kattel, Jo¨ el L Lavanchy, Nassir Navab, Vinkle Srivastav, and Nicolas Padoy. Advancing surgical vqa with scene graph knowledge. International Journal of Computer Assisted Radiology and Surgery , 19(7):1409–1417, 2024. 19

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Carts: Causality-driven robot tool segmentation from vision and kinematics data

Hao Ding, Jintan Zhang, Peter Kazanzides, Jie Ying Wu, and Mathias Unberath. Carts: Causality-driven robot tool segmentation from vision and kinematics data. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VII, pages 387–398. Springer, 2022

work page 2022

[2] [2]

Clap learning audio concepts from natural language supervision

Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

work page 2023

[3] [3]

Egosurgery-phase: a dataset of surgical phase recognition from egocentric open surgery videos

Ryo Fujii, Masashi Hatano, Hideo Saito, and Hiroki Kajita. Egosurgery-phase: a dataset of surgical phase recognition from egocentric open surgery videos. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 187–196. Springer, 2024

work page 2024

[4] [4]

Tri-modal confluence with temporal dynamics for scene graph generation in operating rooms

Diandian Guo, Manxi Lin, Jialun Pei, He Tang, Yueming Jin, and Pheng-Ann Heng. Tri-modal confluence with temporal dynamics for scene graph generation in operating rooms. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 714–724. Springer, 2024

work page 2024

[5] [5]

Pitvqa: Image-grounded text embedding llm for visual question answering in pituitary surgery

Runlong He, Mengya Xu, Adrito Das, Danyal Z Khan, Sophia Bano, Hani J Marcus, Danail Stoyanov, Matthew J Clarkson, and Mobarakol Islam. Pitvqa: Image-grounded text embedding llm for visual question answering in pituitary surgery. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 488–498. Springer, 2024

work page 2024

[6] [6]

Distilling the knowledge in a neural network

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015

arXiv 2015

[7] [7]

Lora: Low-rank adaptation of large language models

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021

arXiv 2021

[8] [8]

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Yueming Jin, Huaxia Li, Qi Dou, Hao Chen, Jing Qin, Chi-Wing Fu, and Pheng- Ann Heng. Multi-task recurrent convolutional network with correlation loss for surgical video analysis. Medical image analysis, 59:101572, 2020

work page 2020