{"id":"270f2ba8-40e7-4153-924f-0a8cf532c891","arxiv_id":"2505.12890","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ORQA is a multimodal question-answering benchmark and specialized model family that beats generalist vision-language models on 23 surgical scene understanding tasks.","lead":"This paper introduces ORQA, a question-answering benchmark and model family that lets AI systems understand operating room scenes through images, sound, and sensor data. The specialized model beats general-purpose AI assistants like ChatGPT and Gemini on surgical scene understanding, but only after being trained on surgical data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ORQA's headline margin may reflect train/test overlap and template memorization; the missing fine-tuned generalist baseline leaves the 'domain-specialized training is required' claim unproven.","rationale":"I read the paper as a benchmark-plus-model contribution whose headline is the large margin of ORQA over zero-shot generalists. The margin itself is probably real in the narrow sense: a model fine-tuned on 1M QA pairs from these datasets will beat models that have never seen them. The load-bearing question is whether that margin supports the inference that domain-specialized, multimodal training is required for reliable OR understanding. Two missing controls threaten that inference. The first is dataset hygiene: the paper does not state that test QA pairs come from timepoints disjoint from training, and because all QA pairs are generated by the same templates from the same annotation schemas, the test can be closer to a memorization probe than to a generalization benchmark. The second is baseline fairness: Qwen2-VL-2B, which is the exact backbone of ORQA, is never fine-tuned on the same training pairs, so the comparison cannot separate 'training on surgical QA' from 'ORQA's extra encoders and architecture.' The reader's verdict is CONDITIONAL, and my concern reinforces that conditionality rather than overturning the paper. I would keep the reader's CONDITIONAL verdict, so verdict_should_be is UNCHANGED: the empirical existence proof is useful, but the 'required' interpretation should not be accepted until the split hygiene and fine-tuned baseline checks are reported. The concrete test above would settle both issues. I found no evidence of internal inconsistency or bad faith; the leave-one-out experiments are a genuine attempt to address generalization, and the paper is transparent that code and data will be released on acceptance.","tokens_in":12642,"tokens_out":5969,"duration_ms":64017,"concrete_test":"Release the exact timepoint-level split used to create the train and test QA pairs. Then run two controlled evaluations: (1) retrain ORQA on QA pairs from training timepoints only and evaluate on QA pairs from held-out timepoints within the same four datasets; if the score drops by more than about 0.1 from the reported 0.559, train/test overlap inflates the result; (2) fine-tune Qwen2-VL-2B (image-only) on the same 1M QA pairs and evaluate on the same disjoint test set; if it reaches within 0.05 of ORQA, the margin is attributable to fine-tuning rather than to ORQA's specialized multimodal encoders. Report per-dataset and per-task scores for both checks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that domain-specialized multimodal training on surgical annotation data is required—rests on the margin between ORQA (0.559) and zero-shot generalists. Two unexamined conditions are load-bearing. First, the test set is generated by the same automated templates and, the paper does not state, from timepoints/splits disjoint from the 1M training QA pairs (Method: Question–Answer Generation and Sampling; Evaluation Procedure). If the same recordings or even the same annotation schema appear in both, ORQA can score highly by reproducing dataset-specific labels and answer priors rather than by general OR understanding. The authors' own leave-one-out results (Figure 6) show that zero-shot transfer to a held-out dataset is 'moderate' and far below the full-model score, which is consistent with template/schema memorization rather than holistic generalization. Second, the comparison is not controlled: Qwen2-VL-2B, the exact LLM/vision backbone of ORQA, is evaluated only zero-shot; no Qwen2-VL model is fine-tuned on the same 1M QA pairs. Without that control, the large margin may be due to task fine-tuning alone, not to ORQA's specialized multi-encoder design. Thus the strongest claim's 'required' inference is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ORQA, a multimodal question-answering benchmark and model for operating-room scene understanding. The benchmark is generated from four publicly available OR datasets (MVOR, 4D-OR, EgoSurgery, MM-OR) by applying automated templates to their annotations, yielding 23 tasks and, after diversity sampling, 1 million training QA pairs plus 10,000 validation and 10,000 test pairs. The ORQA model extends Qwen2-VL with encoders for RGB, point clouds, audio, speech, robot logs, and tracking data, and is trained on the 1M QA pairs. The paper reports that ORQA achieves an ORQA Score of 0.559 (95% CI 0.551–0.567), far above zero-shot GPT-4o, GPT-4.1, Gemini 2.5 Flash, and Qwen2-VL (2B/7B), and introduces distilled variants with lower parameter counts and higher throughput. The central claim is that domain-specialized multimodal training on surgical annotation data is required for reliable OR scene understanding, whereas generalist vision-language models are inadequate.","tokens_in":12926,"tokens_out":2600,"duration_ms":28769,"significance":"If the central comparison were clean, ORQA would be a valuable contribution: it unifies four disparate surgical datasets into a single QA benchmark, provides a reproducible evaluation protocol, and demonstrates that a moderately sized multimodal model can handle a wide range of OR perception tasks with structured outputs. The planned public release of the benchmark, model, and evaluation code is a clear strength, as is the inclusion of leave-one-out transfer experiments and distilled deployment variants. However, the significance is substantially contingent on resolving the evaluation asymmetry: the benchmark serves simultaneously as training and test data, and the strongest result—the large margin over generalists—may reflect template memorization or task fine-tuning rather than domain-specific architectural design. As written, the paper does not yet establish that domain-specialized training is *required*.","major_comments":[{"comment":"The paper does not describe any guarantee that the 10,000 test QA pairs are disjoint from the 1 million training QA pairs at the level of timepoints, clips, or recordings. The training pairs are generated from the training splits of the four datasets, and the test pairs are generated \"analogously,\" but no temporal or scene-level split is specified. Because all QA pairs are produced by the same automated templates from the same annotation schemas, an ORQA model trained on 1M template-generated pairs could score highly by reproducing dataset-specific label distributions and template answer priors rather than by general OR understanding. The authors should state the exact split construction (e.g., disjoint clips, disjoint procedure recordings, or disjoint timepoints) and, ideally, report performance when training and test are separated by recording session.","section":"Question–Answer Generation and Sampling; Evaluation Procedure"},{"comment":"The comparison to generalist models is not controlled. The ORQA model is initialized from Qwen2-VL-2B, but Qwen2-VL-2B is evaluated only zero-shot. No vision-language model of any family is fine-tuned on the same 1M QA pairs (or an equivalent surgical QA set) and then evaluated on the ORQA test set. Without such a fine-tuned generalist baseline, the large margin in Figure 2 could be due to task fine-tuning and template exposure alone, rather than to ORQA's specialized multi-encoder architecture or to a fundamental advantage of domain-specialized training. Adding a Qwen2-VL-2B (and possibly a 7B) model fine-tuned on the ORQA trainingQA pairs under the same training budget is essential to support the claim that domain-specialized multimodal training is required.","section":"Evaluation Procedure; Figure 2"},{"comment":"The paper's own leave-one-out results weaken the generalization claim. When each dataset is held out during training, zero-shot ORQA performance is described as \"moderate\" and is far below the full-model score, and only after 100–1000 fine-tuning steps does performance rise. This pattern is consistent with the model having learned dataset- and annotation-schema-specific regularities rather than a holistic, transferable understanding of OR scenes. The authors should temper the claim of \"broad generalizability\" and instead frame ORQA as a strong within-distribution benchmark model requiring adaptation to new annotation conventions. At minimum, the discussion of Figure 6 should be reconciled with the more general language used elsewhere (e.g., the results section paragraph beginning \"Taken together\") about generalization to unseen settings.","section":"Leave-one-out generalization experiments; Figure 6"}],"minor_comments":[{"comment":"The hand-chosen thresholds in the ORQA Score (e.g., 10%/25% relative distance bands, 0.125–0.75 IoU bands, 0.5 for off-by-one counting) are presented without sensitivity analysis. Since all headline comparisons rely on this scalar metric, a brief robustness check (e.g., varying the thresholds and confirming the ranking of models is unchanged) would strengthen the conclusions.","section":"ORQA Score in Evaluation Procedure"},{"comment":"The trivial statistical baseline achieves an ORQA Score of 0.191, yet GPT-4o scores 0.180 and Qwen2-VL-2B scores 0.149, below that baseline. This is not impossible, but it is surprising for models with strong vision-language capabilities and deserves a sentence of interpretation to rule out systematic prompt-format mismatch or metric artifacts.","section":"Evaluation Procedure and results"},{"comment":"The paper states that over 100 million QA pairs are generated but only 1 million are kept. It would be helpful to report the distribution of kept pairs across the four datasets and 23 tasks, since Figure 5 reports per-task scores but the reader cannot judge whether performance is driven by a few dominant tasks.","section":"Question–Answer Generation and Sampling"},{"comment":"The table in Figure 3 lists ORQA Score confidence intervals but no confidence intervals for throughput (T/s); reporting variance across runs would make the efficiency claims more precise. Also, the table headers \"Base,\" \"Temp,\" and \"Dist-*\" are not defined in the caption beyond the surrounding text.","section":"Figure 3 table"},{"comment":"The paper repeatedly uses \"broad generalizability\" and \"holistic OR understanding\" in the results and discussion, but the evidence is mostly within-distribution (trained and tested on the same four datasets). Suggest using more measured phrasing such as \"strong within-benchmark performance\" and \"moderate transfer to unseen annotation schemas.\"","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the resource (benchmark + model) is potentially useful, but the central comparative claim is currently under-supported. The missing fine-tuned generalist baseline and the lack of an explicit train/test disjointness guarantee are both fixable with additional experiments that do not require re-framing the entire contribution. I would condition acceptance on (a) a controlled fine-tuning baseline, (b) a clear statement and verification of temporal/session-level splitting, and (c) a revised generalization narrative that aligns with the leave-one-out results. The hand-chosen score thresholds are a lesser concern but should be acknowledged with sensitivity analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: ORQA is a real benchmark-and-model contribution to surgical AI, but the headline margin over generalist VLMs is confounded by the fact that the model trains on the same benchmark it is tested on. Worth a careful referee, not a desk reject.\n\nWhat's genuinely new: The paper unifies four public OR datasets (MVOR, 4D-OR, EgoSurgery, MM-OR) into one multimodal QA benchmark spanning 23 tasks, generating 100M QA pairs and curating 1M for training plus 10k each for validation and test. That unification alone is useful—the field is fragmented by dataset- and task-specific models. The model is an honest assembly of proven parts: Qwen2-VL backbone, Point Transformer V3, CLAP, and the authors' own MM2SG scene-graph architecture. The distilled variants (3.3x speedup for ~0.03 score drop) are a practical plus. The leave-one-out experiments are the right instinct and show that adaptation to a new dataset works better than zero-shot, even if zero-shot transfer is only moderate.\n\nThe soft spots, in proportion. The main one is experimental control. ORQA trains on 1M QA pairs from the same four datasets and annotation schemas that define the test set; the generalist baselines are all zero-shot. There is no fine-tuned generalist baseline—no Qwen2-VL-2B trained on those same pairs. Without that, the margin between 0.559 and 0.249 could be mostly task fine-tuning on the benchmark itself, not ORQA's multimodal architecture or domain specialization. The paper also never says the test timepoints are disjoint from training timepoints, which is a leakage risk. The ORQA Score is a reasonable composite but uses hand-picked thresholds, so it is not a neutral yardstick.\n\nNone of this kills the contribution. The benchmark is a credible evaluation substrate even if the model claim needs qualification. The authors are candid about moderate cross-dataset transfer, which actually supports the worry about schema memorization. The heavy self-citation is justified: the model builds directly on their MM2SG and 4D-OR work.\n\nWho this is for: anyone working in surgical scene understanding or OR monitoring. Read it for the benchmark; treat the model results as promising but confounded.\n\nRecommendation: send to peer review, but require a fine-tuned generalist baseline, a clear statement on train/test timepoint disjointness, and release of benchmark and code with the revision. That would move it from conditionally acceptable to solid.","headline":"A useful surgical QA benchmark undermined by a missing control: the model trains on the same benchmark it tests on, so the claim that specialized training is required is not yet established.","tokens_in":13435,"tokens_out":3703,"would_cite":true,"duration_ms":35058,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ORQA model beats generalist chatbots at reading the OR","keywords":["operating room understanding","multimodal question answering","surgical scene understanding","vision-language foundation models","domain-specialized training","knowledge distillation","scene graphs","surgical safety"],"falsifier":"Fine-tune a comparable generalist vision-language model on the same one million ORQA training pairs and evaluate it on the ORQA test set; if it reaches ORQA's 0.559 score without the OR-specific encoders or scene-graph machinery, the claim that domain-specialized surgical training is required would collapse.","tokens_in":12474,"feed_emoji":"🩺","tokens_out":6994,"duration_ms":67231,"temperature":0.7,"pith_summary":"The paper introduces ORQA, a benchmark and a family of models for holistic operating-room (OR) understanding, and argues that this kind of understanding cannot be achieved by general-purpose vision-language models alone. The evidence is a head-to-head comparison: on the ORQA test set, the specialized model scores 0.559, while GPT-4o, GPT-4.1, Gemini 2.5 Flash, and Qwen2-VL score between 0.149 and 0.249, hovering around a trivial statistical baseline of 0.191. The authors' claim is that training a multimodal language model on one million question-answer pairs generated from four public OR datasets yields reliable spatial, temporal, and safety-related reasoning about surgical scenes, and that distilled smaller versions remain accurate enough for real-time use. If true, this points to domain-specialized training as a necessary ingredient for intelligent OR systems.","feed_headline":"ORQA model beats generalist chatbots at reading the OR","feed_subtitle":"Trained on 1M OR question-answer pairs, ORQA scores 0.559 versus 0.249 for the best generalist model.","key_machinery":"The central object is the ORQA benchmark and model, which together recast 23 surgical scene-understanding tasks as a single question-answering framework. Questions are generated by template from the annotations of four public OR datasets, then diversity-sampled to one million training pairs, with the same procedure yielding 10,000 validation and 10,000 test pairs. The model is a multimodal LLM built on the Qwen2-VL backbone, with encoders for RGB images, point clouds, audio, speech transcripts, robot logs, and tracker data, plus an optional memory of prior scene graphs for temporal reasoning; the ORQA Score metric gives task-specific partial credit, and performance is compared against generalist vision-language models in a zero-shot setting.","core_discovery":"ORQA (Operating Room Question Answering) establishes that a domain-specialized multimodal foundation model can read an operating room in a way that generalist chatbots cannot. Trained on one million curated QA pairs drawn from MVOR, 4D-OR, EgoSurgery, and MM-OR, the model reaches an ORQA Score of 0.559 (95% CI 0.551–0.567), far above the best generalist baseline, GPT-4.1 at 0.249, and above the 0.191 trivial baseline that guesses the most frequent answer per task. The model integrates RGB images, point clouds, audio, speech, robot logs, and tracking data through modality-specific encoders feeding a language model, with optional scene-graph reasoning, and the paper reports strong performance across all 23 tasks, including tool localization, 3D distance estimation, sterility breach detection, gaze-object detection, and next-robot-step prediction. The paper further shows that knowledge-distilled variants, down to 278M parameters, keep scores near 0.525 while running more than three times faster, supporting local real-time deployment.","pith_inferences":["Because the QA pairs are generated from each dataset's own annotation schemas, the benchmark may reward reproducing those labels rather than genuine clinical reasoning; a generalist model fine-tuned on the same pairs might close much of the reported gap.","The ORQA Score's partial-credit rules and BLEU-based OCR scoring can inflate apparent competence, so the headline margin may narrow under stricter exact-match evaluation.","A natural next experiment is a prospective test on real, unannotated OR videos from a new hospital; the paper's leave-one-out results suggest zero-shot transfer to unseen annotation conventions remains moderate.","The 23 tasks are largely perception and state recognition, so the path from ORQA's scene understanding to improved patient outcomes or surgical safety is not directly demonstrated by this paper."],"forward_implications":["Generalist vision-language models, as currently trained, cannot be relied on for surgical scene understanding even with careful prompting, so OR-specific training data is needed for this domain.","ORQA provides a single, interpretable evaluation format for comparing OR perception models across modalities, datasets, and task types.","The distilled ORQA variants make local, real-time, and privacy-preserving intraoperative assistance feasible on resource-constrained hardware.","Leave-one-out experiments suggest that adapting ORQA to a new OR dataset requires only light fine-tuning, not full retraining.","The QA-template framework is extensible to new modalities and clinical tasks, allowing the benchmark and model to grow with new OR technologies."],"supporting_citations":[{"why":"Supplies the real-OR multi-view RGB-D pose and phase annotations used for people-counting, role, and temporal tasks.","marker":"[24]"},{"why":"Supplies simulated knee-replacement scene graphs and panoptic annotations that ground interaction and entity-relation questions.","marker":"[16]"},{"why":"Supplies egocentric video, gaze, tool bounding boxes, and phase labels for gaze and tool-localization tasks.","marker":"[3]"},{"why":"Supplies the multimodal MM-OR data (audio, speech, robot logs, tracking, scene graphs) and the MM2SG architecture that ORQA extends.","marker":"[17]"},{"why":"Provides the Qwen2-VL language model and vision backbone that ORQA initializes from and compares against.","marker":"[26]"},{"why":"Provides the synthetic data augmentation, image-summarizer components, and scene-graph reasoning ideas used in training.","marker":"[18]"},{"why":"Supplies the knowledge-distillation method used to create the smaller real-time ORQA variants.","marker":"[6]"}],"fun_headline_variants":["ORQA outreads GPT-4.1 in surgical scenes","Specialized OR model beats generalist chatbots","ORQA: The AI that truly understands surgery","Surgical foundation model surpasses generalists"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automatically generated QA pairs, derived from the annotation schemas of the four datasets, are a valid measure of clinically meaningful surgical scene understanding.","fun_headline_variants_meta":{"raw":{"variants":["ORQA outreads GPT-4.1 in surgical scenes","Specialized OR model beats generalist chatbots","ORQA: The AI that truly understands surgery","Surgical foundation model surpasses generalists"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1276,"prompt_tokens":968,"completion_tokens":308,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":248}},"tokens_in":584,"tokens_out":308,"duration_ms":4277,"temperature":1.0,"reasoning_tokens":248,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:24:03.791060+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune a comparable generalist vision-language model on the same one million ORQA training pairs and evaluate it on the ORQA test set; if it reaches ORQA's 0.559 score without the OR-specific encoders or scene-graph machinery, the claim that domain-specialized surgical training is required would collapse.","supporting_citations":[{"cited_title":"Mvor: A multi-view rgb-d operating room dataset for 2d and 3d human pose estimation","cited_arxiv_id":null,"evidence_quote":"Supplies the real-OR multi-view RGB-D pose and phase annotations used for people-counting, role, and temporal tasks."},{"cited_title":"4d-or: Semantic scene graphs for or domain modeling","cited_arxiv_id":null,"evidence_quote":"Supplies simulated knee-replacement scene graphs and panoptic annotations that ground interaction and entity-relation questions."},{"cited_title":"Egosurgery-phase: a dataset of surgical phase recognition from egocentric open surgery videos","cited_arxiv_id":null,"evidence_quote":"Supplies egocentric video, gaze, tool bounding boxes, and phase labels for gaze and tool-localization tasks."},{"cited_title":"Mm-or: A large multimodal operating room dataset for semantic understanding of high-intensity surgical environments","cited_arxiv_id":null,"evidence_quote":"Supplies the multimodal MM-OR data (audio, speech, robot logs, tracking, scene graphs) and the MM2SG architecture that ORQA extends."},{"cited_title":"Oracle: Large vision-language models for knowledge-guided holistic or domain modeling","cited_arxiv_id":null,"evidence_quote":"Provides the synthetic data augmentation, image-summarizer components, and scene-graph reasoning ideas used in training."}],"review_version":1}