{"id":"cb33a359-72a6-4d8e-8243-fab3cc32a50f","arxiv_id":"2611.11437","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"General-purpose MLLMs show limited surgical understanding, especially on relational and procedural tasks, and tool-augmented agents improve perception-heavy tasks but fail on fine-grained localization and expert assessment.","lead":"This paper introduces SurgKnowBench, a benchmark that evaluates how well general-purpose multimodal AI models understand surgical scenes across seven procedures and seven tasks without any training. It then tests whether equipping these models with visual perception and knowledge retrieval tools, called SurgAgentGym, can improve their surgical understanding at inference time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"No significant objection identified.","rationale":"The reader identified the perceive() tool's quality as the weakest assumption. While this is a real challenge, the paper acknowledges it explicitly, quantifies its impact (90-100% no-proposal rates for some models), and addresses it through matched comparisons that separate tool access from the agent setting. The central claim does not depend on the tool being perfect; it depends on the pattern of task-specific improvement, which holds even with the acknowledged tool failures. Therefore, I find no load-bearing concern that would alter the verdict.","tokens_in":57398,"tokens_out":2962,"duration_ms":31999,"concrete_test":"Re-run the matched tool comparison (Tab. 3 Panel B) on SURGKNOWBENCH-Hard using SurgSAM-2 instead of SAM2 as the perceive() backbone, with all other settings identical. If the task-specific pattern (perception gains, workflow mixed) persists, the central claim is reinforced; if it changes substantially, the original conclusion is tool-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper conducts an honest, thorough evaluation with appropriate controls (matched comparison, ablation, trajectory analysis) and clearly acknowledges its limitations, including the perceive() tool's failures and the knowledge base's bounded coverage. The central claim—that tool-mediated generalist agents improve perception-heavy tasks but not fine-grained localization or expert assessment—is well-supported by the evidence, and the authors do not overstate it. I find no internal inconsistency or unwarranted inference that would threaten the conclusion.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents a two-stage evaluation framework for surgical scene understanding with general-purpose multimodal large language models (MLLMs). Stage I introduces SurgKnowBench, a diagnostic benchmark that harmonizes 11 public surgical datasets across 7 procedures and 7 tasks, and evaluates 17 proprietary and open-source MLLMs in a zero-shot setting. The results show that current models retain partial visual priors but are weak on relational, procedural, and workflow assessment tasks, often collapsing to majority responses or exhibiting optimistic over-crediting. Stage II introduces SurgAgentGym, an agentic environment where generalist MLLM agents have access to a visual grounding tool (perceive(), based on SAM2) and a knowledge retrieval tool (retrieve(), over SurgWiki, a curated multimodal surgical knowledge base). On a hard subset of SurgKnowBench constructed from episodes where all Stage I models failed, the paper evaluates whether tool-mediated inference can close the gap without additional training. Results show task-specific gains: tool detection and action recognition improve for most open-source models (e.g., +27.6 macro-F1 for Pixtral-12B on action recognition), while fine-grained localization and expert-style rubric assessment (CVS grading, skill assessment) show mixed or negligible improvements. Trajectory analysis identifies bottlenecks at three stages: query formulation, evidence integration, and evidence grounding.","tokens_in":57542,"tokens_out":3954,"duration_ms":35589,"significance":"If the empirical findings hold, this paper makes several valuable contributions. (1) SurgKnowBench provides a harmonized, multi-procedure, multi-task evaluation suite that will help standardize future surgical scene understanding research; the dataset aggregation and unit conversion are done carefully with explicit metadata. (2) The two-stage evaluation design (direct inference then agentic inference) is a useful methodological contribution for disentangling pretrained competence from tool-mediated reasoning. (3) The trajectory analysis, including the failure-mode decomposition (wrong queries, ignoring retrieved evidence, robustness to noisy evidence), gives concrete targets for future work on agentic surgical AI. (4) The paper is unusually honest about failures: it reports that perceive() returns no usable proposals in 90-100% of trajectories for proprietary models on action recognition, that the best action-recognition score (27.6%) remains below the majority baseline (30.0%), and that tool access does not help subjective rubric-based assessment. This candor strengthens the paper's credibility.","major_comments":[{"comment":"The matched tool comparison includes only 5 of the 16 backbones (InternVL2.5-8B, Qwen2.5-VL-7B-Instruct, Qwen3-VL-8B-Instruct, Gemma-3-27B-it, Lingshu-32B) with no stated selection criteria. These five do not, for example, include the best-performing model (Qwen3-VL-32B-Instruct) or any proprietary model. The claim that 'tool access benefits perception-heavy tasks' would be stronger if the matched comparison covered a broader, more representative set of backbones or if the selection rationale (e.g., spanning scale tiers and medical specialization) were stated. As it stands, the generalizability of the matched-comparison results to the full model set is unclear.","section":"Table 3, Panel B (matched tool comparison); §4.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The across-stage comparison (Panel A) contrasts Stage I direct answers with Stage II agent answers, but the two stages differ not only in tool availability but also in prompts, system instructions, and potentially reasoning mode. The paper acknowledges this by including the matched comparison (Panel B), but the main narrative (Abstract: 'agentic inference narrows the gap mainly for perception-heavy tasks'; Conclusion: 'Agentic inference improves some tasks') primarily references the across-stage changes. To avoid conflating confounds, the paper should more explicitly state which claim relies on which comparison and note that the Panel A deltas are a composite of tool access, prompt engineering, and reasoning-mode changes, whereas the pure tool contribution is isolated in Panel B.","section":"§4.2 / Table 3, Panel A; Conclusion and Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The knowledge base includes case records from the same datasets used for evaluation. The paper states that only training-side annotations are included and that evaluation labels are excluded from SurgWiki. This separation is critical to avoid label leakage. The paper should explicitly confirm that no episode in the SurgKnowBench evaluation set shares a video identifier or frame timestamp with a case-record chunk in SurgWiki, and that the procedure for excluding evaluation labels has been verified (e.g., by checking that the hard-set episodes' ground-truth labels do not appear in any case-record text). A brief description of this verification would resolve a potential concern about information leakage.","section":"Appendix D.3.2 / §3.1 (SurgWiki construction)","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The figure panels, particularly the 'Skill Assessment' subplot, overlay multiple model lines that are hard to distinguish in grayscale or print. Consider adding distinct marker shapes or a more detailed legend that maps colors to model names, as the current line-only encoding is difficult to read.","section":"Fig. 2 (line plots)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper states 'overstating the surgical skills' in reference to GPT-4o-mini and Lingshu-32B. Consider rephrasing to 'overstating surgical competence' or 'over-crediting' (the term used later) for terminological consistency with the rest of the section.","section":"§2.3, paragraph on CVS assessment","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The tool specification says 'repeating the same (frame, granularity) is redundant' and that duplicate calls are rejected. The paper does not specify how duplicate rejection is communicated to the agent — whether via an error message, a silent no-op, or a warning in the observation. Clarifying this would help reproducibility.","section":"§3.2 / Appendix D.1 (perceive tool constraints)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The PanelError definition uses range-normalized agreement for ordinal tasks. It should state explicitly whether the average is over criteria within an episode (macro) or over all criterion-episode pairs, as the choice affects the difficulty distribution.","section":"Appendix C.4, Eq. (1)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The expert–expert linearly-weighted κ is 0.460, indicating only moderate agreement. The paper notes that the LLM judge agrees with each expert at least as well as the experts agree with each other. While this is factually stated, it would be helpful to discuss the implications of moderate expert agreement for the reliability of the retrieval-quality ground truth, which the paper uses as a gold standard for the AUROC calculation.","section":"Appendix D.3.5 (retrieval quality estimate)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Several tables in the appendix (e.g., Table 8 onward) use 'Bold/ underline mark the best / second-best per column among models' but the formatting is not visible in plain-text PDF. Ensure that the formatting (bold and underline) renders correctly in the camera-ready version.","section":"Throughout","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper does what a good benchmark paper should: it harmonizes 11 surgical datasets into a single evaluation protocol, tests 17 MLLMs across 7 tasks, and honestly reports that zero-shot performance is weak on relational, procedural, and assessment tasks. The Stage I diagnostic is the strongest contribution here—it gives the community a clear picture of where general-purpose models stand on surgical scene understanding.\n\nThe Stage II agentic environment (SurgAgentGym) is a reasonable next step, but its conclusions are softer. The perceive() tool, built on SAM2 with relaxed thresholds, returns no usable proposals in 90-100% of action-recognition trajectories for GPT-4o-mini and Claude-Haiku. That means the agentic evaluation for those models on that task is essentially just the retrieve() tool, which also shows limited precision. The paper acknowledges this, which I appreciate, but it means the headline claim that 'agentic inference narrows the gap mainly for perception-heavy tasks' rests on a handful of open-source models where perceive() works passably. The matched tool comparison (Panel B) helps, but it covers only five backbones.\n\nOther concerns are minor. The hard-set construction uses a panel of open-source MLLMs, which could introduce blind spots, though the reported flip rate of 0.32% suggests stability. The retrieval quality estimate relies on an LLM judge calibrated on only 50 human-annotated calls, with moderate expert-expert agreement (κ=0.46). And the knowledge base includes case records from the same datasets as evaluation episodes (training splits), which is fine in principle but not checked for near-duplicate visual patterns.\n\nThese are not fatal. The paper is honestly written, the experimental design is sound, and the central takeaway—that current models cannot handle relational or procedural surgical tasks even with tool support—is well-evidenced. The limitations are clearly stated.\n\nVerdict: this deserves peer review. A minor revision should re-center the claims about agentic inference, noting that the perception tool's failure for proprietary models weakens the cross-model generalizability of the Stage II results. I'd bring this to a reading group and would cite it for the benchmark.","headline":"A thorough and honest benchmark for surgical MLLM evaluation; the agentic stage is less conclusive due to tool failures but the diagnosis of current model limitations is solid.","tokens_in":58073,"tokens_out":1559,"duration_ms":28290},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"General-purpose MLLMs, even with visual and knowledge tools, fail at fine-grained localization and expert-level surgical assessment, with best action-recognition score (27.6% macro-F1) still below the majority baseline (30.0%).","keywords":["surgical scene understanding","multimodal large language model","benchmark","agentic inference","tool use","knowledge retrieval","epistemic control","visual grounding"],"falsifier":"If a generalist agent with the same tools were to achieve expert-level performance on fine-grained localization (e.g., >60% F1@0.5) and rubric-based CVS assessment (e.g., quadratic-weighted κ > 0.6) on the same SurgKnowBench-Hard set, that would refute the paper's claim that these tasks are persistent bottlenecks for tool-augmented generalist agents. Alternatively, replacing the perceive() tool with a surgical-domain-specific segmentation model (e.g., surgically-adapted SAM) and the retrieve() backend with an oracle that always returns the single most relevant ground-truth knowledge chunk; if the agent still fails on localization and assessment, then the bottleneck is epistemic control, confirming the paper's thesis.","tokens_in":58875,"feed_emoji":"🔬","tokens_out":5165,"duration_ms":44841,"temperature":0.7,"pith_summary":"This paper argues that current general-purpose multimodal large language models (MLLMs) have only superficial understanding of surgical scenes, and that equipping them with visual grounding and knowledge retrieval tools does not fix their core deficiencies in fine-grained spatial localization and expert-style assessment. The authors introduce a two-stage benchmark: SurgKnowBench tests 17 MLLMs zero-shot across 7 surgical tasks, revealing that models can recognize instruments but fail at relational, procedural, and safety judgments. In the second stage, SurgAgentGym tests whether tool-mediated agents can improve at inference time; they find gains in perception-heavy tasks like tool detection and action recognition, but the highest action-recognition score (27.6% macro-F1) is still below the majority baseline (30.0%). Agent trajectory analysis shows that agents frequently formulate wrong queries, retrieve irrelevant evidence, or ignore useful evidence, making epistemic control the binding constraint. The paper concludes that generalist agents cannot achieve surgical competence through tool use alone without better metacognitive capabilities.","feed_headline":"Tool-aided agents still fail fine-grained surgical tasks","feed_subtitle":"Benchmark shows agentic inference narrows perception gaps but fine-grained localization and expert assessment remain unsolved.","key_machinery":"The argument is carried by a two-stage evaluation framework: Stage I uses SurgKnowBench, a diagnostic benchmark that harmonizes 11 public surgical datasets (376,602 samples across 7 procedures and 7 tasks) into a unified QA format to map the boundary of pretrained MLLM surgical competence. Stage II uses SurgAgentGym, an agentic environment where MLLM agents wield a perceive() tool (SAM2-based class-agnostic region proposals) and a retrieve() tool (hybrid lexical-dense search over SurgWiki, a curated multimodal knowledge base of 74,315 chunks from textbooks, guidelines, tutorial videos, and case records). The key evaluative mechanism is trajectory-level analysis that tracks whether the agent recognizes an information need, selects appropriate tools, retrieves relevant evidence, and grounds it in the surgical input, exposing three failure modes: poor query formulation, low robustness to imperfect evidence, and inability to integrate retrieved evidence when it is relevant.","core_discovery":"The central discovery is that general-purpose MLLMs, while retaining partial visual priors for surgical instruments, are unable to reason about surgical interactions (triplet detection below 5% macro-F1), procedural phase (accuracy below majority baseline on most datasets), or expert-level workflow assessment (CVS and skill assessment show near-chance or negative agreement with expert rubrics). When deployed as agents with access to a regulated surgical knowledge base and a general-purpose visual grounding tool, they improve on perception-heavy tasks (tool detection, action recognition) but remain unable to localize tools precisely or make rubric-consistent judgments. The binding failure is not the absence of domain knowledge but the agents' inability to formulate effective search queries, reconcile imperfect evidence, and ground retrieved knowledge in the surgical scene. This work establishes that epistemic control—the metacognitive ability to seek, evaluate, and integrate evidence—is the missing ingredient for generalist surgical understanding, and that current tool-mediated inference is insufficient to supply it.","pith_inferences":["The fact that retrieval hurt performance in the medical-domain model Lingshu-32B suggests that domain-specific pretraining may already embed knowledge that is more reliable than retrieved evidence; a hybrid strategy that selectively invokes retrieval only when model confidence is low could outperform full tool access.","The reiteration of the same three failure modes (query quality, robustness, grounding) across diverse backbones hints that epistemic control is a general limitation of transformer-based autoregressive generation as a reasoning substrate for open-world medical tasks, not an artifact of any single model family.","If the trajectory-level analysis is replicated, it could inform a new class of training objectives that explicitly reward the agent for improving its query quality and evidence-use consistency, rather than just final answer accuracy."],"forward_implications":["Any attempt to build surgical superintelligence from general-purpose MLLMs alone must first solve the epistemic control problem; simply adding more domain data at inference time via retrieval will not suffice if the agent cannot formulate good queries or integrate conflicting evidence.","Surgical AI benchmarks that only measure final-answer accuracy may be misleading; trajectory-level analysis is necessary to distinguish cases where the agent retrieved the right evidence but ignored it from cases where retrieval failed outright.","The finding that tool access can actually degrade performance in proprietary models (GPT-4o-mini and Claude-Haiku-4.5 declined on action recognition) suggests that tool-use policies must be learned, not just given, and that naive tool access can backfire.","Persistent failure on fine-grained localization and expert-style assessment points to fundamental limitations in current vision encoders and reasoning chains that no amount of external knowledge can bypass without architectural changes."],"supporting_citations":[{"why":"Supplies the triplet-detection task and tool-localization annotations for the CholecT50 dataset; baseline for these axes.","marker":"[1]"},{"why":"Supplies the Cholec80 dataset used for phase recognition; one of the largest surgical video datasets.","marker":"[5]"},{"why":"Supplies the HeiChole dataset for action recognition and skill assessment; provides the skill-rubric baseline.","marker":"[2]"},{"why":"Supplies the Cholec80-CVS dataset with expert CVS annotations; defines the CVS assessment task.","marker":"[8]"},{"why":"Supplies the GraSP dataset for tool detection and phase recognition in robotic prostatectomy.","marker":"[12]"},{"why":"Supplies the PitVis dataset for phase recognition and tool detection in endoscopic pituitary surgery.","marker":"[13]"},{"why":"Provides the SAM2 model used as the perceive() visual grounding tool in SurgAgentGym.","marker":"[32]"},{"why":"Provides jina-embeddings-v4, the multimodal embedding model used for dense retrieval over SurgWiki.","marker":"[54]"},{"why":"Provides the jina-reranker-v2 cross-encoder used to rescore retrieval candidates in the SurgWiki backend.","marker":"[57]"}],"fun_headline_variants":["Agents boost surgical perception but fail at fine localization","Tool-aided MLLMs: perception narrows, expert judgment stalls","Surgical agent benchmark: evidence use fails where precision counts"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper assumes that the SAM2-based perceive() tool with general-purpose thresholds (pred_iou_thresh=0.70, stability_score_thresh=0.90) provides region proposals usable enough for agents to improve surgical understanding, but this tool returns no usable region proposals in 90-100% of action-recognition trajectories for GPT-4o-mini and Claude-Haiku-4.5, and the retrieval relevance score (AUROC 0.817 against experts) still leaves many calls with low-quality results that depend heavily on agent query formulation.","fun_headline_variants_meta":{"raw":{"variants":["Agents boost surgical perception but fail at fine localization","Tool-aided MLLMs: perception narrows, expert judgment stalls","Surgical agent benchmark: evidence use fails where precision counts"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"cost":0.000189588,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000189588,"upstream_inference_prompt_cost":0.000103348,"upstream_inference_completions_cost":8.624e-05},"total_tokens":1251,"prompt_tokens":943,"completion_tokens":308,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":252}},"tokens_in":943,"tokens_out":308,"cost_source":"openrouter_usage","duration_ms":8438,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":252,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-10T18:21:47.020947+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If a generalist agent with the same tools were to achieve expert-level performance on fine-grained localization (e.g., >60% F1@0.5) and rubric-based CVS assessment (e.g., quadratic-weighted κ > 0.6) on the same SurgKnowBench-Hard set, that would refute the paper's claim that these tasks are persistent bottlenecks for tool-augmented generalist agents. Alternatively, replacing the perceive() tool with a surgical-domain-specific segmentation model (e.g., surgically-adapted SAM) and the retrieve() backend with an oracle that always returns the single most relevant ground-truth knowledge chunk; if the agent still fails on localization and assessment, then the bottleneck is epistemic control, confirming the paper's thesis.","supporting_citations":[{"cited_title":"Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos","cited_arxiv_id":null,"evidence_quote":"Supplies the triplet-detection task and tool-localization annotations for the CholecT50 dataset; baseline for these axes."},{"cited_title":"EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos","cited_arxiv_id":null,"evidence_quote":"Supplies the Cholec80 dataset used for phase recognition; one of the largest surgical video datasets."},{"cited_title":"Lubotsky, Benjamin Müller, Tornike Davitashvili, Manuela Capek, et al","cited_arxiv_id":null,"evidence_quote":"Supplies the HeiChole dataset for action recognition and skill assessment; provides the skill-rubric baseline."},{"cited_title":"Cholec80-CVS: An open dataset with an evaluation of Strasberg’s critical view of safety for AI.Scientific Data, 10(1): 194, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the Cholec80-CVS dataset with expert CVS annotations; defines the CVS assessment task."},{"cited_title":"Pixel-wise recognition for holistic surgical scene understanding","cited_arxiv_id":null,"evidence_quote":"Supplies the GraSP dataset for tool detection and phase recognition in robotic prostatectomy."},{"cited_title":"PitVis-2023 challenge: Workflow recognition in videos of endoscopic pituitary surgery.Medical Image Analysis, page 103716, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the PitVis dataset for phase recognition and tool detection in endoscopic pituitary surgery."},{"cited_title":"SAM 2: Segment anything in images and videos","cited_arxiv_id":null,"evidence_quote":"Provides the SAM2 model used as the perceive() visual grounding tool in SurgAgentGym."},{"cited_title":"Mazomenos, and Yueming Jin","cited_arxiv_id":null,"evidence_quote":"Provides jina-embeddings-v4, the multimodal embedding model used for dense retrieval over SurgWiki."},{"cited_title":"jina-embeddings- v4: Universal embeddings for multimodal multilingual retrieval","cited_arxiv_id":null,"evidence_quote":"Provides the jina-reranker-v2 cross-encoder used to rescore retrieval candidates in the SurgWiki backend."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}