{"id":"2a91560c-4d34-41dd-a237-4708f00e561a","arxiv_id":"2506.02555","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SurgVLM, a family of surgical vision-language models trained on 1.81M frames and 7.79M conversations, outperforms 14 commercial VLMs on a six-dataset surgical benchmark.","lead":"The paper builds SurgVLM, a vision-language model fine-tuned on a new large surgical dataset, and reports it beats major commercial models across ten surgical tasks. A smart generalist might read it to see how domain-specific fine-tuning compares to general chatbots in a high-stakes medical setting.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline superiority result is not yet supported: all six SurgVLM-Bench datasets appear in training DB, so SurgVLM's 75.4% gain over Gemini may be in-distribution adaptation or worse; no same-data fine-tuned baseline or held-out evaluation exists.","rationale":"Good-faith summary: SurgVLM-DB is a genuinely large aggregation effort, the model family is clearly specified, and the benchmark provides useful standardized metrics. The central claim, however, is comparative: SurgVLM-72B achieves a 75.4% higher arena score than Gemini 2.0 Flash and surpasses 14 commercial VLMs on 24 metrics. That comparison is only meaningful if both sides face the same task regime. The manuscript's own supplement shows that the six evaluation datasets are drawn from the training database, and no fine-tuned commercial/open-source control is included. The most load-bearing assumption is therefore that zero-shot commercial VLM performance is an appropriate baseline for a model fine-tuned on the benchmark's source distribution. This is also the reader's weakest_assumption, so I agree. My proposed test separates the two interpretations: if test frames are in training, the result is contaminated; if holding out benchmark test splits collapses the gains, the method's contribution is domain adaptation rather than general surgical intelligence. Under either outcome, the paper's contribution remains substantial, but the claims need to be reframed and the benchmark protocol needs a same-data fine-tuned baseline, which is exactly the conditional revision the reader requested. Therefore I do not move the verdict.","tokens_in":33160,"tokens_out":5413,"duration_ms":55426,"concrete_test":"Contamination audit plus held-out retraining: (1) Release a split table mapping every SurgVLM-Bench test frame (the 1000-frame random subsets) to SurgVLM-DB, and verify whether those frames or their source videos/sequences appear in training; if they do, the reported numbers are invalid. (2) Independently of that audit, retrain SurgVLM-72B-LoRA on SurgVLM-DB with the six benchmark test splits (or the source videos containing them) excluded, then re-evaluate on the same 1000-frame subsets. If the arena score remains close to 336.21 and still exceeds Gemini 2.0 Flash by a large margin, the claim is supported. If it drops toward the 184.85 of Qwen2.5-VL-72B-Instruct or below, the reported superiority is attributable to training/evaluation overlap rather than to the proposed method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table S2 lists all six SurgVLM-Bench datasets (EndoVis2017, EndoVis2018-VQA, Cholec80, CholecT50, SAR-RARP, Endoscapes2023) inside SurgVLM-DB, and gives nonzero 'test' image counts for each. Supplement C says SurgVLM-Bench randomly samples 1000 frames from the official test split of each dataset. If those frames, or the videos they come from, are included in the 1.81M-frame training set, the arena-score gap (336.21 vs 191.70) may be inflated by direct contamination. Even under the weaker reading that only official train splits were used, every benchmark task belongs to the same datasets and label distributions used for supervised instruction tuning, while Gemini 2.0 Flash, Qwen 2.5 Max, and GPT-4o are evaluated zero-shot. The paper acknowledges 'All SurgVLM models are SFT exclusively on SurgVLM-DB' and that the benchmark is 'selected from SurgVLM-DB,' but it never supplies a control: a same-base, same-data fine-tuned baseline (e.g., Qwen2.5-VL-72B-Instruct LoRA-tuned on SurgVLM-DB) or a holdout evaluation on surgical data not seen in training. Without one of these, 'consistently surpasses 14 mainstream commercial VLMs across all 24 metrics' cannot be read as evidence of superior surgical intelligence; it is at best evidence of successful domain adaptation to the benchmark distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes SurgVLM, a family of surgical vision-language models (7B/32B/72B) built on Qwen2.5-VL, together with a large multimodal surgical database SurgVLM-DB (approximately 1.81M frames and 7.79M conversations assembled from 23 public datasets) and a six-dataset benchmark SurgVLM-Bench. The authors report that SurgVLM-72B achieves an arena score of 336.21 versus 191.70 for Gemini 2.0 Flash, and claim consistent improvements over 14 commercial VLMs across all 24 metrics. The central comparison is confounded: all six benchmark datasets are included in SurgVLM-DB, so the reported superiority reflects in-distribution supervised fine-tuning rather than a head-to-head test of general surgical intelligence against zero-shot commercial models. The evaluation protocol also contains internal inconsistencies in benchmark sampling and in the metrics reported across protocols.","tokens_in":33470,"tokens_out":7100,"duration_ms":65570,"significance":"The dataset contribution is potentially substantial: SurgVLM-DB unifies 23 public datasets across 16 surgical types and provides a task hierarchy spanning perception, temporal analysis, and reasoning, and SurgVLM-Bench covers six widely used tasks. If the evaluation were made fair, this would be a valuable community resource. However, the headline performance claims are not currently supported because the benchmark is drawn from the training database and the baselines are evaluated zero-shot; the paper provides no same-base fine-tuned control and no holdout evaluation on surgical data absent from SurgVLM-DB. The claimed 75.4% improvement over Gemini 2.0 Flash is therefore, at best, evidence of successful domain adaptation to the benchmark distribution rather than of superior surgical intelligence.","major_comments":[{"comment":"The central comparison is confounded by train/evaluation overlap. Table S2 explicitly lists EndoVis2017, EndoVis2018-VQA, Cholec80, CholecT50, SAR-RARP, and Endoscapes2023 as components of SurgVLM-DB, with nonzero test counts, and Supplement C states that SurgVLM-Bench is selected from SurgVLM-DB and that frames are sampled from the official test split of each dataset. Since all SurgVLM variants are SFT on SurgVLM-DB, the arena-score gap in Table 2 may be inflated by direct test-frame contamination or, if only train splits were used, by in-distribution adaptation. The commercial baselines are evaluated zero-shot, so the comparison does not support the claim of superior surgical intelligence. Provide a control where the same base model (e.g., Qwen2.5-VL-72B-Instruct) is LoRA-tuned on SurgVLM-DB and evaluated under the identical protocol, and/or evaluate on a surgical dataset not present in SurgVLM-DB; otherwise the superiority claims should be reframed as domain-adaptation results.","section":"Section 2.3, Supplement C, Table S2"},{"comment":"The stated benchmark sampling protocol is internally inconsistent. Supplement C says that 1000 frames are randomly sampled from the official test split of each dataset, but Table S2 reports EndoVis2017 test count as 600 and EndoVis2018-VQA test count as 447. Clarify the exact number of frames actually used per dataset and report per-dataset sample sizes in the main text, since the arena score sums normalized scores and unequal sampling affects the comparability of the aggregate metric.","section":"Supplement C"},{"comment":"The claim that SurgVLM 'consistently outperforms commercial VLMs across all 24 metrics' mixes evaluation protocols. In Tables 3 and 4, all commercial baselines are evaluated under MCQ, while SurgVLM-7B and SurgVLM-32B are reported only under OV, with SurgVLM-72B reported under both protocols. The paper should state unambiguously which model/protocol combination underlies each headline claim and present same-protocol results for every model; the exact-match keyword rule for OV and the MCQ option-selection rule are not interchangeable, so mixing them makes the aggregate comparison difficult to verify.","section":"Section 2.3, Tables 3-4"},{"comment":"The reported scale of SurgVLM-DB is inconsistent: the abstract and introduction say 'over 1.81 million frames,' Table 1 says '1810K' images, and Section 2.1 says '1.181M annotated images.' Correct the mismatch, since dataset scale is a primary claimed contribution.","section":"Section 2.1, Table 1, Abstract"}],"minor_comments":[{"comment":"The sentence 'SurgVLM-72B under MCQ evaluation delivers an additional 4.35 point boost over its MCQ counterpart by demonstrating true OV reasoning' appears to be a typo: the comparison is between OV and MCQ evaluation of the same 72B model, not between two MCQ runs.","section":"Section 2.3"},{"comment":"The text refers to 'SurgX-MMBench' in the evaluation metrics section; this appears to be a leftover from a different benchmark template and should be replaced with SurgVLM-Bench.","section":"Supplement C.1"},{"comment":"The spelling 'Endovis2018' is inconsistent; elsewhere the paper uses 'EndoVis2018.' Please standardize dataset names throughout.","section":"Table 3"},{"comment":"The model name 'MiniCPM-O-2 6' is missing a dot; it should be 'MiniCPM-O-2.6'.","section":"Table 3"},{"comment":"The text states that Cholec80 phase recognition accuracy 'notably decreases to 69.99%' with SurgVLM-72B MCQ, but Table 3 reports 69.66%; the numbers should be reconciled.","section":"Section 2.4"},{"comment":"The labels 'Improve 15.62%' and 'Improve 51.68%' do not state the reference model for the improvement; specify what each percentage is relative to.","section":"Figure 3a"},{"comment":"The sentence 'All datasets in SurgVLM-Bench employ the official training/testing splits and are balanced in terms of surgical types and anatomical structures, ensuring a comprehensive, end-to-end evaluation of multimodal surgical intelligence without fine-tuning' should clarify that only the baselines are evaluated without fine-tuning, since the SurgVLM models are fine-tuned on SurgVLM-DB.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The core issue is the train/evaluation overlap: all six benchmark datasets appear in the training database. If the authors can add a same-base fine-tuned control and clarify the sampling protocol, the manuscript could be reframed as a domain-adaptation study with a useful dataset contribution. As it stands, the 'superior surgical intelligence' claim over zero-shot commercial APIs is not supported. I would not reject outright because the dataset and task hierarchy may be valuable, but the revision needs to address the confounding head-on rather than merely add caveats."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result is the arena-score gap: SurgVLM-72B at 336 versus Gemini 2.0 Flash at 191. I would not trust that gap as evidence of surgical intelligence. All six SurgVLM-Bench datasets are listed inside the training database (Table S2), and SurgVLM is SFT exclusively on that database. The commercial baselines are zero-shot. So the comparison primarily shows that supervised fine-tuning on a dataset moves scores on that same dataset's benchmark. The stress-test note is correct: without a same-data fine-tuned baseline (e.g., Qwen2.5-VL-72B LoRA-tuned on SurgVLM-DB) or a held-out evaluation, \"consistently surpasses 14 mainstream commercial VLMs\" is an overclaim.\n\nThat said, there is real substance here. SurgVLM-DB is the largest surgical multimodal dataset I know of: 1.81M frames, 7.79M conversations, 16 surgical types, 10 tasks, assembled from 23 public datasets. The four-stage construction pipeline (label cleaning, cross-task enrichment, explainable answer generation, conversational diversity) is genuinely careful, and the release of such a resource would benefit the surgical-AI community. The benchmark itself, with six datasets and 24 metrics, is a reasonable snapshot of current VLM capability in surgery. The model recipe is standard Qwen2.5-VL with instruction tuning and LoRA, but the paper doesn't overclaim architectural novelty; the contribution is the data and the systematic evaluation.\n\nThe soft spots are real but mostly addressable. First, the train/test overlap is the big one, and it affects the central claim. Second, the supplement says 1000 frames are randomly sampled per benchmark dataset, but EndoVis2017 has only 600 test frames. That inconsistency needs clarification. Third, the open-vocabulary evaluation uses exact-match keywords, which systematically penalizes verbose but semantically correct responses from commercial models; the paper mentions this but should report MCQ and OV side by side for all models, not just SurgVLM. The absence of temporal modeling is acknowledged, so I won't belabor it.\n\nWho should read this? Surgical-AI researchers who want a unified instruction-tuning dataset or a baseline leaderboard. I would engage with the dataset once released, and I would want the authors to fix the baseline and sampling issues before quoting any performance numbers.\n\nRecommendation: send to peer review with a request for major revisions. The data contribution alone justifies referee time, and the evaluation issues are fixable in a revision.","headline":"Valuable dataset and benchmark, but the headline superiority over commercial VLMs is in-distribution fine-tuning, not general surgical intelligence.","tokens_in":688,"tokens_out":3736,"would_cite":true,"duration_ms":59970,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SurgVLM claims that a single vision-language model fine-tuned on a unified multimodal surgical database can handle ten surgical tasks and outperform fourteen commercial vision-language models on every metric of a new six-dataset benchmark.","keywords":["surgical vision-language model","surgical intelligence","vision-language foundation model","multimodal surgical database","surgical phase recognition","surgical action triplet recognition","critical view of safety","instruction tuning"],"falsifier":"Retrain SurgVLM-72B on SurgVLM-DB with the training splits of all six SurgVLM-Bench datasets removed, then evaluate on the same benchmark; if the arena score falls toward the 118–192 range of the commercial baselines, the reported 75% margin reflects in-distribution adaptation rather than transferable surgical understanding. A complementary check is zero-shot evaluation on a procedure type absent from the 16 in SurgVLM-DB, comparing SurgVLM-72B against Gemini 2.0 Flash under the same open-vocabulary protocol.","tokens_in":32943,"feed_emoji":"🏥","tokens_out":6711,"duration_ms":58471,"temperature":0.7,"pith_summary":"SurgVLM is a family of vision-language models built on Qwen2.5-VL and fine-tuned on a newly assembled multimodal surgical database, SurgVLM-DB, which folds 23 public datasets (16 surgical types, 18 anatomical structures, over 1.8 million frames, 7.79 million conversations) into ten surgical tasks organized in a perception-to-reasoning hierarchy. The paper's claim is that one universal model can handle all ten tasks — instrument and tissue recognition and localization, phase, step, action and triplet recognition, and critical-view-of-safety assessment — and that it outperforms 14 mainstream commercial vision-language models on every metric of a new six-dataset benchmark, SurgVLM-Bench. The headline result is an overall arena score of 336.21 for SurgVLM-72B against 191.70 for Gemini 2.0 Flash, a 75.4% gain, with especially large jumps on triplet recognition and instrument localization. The authors argue the gains come primarily from the database's four-stage construction pipeline — label cleaning, cross-task correlation enrichment, explainable answer generation, and conversational diversity expansion — rather than from new architecture, and that a mid-sized SurgVLM-32B already captures most of the benefit on recognition tasks.","feed_headline":"75.4% score jump: surgical VLM tops Gemini 2.0 Flash","feed_subtitle":"Trained on 1.8M surgical frames, SurgVLM-72B scores 336 vs 192 on a six-dataset surgical benchmark.","key_machinery":"The load-bearing mechanism is the SurgVLM-DB construction pipeline feeding a hierarchical vision-language alignment objective. The four-stage pipeline (1) cleans and standardizes labels to professional terminology, (2) merges correlated attributes such as phase–step and instrument–action into joint prompts, (3) augments answers with explainable, vision-grounded descriptions, and (4) expands 100–200 conversation formats per task with single- and multi-turn variants. The ten tasks are organized into three tiers — visual perception (five tasks), temporal analysis (four tasks), and reasoning (critical-view-of-safety assessment) — and trained with a single unified autoregressive loss so the model learns task dependencies instead of isolated labels. The model itself follows the Qwen2.5-VL architecture with dynamic-resolution patch encoding, windowed attention, M-RoPE positional encoding aligned to real timestamps, and 4x visual token compression.","core_discovery":"The paper sets out to establish that a single vision-language foundation model can carry the full range of surgical-intelligence tasks once it is given enough structured, hierarchically organized domain supervision. SurgVLM-DB supplies that supervision: labels from 23 public datasets are cleaned to professional medical terminology, paired across correlated tasks (phase with step, instrument with action), expanded into explainable descriptions that tie text to visual evidence, and rendered into 100–200 varied single- and multi-turn conversation templates per task. The models are then trained with a single unified autoregressive loss over all ten tasks, with size scaled from 7B to 32B to 72B under progressively lighter fine-tuning (full fine-tuning, frozen backbone with adapters, LoRA). On SurgVLM-Bench the authors report consistent superiority across all 24 metrics: phase recognition accuracy rises from 38.89% (Gemini 2.0 Flash) to 76.40% (SurgVLM-72B open-vocabulary), triplet accuracy from 2.08% to 13.10%, instrument localization mIoU from 19.90% to 59.40%, and critical-view-of-safety assessment from 59.61% to 76.73%. The authors also report that scaling behaves differently by task: localization keeps improving with model size, recognition plateaus near 30B parameters, and CVS stays flat, which they read as evidence that the bottleneck for surgical adaptation is data quality and task structure rather than backbone capacity.","pith_inferences":["The benchmark is in-distribution: all six SurgVLM-Bench datasets appear in SurgVLM-DB, so the comparison against commercial VLMs measures fine-tuned adaptation on seen task distributions rather than zero-shot transfer; a genuinely out-of-distribution evaluation on a procedure type absent from the 16 included would likely shrink the reported margin.","The exact-keyword open-vocabulary protocol penalizes synonyms and paraphrases, which may understate commercial VLMs; a paraphrase-tolerant scorer would give a fairer picture of general models while arguably keeping the precision bar that clinical use requires.","A direct ablation of the four pipeline modules — for instance, removing cross-task correlation enrichment while keeping everything else identical — would attribute the reported gains and is the natural next experiment the current study leaves implicit.","If the gains hold under out-of-distribution testing, the practical consequence is that hospitals could adapt existing open VLMs to their own surgical video archives with modest compute, since the authors find no need for a surgical-specific vision backbone."],"forward_implications":["A single fine-tuned SurgVLM can replace multiple task-specific surgical AI pipelines: the same weights handle perception, temporal analysis, and safety reasoning, with open-vocabulary responses rather than fixed-choice answers.","On the authors' benchmark, commercial VLMs are far behind — GPT-4o scores 118.71 against SurgVLM-72B's 336.21 — implying general-purpose multimodal models lack the surgical vocabulary and spatial grounding needed for clinical use.","The reported scaling pattern implies that a mid-sized model around 30B parameters with adapter tuning reaches competitive recognition and CVS performance at a fraction of the cost of a 72B model, so surgical deployment does not require the largest model.","The data pipeline is transferable: the same cleaning-correlation-explanation-diversification recipe could be applied to other procedurally annotated medical video corpora to build domain-specific VLMs without new architectures."],"supporting_citations":[{"why":"Qwen2.5-VL technical report, the base architecture and pretrained weights that SurgVLM is built on.","marker":"[7]"},{"why":"Cholec80 dataset, used for phase recognition in SurgVLM-Bench and as a training source in SurgVLM-DB.","marker":"[57]"},{"why":"CholecT50 dataset, used for instrument–verb–target triplet recognition in both training and benchmarking.","marker":"[40]"},{"why":"Surgical-VQA work supplying EndoVis2018-VQA, the vision-grounding benchmark for instrument and tissue recognition.","marker":"[48]"},{"why":"EndoVis2017 dataset, the instrument localization benchmark scored by mIoU and mAP.","marker":"[4]"},{"why":"SAR-RARP dataset, the fine-grained action recognition benchmark on robot-assisted prostatectomy.","marker":"[45]"},{"why":"Endoscapes2023 dataset, the critical-view-of-safety assessment benchmark and a training source.","marker":"[38]"},{"why":"Gemini 2.0 Flash, the strongest commercial baseline that the 75.4% arena-score improvement is measured against.","marker":"[51]"},{"why":"DFN CLIP vision encoder used by the 32B and 72B variants, supporting the claim that off-the-shelf encoders suffice.","marker":"[16]"},{"why":"LoRA, the parameter-efficient adaptation strategy used for the 72B model.","marker":"[20]"}],"fun_headline_variants":["SurgVLM-72B tops GPT-4o, Gemini on 24 surgical metrics","Single surgical VLM beats 14 commercial models in benchmark","SurgVLM: 1.8M-frame training outdoes Gemini 2.0 Flash","Surgical AI: one model, 10 tasks, outperforms GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that beating untrained commercial models on benchmark datasets whose training splits are included in SurgVLM-DB demonstrates superior surgical intelligence, rather than successful memorization of those datasets' label distributions.","fun_headline_variants_meta":{"raw":{"variants":["SurgVLM-72B tops GPT-4o, Gemini on 24 surgical metrics","Single surgical VLM beats 14 commercial models in benchmark","SurgVLM: 1.8M-frame training outdoes Gemini 2.0 Flash","Surgical AI: one model, 10 tasks, outperforms GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001299,"raw_usage":{"total_tokens":5433,"prompt_tokens":1209,"completion_tokens":4224,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":825,"completion_tokens_details":{"reasoning_tokens":4134}},"tokens_in":825,"tokens_out":4224,"duration_ms":31668,"temperature":1.0,"reasoning_tokens":4134,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:20:55.292860+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain SurgVLM-72B on SurgVLM-DB with the training splits of all six SurgVLM-Bench datasets removed, then evaluate on the same benchmark; if the arena score falls toward the 118–192 range of the commercial baselines, the reported 75% margin reflects in-distribution adaptation rather than transferable surgical understanding. A complementary check is zero-shot evaluation on a procedure type absent from the 16 in SurgVLM-DB, comparing SurgVLM-72B against Gemini 2.0 Flash under the same open-vocabulary protocol.","supporting_citations":[{"cited_title":"Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy","cited_arxiv_id":null,"evidence_quote":"Cholec80 dataset, used for phase recognition in SurgVLM-Bench and as a training source in SurgVLM-DB."},{"cited_title":"Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos","cited_arxiv_id":null,"evidence_quote":"CholecT50 dataset, used for instrument–verb–target triplet recognition in both training and benchmarking."},{"cited_title":"Surgical-VQA: Visual question answering in surgical scenes using trans- former","cited_arxiv_id":null,"evidence_quote":"Surgical-VQA work supplying EndoVis2018-VQA, the vision-grounding benchmark for instrument and tissue recognition."},{"cited_title":"Sar-rarp50: Segmentation of surgical in- strumentation and action recognition on robot-assisted radical prostatectomy challenge, 2024","cited_arxiv_id":null,"evidence_quote":"SAR-RARP dataset, the fine-grained action recognition benchmark on robot-assisted prostatectomy."},{"cited_title":"Endoscapes2023, a critical view of safety and surgical scene segmentation dataset for laparoscopic cholecys- tectomy, 2024","cited_arxiv_id":null,"evidence_quote":"Endoscapes2023 dataset, the critical-view-of-safety assessment benchmark and a training source."},{"cited_title":"Lora: Low-rank adaptation of large lan- guage models","cited_arxiv_id":null,"evidence_quote":"LoRA, the parameter-efficient adaptation strategy used for the 72B model."}],"review_version":1}