{"id":"a9f2d333-0f7c-4c2b-94af-ee40cf413cc5","arxiv_id":"2509.00357","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A surgical-video multimodal LLM with instrument-centric masked pretraining and interleaved temporal embeddings beats general video LLMs on a new CholecT50-derived benchmark.","lead":"SurgLLM is a large multimodal model that watches surgical videos and answers questions about instruments, phases, locations, and timing. It reports large improvements over general video LLMs on a new GPT-4-generated surgical video benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Instrument-detector pseudo-labels from 5 videos define both MV-Recon masks and location/movement ground truth; without validation of these boxes, the reported location, movement, and caption gains may be artifacts of shared label bias.","rationale":"The strongest claim is SOTA improvement across captioning, general VQA, and temporal VQA. That claim depends on a valid benchmark. The pseudo-label pipeline is the most load-bearing because it simultaneously controls the pretraining signal and the evaluation labels. Unlike baseline tuning or GPT-4 judge variability, which affect comparison fairness, wrong pseudo-labels corrupt the ground truth itself. Since MV-Recon masking (Eq. 1-2) is instrument-centric, SurgLLM's encoder is directly shaped by these boxes; if they are noisy, the location/movement QA task becomes a test of mimicking detector errors. No detection metrics are reported, so the assumption is unsupported. The fix is simple and empirical: human-annotate a sample, compare pseudo-labels, and re-score. This agrees with the reader's identified weakest assumption. The manuscript is otherwise coherent and ablations are internally consistent, so the appropriate verdict remains conditional pending release and validation rather than rejection.","tokens_in":24354,"tokens_out":5749,"duration_ms":72299,"concrete_test":"Select 200 frames stratified across the 10 held-out test videos and phase labels. Have two surgical experts draw instrument bounding boxes (grasper, hook, clipper, etc.). Compute mAP@0.5 and mean IoU of the detector pseudo-labels used in the paper versus expert boxes. Then re-score SurgLLM and VideoLLaMA-v2-7B on the Location and Movement VQA subsets using expert boxes as ground truth, and re-run the MV-Recon ablation with masks recomputed from expert boxes. If mAP@0.5 is high (>0.7) and SurgLLM's margins over the best baseline persist, the concern is refuted; if margins collapse, the pseudo-label bias drives the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 (Fig. 5) constructs the benchmark by training an instrument detector on the 5 CholecT50-Challenge videos and applying it to all 50 CholecT50 videos to produce bounding-box pseudo-labels. These boxes are load-bearing twice: (1) Eq. (1)-(2) use them to create instrument-centric MV-Recon masks, so the video encoder is pretrained to focus on detector-defined instruments; (2) the Location and Movement VQA ground truth, as well as GPT-4 caption prompts, are derived from the same boxes. The paper reports no mAP/IoU for this detector and no quantitative validation of the pseudo-labels; the stated 'manual filtering and automatically validate' is unspecified. If the detector transfers poorly, location/movement results are measured against incorrect boxes, and SurgLLM is uniquely advantaged because its pretraining and instruction data encode the same detector bias. The claimed margins (e.g., 61.4% vs 46.5% Location; 64.4% vs 56.4% Movement) could then reflect benchmark-label artifact rather than surgical understanding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SurgLLM, a large multimodal model for surgical video understanding. Three contributions are presented: (i) Surgical Context-aware Multimodal Pretraining (Surg-Pretrain), which uses instrument-centric masked video reconstruction (MV-Recon) with multi-scale tube masking (Eq. 1-2) followed by video-text contrastive alignment; (ii) Temporal-aware Multimodal Tuning (TM-Tuning), which interleaves temporal descriptors with visual segment embeddings (Eq. 6); and (iii) a Surgical Task Dynamic Ensemble that routes queries to task-specific Q-Former memories and LoRA weights (Eq. 7-9). The authors construct a new benchmark from CholecT50 by using an instrument detector trained on CholecT50-Challenge to generate bounding-box pseudo-labels for all 50 videos, then using these boxes and CholecT50 annotations to prompt GPT-4 for captions and VQA data. Experiments compare SurgLLM against general video LLMs (VideoLLaMA, LLaMA-VID, Video-LLaVA, VideoLLaMA-v2, VideoChat2, Qwen2.5-VL) on captioning, general VQA, and temporal VQA, reporting large gains, e.g., 60.9% vs 42.2% time-spot accuracy and 42.6% vs 23.3% duration IoU.","tokens_in":24627,"tokens_out":3896,"duration_ms":45507,"significance":"If the reported results are substantiated, SurgLLM would be a meaningful step toward adapting video MLLMs to surgical domains, with a sensible focus on instrument-centric masking and interleaved temporal embeddings. The paper includes ablations for most components and a hyperparameter analysis. However, the strength of the empirical claims is currently limited by three protocol issues: unvalidated pseudo-labels that are load-bearing in both pretraining and evaluation, an underspecified baseline comparison that may be zero-shot, and a GPT-4-as-judge evaluation without human validation. These issues are addressable, so the underlying ideas have promise, but the evidence as presented does not yet support the 'significant improvements over SOTA' claim at face value.","major_comments":[{"comment":"The instrument detector is trained on the 5 CholecT50-Challenge videos and applied to all 50 CholecT50 videos to produce bounding-box pseudo-labels. These boxes drive both the MV-Recon masks in Eq. (1)-(2) and the Location/Movement ground truth and GPT-4 caption prompts. The paper reports no mAP/IoU for the detector and no quantitative validation of the pseudo-labels; the statement 'manual filtering and automatically validate' is unspecified. If the detector transfers poorly, location/movement/temporal results are measured against incorrect boxes, and SurgLLM is uniquely advantaged because its pretraining and evaluation labels share the same detector bias. Please report detector performance on a held-out set with human-annotated boxes, describe the filtering/validation process, and ideally re-run the location/movement experiments on a subset with verified boxes.","section":"Section 4.1, Fig. 5, Eq. (1)-(2)"},{"comment":"The comparison protocol is underspecified. It is unclear whether the baseline video LLMs were fine-tuned on the same 80% training split or evaluated zero-shot. If they were zero-shot, the comparison is not apples-to-apples because SurgLLM is fine-tuned on the benchmark (including the GPT-4-generated question formats and temporal QA style). This would explain the large margins in temporal VQA. Please state explicitly the training protocol for each baseline; if zero-shot, include at least one strong baseline fine-tuned on the same data (e.g., Video-LLaVA or Qwen2.5-VL with LoRA) to isolate the contribution of the proposed architectural components.","section":"Section 4.2, Tables 1 and Fig. 8"},{"comment":"General VQA accuracy is computed by GPT-4 judging whether the model's answer matches the ground truth. No human validation of these judgments is reported, and no error bars or confidence intervals are given. Since the questions and answers were themselves generated by GPT-4 from the same annotations, the evaluation is partly circular: SurgLLM is fine-tuned to mimic GPT-4's phrasing, and GPT-4 then judges its outputs. Please provide (i) a human-validated subset (e.g., 200-300 samples) with agreement rate between GPT-4 and human judgments, and (ii) per-task accuracy with standard deviations across multiple runs or bootstrapped confidence intervals.","section":"Section 4.1, 'Evaluation Metrics'"},{"comment":"The time-spot accuracy is defined only as 'accuracy of time spot prediction with the ground truth.' It is unclear what tolerance is allowed (exact second? +/-1 second? within the same clip?). This matters because the reported margins (60.9 vs 42.2) are central to the temporal-awareness claim. Please define the matching criterion precisely and, if possible, evaluate also with a tolerance curve. Additionally, the ground-truth time spots are derived from the unvalidated pseudo-labels, so a human-verified subset is essential.","section":"Section 4.1, Temporal VQA metrics"},{"comment":"The ablation table's first row reports accuracy without MV-Recon, without Task Dynamic Ensemble, and without multi-task Q-Former, yet it still achieves 56.8% average general VQA. What is the architecture in this row? A Q-Former is necessary to map visual tokens to the LLM, so the row likely uses a standard Q-Former; please clarify. The column labeling should distinguish 'multi-task Q-Former with shared memories' from 'no multi-task Q-Former (standard Q-Former)' to make the ablations interpretable.","section":"Table 2(b), ablation"}],"minor_comments":[{"comment":"Typo: 'Surg-Petrain' should be 'Surg-Pretrain'.","section":"Fig. 1 caption"},{"comment":"Typo: 'textural-visual temporal interleave embeddings' should be 'textual-visual'.","section":"Section 3.3"},{"comment":"The comparison only includes general video LLMs. If surgical-domain MLLMs or surgical VQA models exist (e.g., SurgicalGPT, SurgVLP, Surgical-VQA), they should be discussed or compared to support the 'state-of-the-art' claim in the surgical context.","section":"Section 4.2.3"},{"comment":"Bar charts lack error bars and numeric labels on every bar; adding error bars and values would improve readability and support the robustness claims.","section":"Fig. 8"},{"comment":"The hint retention ratio r, multi-scale tube durations, and LoRA rank/alpha are hyperparameters, but only video segment length is analyzed. A brief sensitivity analysis for r and LoRA rank would strengthen the hyperparameter section.","section":"Section 3.2.1 and 4.4"}],"recommendation":"major_revision","confidential_remarks":"The pseudo-label issue is the most serious concern and could, if unaddressed, invalidate the location/movement/temporal results. However, the paper's other contributions (interleaved temporal embeddings, task ensemble) are reasonable and the ablations show consistent trends. I believe the paper is fixable with additional validation and a clarified experimental protocol, hence major_revision rather than reject. I would also encourage the authors to consider releasing the benchmark and evaluation code to facilitate reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: SurgLLM combines three sensible ideas for surgical video MLLMs — instrument-centric masked reconstruction, interleaved temporal embeddings, and task-routed LoRA/Q-Former — and the ablations show each piece does something. But the reported margins over baselines are built on a benchmark with a pseudo-label loop and a GPT-4 judge, so I'd treat the numbers as conditional until the authors release data and fix the protocol.\n\nWhat's genuinely new and good: the interleaved temporal descriptor design (Table 3) is a simple, effective way to bind timestamps to visual tokens; the dynamic task ensemble is a reasonable method to avoid multi-task interference; and the ablations cover the important design choices, including shared vs. independent Q-Formers and a mismatched text-video sanity check. The framework is coherent and the paper is clearly written.\n\nThe soft spots are real but addressable. The biggest is the instrument-detector loop: a detector trained on 5 CholecT50-Challenge videos is applied to all 50 videos to create bounding-box pseudo-labels, and those boxes serve double duty — they define the MV-Recon masks (Eq. 1–2) and the location/movement ground truth. The paper reports no mAP or IoU for the detector and describes validation as 'manual filtering and automatically validate' without any numbers. If the detector transfers poorly, SurgLLM is the only model pretrained to attend exactly where those boxes say the instruments are, so its location and movement gains would reflect label bias rather than real understanding.\n\nSecond, the comparison protocol is underspecified. The paper never states whether the baseline video LLMs were fine-tuned on the same paired data or evaluated zero-shot. If they were zero-shot, the comparison is not apples-to-apples.\n\nThird, GPT-4 is used both to generate the QA ground truth and to judge the model's answers. Since SurgLLM is fine-tuned on GPT-4-written text, that is a meaningful confound, especially without human validation or judge-variability numbers.\n\nFourth, the 4-second segment length appears to be selected from test-set results (Table 4); no validation split is mentioned.\n\nNone of these are fatal — the central idea is plausible, the math is straightforward, and the citation pattern is acceptable (the self-references are relevant). The weak point is the data and evaluation. The claimed 10–20 point gains over VideoLLaMA-v2 and others should be read as upper bounds under the authors' setup. This paper deserves a serious referee, but major revisions are needed: release the benchmark and code, report detector accuracy on held-out boxes, enforce a common fine-tuning protocol for all baselines, add human evaluation or judge variance, and use a validation split for hyperparameters. If those are addressed, it could be a solid contribution for the surgical-video MLLM community.","headline":"SurgLLM is a coherent, well-ablated MLLM for surgical video, but the reported margins over baselines rest on a pseudo-label loop and a GPT-4 judge, so the numbers are conditional until the authors release data and tighten the protocol.","tokens_in":25134,"tokens_out":3929,"would_cite":true,"duration_ms":45982,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A surgery-specific multimodal LLM beats general video models on temporal questions, reporting 60.9% time-spot accuracy.","keywords":["surgical video understanding","multimodal large language model","masked video reconstruction","temporal visual question answering","instrument-centric masking","task dynamic ensemble","CholecT50","surgical captioning"],"falsifier":"Re-run the temporal VQA benchmark on a test set where instrument bounding boxes and event timestamps are manually annotated by surgeons rather than generated from the 5-video-trained detector, and compare SurgLLM against the same general baselines. If the 60.9% time-spot accuracy and 42.6% duration IoU collapse toward or below the baselines, the central claim that instrument-centric masking and interleaved temporal embeddings cause the gains is undermined—the apparent advantage would instead reflect shared label statistics between pretraining, QA generation, and evaluation.","tokens_in":24255,"feed_emoji":"⏱️","tokens_out":7286,"duration_ms":87344,"temperature":0.7,"pith_summary":"Surgeons rely on endoscopic video, but general-purpose video AI models were trained on natural scenes and cope poorly with surgery's visual character: small moving instruments against a relatively static background, with long stretches of near-redundant frames. SurgLLM is the paper's attempt to close that gap with one adaptable multimodal system. The paper claims that three targeted changes—instrument-centric masked video reconstruction during pretraining, interleaved timestamp-descriptor and vision embeddings during tuning, and a task router that selects per-question specialist parameters—let a single model outperform seven general video LLMs on surgical captioning, general VQA, and temporal VQA. The sharpest reported gain is on temporal questions: 60.9% accuracy on time-spot queries and 42.6% duration IoU, versus 42.2% and 23.3% for the best general baseline.","feed_headline":"Surgery-specific LLM hits 60.9% on video-time questions","feed_subtitle":"Instrument-focused masking and interleaved timestamps give the model its edge on phase, location, and duration queries.","key_machinery":"Three mechanisms carry the argument. (1) Instrument-centric Multi-scale Masked Video Reconstruction (MV-Recon): the input video is divided into 3D tubes at four temporal scales (2/4/8/16 frames); tubes whose reference frame contains an instrument bounding box are masked, with only a random 10% kept as hints, and a video decoder must reconstruct the masked tubes. This pushes the encoder to represent the moving instruments—the foreground that matters—rather than the repetitive background. (2) TM-Tuning builds the LLM input as interleaved sequences [S1, Hv1, S2, Hv2, ..., q], where each Si is a text descriptor such as 'this clip spans from i×t to (i+1)×t seconds' placed immediately before that","core_discovery":"The paper claims that the reason video LLMs struggle on surgery is not model capacity but mismatch: visual encoders pretrained on natural scenes miss instrument-centric foreground dynamics, and LLMs attach timestamps too loosely to the frames they describe. SurgLLM is a domain-adapted multimodal LLM built on four components: Surg-Pretrain (instrument-centric multi-scale Masked Video Reconstruction plus surgical video-text contrastive alignment), TM-Tuning (interleaving each video clip's temporal descriptor directly before its visual tokens), and a Surgical Task Dynamic Ensemble (a multi-task Q-Former with task-specific learnable memories, plus task-specific LoRA weights on the LLM, selected","pith_inferences":["Because the benchmark's QA pairs were generated by GPT-4 from the same triplet, phase, and location annotations the model is trained to reproduce, part of the reported temporal gap may come from the model learning label statistics rather than genuinely watching the video; a human-annotated temporal QA set on unseen procedure types would isolate this.","The interleaved time-descriptor design suggests a cheap extension to continuous time: instead of discrete clip descriptors, the LLM could receive per-frame or per-token timestamps, potentially supporting sub-second event localization beyond the tested 2-to-16-second granularity.","If MV-Recon transfers, the same multi-scale instrument-centric masking could be applied to other tool-centric endoscopic or industrial inspection videos with small foreground objects on near-static backgrounds; that is a testable claim beyond the paper's cholecystectomy experiments.","The task dynamic ensemble routes purely on query text; a natural stress test is whether the router can handle open-set questions (new phrasings, combined tasks) without collapsing to a single routine, which the paper does not evaluate."],"forward_implications":["A single SurgLLM-style model can replace separate specialist tools for phase recognition, triplet detection, location, relation, movement, captioning, and temporal reasoning, making a unified computer-assisted-surgery assistant feasible on one video encoder and LLM backbone.","Time-awareness at 60.9% spot accuracy and 42.6% duration IoU makes intraoperative applications—alerting when an instrument enters a critical region, predicting phase changes, logging instrument usage times—candidates for real-time deployment, pending clinical validation.","Task-routed LoRA weights plus task-specific memories mean a new surgical task can be added by inserting one new memory/LoRA pair and its routing classifier, without re-tuning shared parameters or losing previous task performance.","MV-Recon's instrument-centric masking offers a transferable recipe for long videos: mask the sparse moving foreground rather than random patches, so repetitive background frames do not drown out the rare decisive moments.","The two-stage progressive optimization (surgical visual pretraining, then task-adaptive tuning) provides a concrete template for adapting general video LLMs to other specialized visual domains."],"supporting_citations":[{"why":"CholecT50 supplies the source videos and the phase/triplet annotations from which the entire caption and VQA benchmark is built.","marker":"[47]"},{"why":"CholecT50-Challenge contributes the 5 videos with instrument bounding-box annotations used to train the detector that produces all location pseudo-labels.","marker":"[74]"},{"why":"GPT-4 generates the dense captions and VQA pairs from triplet, location, and phase lists, and is also used to judge answer correctness in evaluation.","marker":"[75]"},{"why":"InstructBLIP provides the Q-Former design that the multi-task Q-Former adapts into task-specific learnable memories.","marker":"[63]"},{"why":"Vicuna-1.5-7B is the base LLM whose frozen weights W0 are augmented with task-specific LoRA adapters.","marker":"[24]"},{"why":"VideoMAE provides the random-masking pretraining baseline and the initialization weights for the video encoder before MV-Recon.","marker":"[37]"},{"why":"VideoMAE v2 is the architecture used as the video encoder inside the SurgLLM pipeline.","marker":"[77]"},{"why":"VideoLLaMA-v2-7B is the strongest general baseline; the paper's temporal VQA gains of +18.7 and +19.3 are measured against it.","marker":"[69]"}],"fun_headline_variants":["SurgLLM: Spatial focus and temporal awareness for surgical video","Surgical video LLM nails temporal queries with interleaved timestamps","Instrument-centric masking gives video LLM a surgical edge","SurgLLM hits 60.9% on surgical video temporal VQA","New multimodal model for surgery: better captions, VQA, temporal reasoning"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"All instrument-location labels that drive both pretraining (the MV-Recon masks) and the location/time benchmark come from a detector trained on only five annotated videos; if that detector does not transfer to the other 45 videos, both the pretraining signal and the evaluation labels are unreliable, and the reported gains could be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["SurgLLM: Spatial focus and temporal awareness for surgical video","Surgical video LLM nails temporal queries with interleaved timestamps","Instrument-centric masking gives video LLM a surgical edge","SurgLLM hits 60.9% on surgical video temporal VQA","New multimodal model for surgery: better captions, VQA, temporal reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000876,"raw_usage":{"total_tokens":3647,"prompt_tokens":787,"completion_tokens":2860,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":2766}},"tokens_in":531,"tokens_out":2860,"duration_ms":20075,"temperature":1.0,"reasoning_tokens":2766,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:39:51.496001+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the temporal VQA benchmark on a test set where instrument bounding boxes and event timestamps are manually annotated by surgeons rather than generated from the 5-video-trained detector, and compare SurgLLM against the same general baselines. If the 60.9% time-spot accuracy and 42.6% duration IoU collapse toward or below the baselines, the central claim that instrument-centric masking and interleaved temporal embeddings cause the gains is undermined—the apparent advantage would instead reflect shared label statistics between pretraining, QA generation, and evaluation.","supporting_citations":[{"cited_title":"Rendezvous: Attention mechanisms for the recogni- tion of surgical action triplets in endoscopic videos","cited_arxiv_id":null,"evidence_quote":"CholecT50 supplies the source videos and the phase/triplet annotations from which the entire caption and VQA benchmark is built."},{"cited_title":"Gonzalez, and Nicolas Padoy","cited_arxiv_id":null,"evidence_quote":"CholecT50-Challenge contributes the 5 videos with instrument bounding-box annotations used to train the detector that produces all location pseudo-labels."},{"cited_title":"Gonzalez, Ion Stoica, and Eric P","cited_arxiv_id":null,"evidence_quote":"Vicuna-1.5-7B is the base LLM whose frozen weights W0 are augmented with task-specific LoRA adapters."},{"cited_title":"Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training","cited_arxiv_id":null,"evidence_quote":"VideoMAE provides the random-masking pretraining baseline and the initialization weights for the video encoder before MV-Recon."},{"cited_title":"Videomae v2: Scaling video masked autoencoders with dual masking","cited_arxiv_id":null,"evidence_quote":"VideoMAE v2 is the architecture used as the video encoder inside the SurgLLM pipeline."}],"review_version":1}