{"id":"009e72ca-1968-4beb-bcfc-2262ed8f857d","arxiv_id":"2506.07603","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SurgBench assembles a 53-million-frame surgical video pretraining corpus from 16 sources plus a 72-task evaluation benchmark, and shows continual pretraining with VideoMAE improves accuracy by 7.9% top-3 over Kinetics pretraining.","lead":"SurgBench combines 16 surgical video sources into a large pretraining corpus and a 72-task evaluation benchmark for surgical video understanding. Models pretrained on the surgical videos beat models pretrained on ordinary internet videos by about 8 points in the benchmark, suggesting domain-specific pretraining helps.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SurgBench-P may include SurgBench-E test frames; the reported CPT gains could reflect pretraining on evaluation data rather than transferable surgical representations.","rationale":"The reader identified the final-stage IID rebalancing as the weakest assumption, which is a real risk of test-distribution matching. My concern is more direct: Table 2's per-source 'Pre-train Frames' and 'Evaluation Frames' columns, combined with the absence of any statement that SurgBench-E test clips are excluded from SurgBench-P, raise the possibility that the pretraining corpus contains the evaluation videos themselves. This would make the headline comparison to Kinetics unfair regardless of how the final rebalancing stage is guided, because the model's pretraining objective has already seen the test content. The paper's clean fine-tuning protocol (frozen backbone, held-out train/test splits within SurgBench-E) does not address pretraining overlap; it only prevents overlap between fine-tuning train and test. The LapGyn OOD experiment is the one result that would survive this critique, but it is a single task and does not support the exact percentage gains claimed on SurgBench-E. A concrete clip-level intersection test is feasible since the artifacts are available upon request, and it would determine whether the central claim stands or whether the benchmark numbers need substantial revision. This does not refute the benchmark's utility as a resource, but it means the empirical validation of the core claim is not yet verifiable from the paper as written.","tokens_in":18497,"tokens_out":5552,"duration_ms":64062,"concrete_test":"Request the exact clip-level assignment from the authors: for each source, list every clip ID and source-video ID in SurgBench-P and in SurgBench-E train/test splits, then run a hash or frame-timestamp intersection. The concern is settled if and only if no frame of any SurgBench-E test video appears in SurgBench-P. If any overlap exists, recompute Table 4 with SurgMAE-CPT pretrained on the SurgBench-P remainder excluding all clips from SurgBench-E test videos; if the +7%/+7.9% gains shrink substantially, the headline claim is an artifact of pretraining on evaluation data.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Section 5.1: +7% top-1, +7.9% top-3 over Kinetics) depends on SurgBench-P being a legitimate pretraining corpus that does not overlap with SurgBench-E test clips. The paper never states that evaluation clips are excluded from pretraining. Table 2 lists both 'Pre-train Frames' and 'Evaluation Frames' for the same source IDs (e.g., S8 JIGSAWS: 569,048 pretrain vs. 537,645 eval; S10 AutoLaparo: 2,155,843 vs. 160,221), and the total of the two columns equals the 59,742,875 frames in Section 3.4, implying nearly all frames from each source—including those later used for evaluation—were available to the pretraining stage. Section 3.3 guarantees only that, within SurgBench-E, clips from the same video do not appear in both the fine-tuning train and test sets; it is silent about pretraining. If SurgBench-P contains frames from SurgBench-E test videos, VideoMAE's masked autoencoding has already observed those test videos, so the frozen-backbone gains over Kinetics reflect test-content familiarity, not transferable surgical representations. The vague 'precise IID-oriented stage' in Section 4, which rebalances pretraining data toward the downstream distribution, compounds this risk: if the rebalancing uses labels or class statistics from SurgBench-E, the comparison is additionally biased by test-distribution matching. The OOD LapGyn result (Section 5.4) provides partial independent support, but it is a single task with no error bars and cannot validate the headline SurgBench-E numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SurgBench, a unified surgical video pretraining corpus (SurgBench-P) and evaluation benchmark (SurgBench-E) assembled from 16 sources, spanning 22 procedures, 11 specialties, and 72 fine-grained tasks in 6 categories. The authors continually pretrain VideoMAE models (Standard and Large) on SurgBench-P, fine-tune on SurgBench-E, and report that surgical-domain pretraining improves top-1 accuracy by 7% and top-3 accuracy by 7.9% over Kinetics-pretrained VideoMAE, with additional evidence from a LapGyn out-of-distribution experiment. The central claim is that self-supervised continual pretraining on a large, mixed-specialty surgical corpus yields transferable representations beyond those learned from natural video.","tokens_in":18724,"tokens_out":2179,"duration_ms":26098,"significance":"If the result holds, SurgBench would be a valuable community asset: it is substantially broader than existing surgical pretraining corpora (e.g., Endo-FM, GSViT, Surg-3M), and the 6C-10S-72T taxonomy provides a unified evaluation protocol. The paper also has methodological strengths: within SurgBench-E, clips from the same video are split so that they do not appear in both train and test, the backbone is frozen during fine-tuning to isolate representation quality, and the long-tail imbalance factor is capped. The LapGyn result, if taken at face value, provides independent evidence of cross-domain transfer. However, the headline comparison depends critically on whether SurgBench-E evaluation frames are excluded from the SurgBench-P pretraining corpus, a point the manuscript does not establish.","major_comments":[{"comment":"The central claim of Section 5.1 (+7% top-1, +7.9% top-3 over Kinetics) rests on SurgBench-P being a legitimate pretraining corpus that does not contain SurgBench-E test clips. The manuscript never states that evaluation frames are excluded from pretraining. Table 2 lists 'Pre-train Frames' and 'Evaluation Frames' for the same source IDs (e.g., S8 JIGSAWS: 569,048 vs. 537,645; S10 AutoLaparo: 2,155,843 vs. 160,221), and the two columns sum to 59,742,875 frames, exactly matching the total in Section 3.4. This arithmetic implies that essentially all frames from each source, including those later used for evaluation, were available to the pretraining stage. Section 3.3 guarantees only that, within SurgBench-E, clips from the same video do not appear in both the fine-tuning train and test sets; it is silent about pretraining. The paper must either state explicitly that SurgBench-E test clips were excluded from SurgBench-P, or the reported gains must be reinterpreted as test-content familiarity rather than transferable surgical representations.","section":"Sections 3.3, 3.4, and Table 2"},{"comment":"The four-stage continual pretraining procedure is underspecified at the load-bearing point. Stage 4 is described as 'a final precise IID-oriented stage with both upsampling and downsampling,' but the manuscript does not say what information guides this rebalancing. If the final stage uses label statistics or other statistics from the SurgBench-E sources to align the pretraining distribution with the evaluation tasks, then the comparison against Kinetics is biased by test-distribution matching. Please specify the exact criterion used for upsampling/downsampling in stage 4 and state whether any SurgBench-E labels, splits, or task definitions influenced the composition of SurgBench-P.","section":"Section 4, final IID-oriented stage"},{"comment":"The frame-count statements are internally inconsistent. The abstract and Section 3.4 state 53 million frames for SurgBench-P and 59.7 million total frames; Section 4 refers to 'SurgBench-P (74.4 million frames total)'; Figure 1 reports '60 million frames in total'; and Table 2 sums to 59,742,875 frames. This inconsistency matters because the pretraining corpus size is part of the experimental description, and the 74.4M figure is not reconcilable with the tabulated source counts. Please unify these numbers and clarify which count corresponds to the actual clips used in the final CPT stage (39,807 clips) versus the raw collected corpus.","section":"Sections 3.4, 4, and Abstract"},{"comment":"The LapGyn experiment is presented as evidence of cross-domain generalization to unseen procedures, but it is a single task with no error bars or significance testing. The reported 2.69% gain is plausible but could be within run-to-run variation. Since this result is the only independent support for the generalizability claim that does not depend on the pretraining/evaluation overlap question, it should be reported with multiple seeds, confidence intervals, or at least a statement of the number of runs.","section":"Section 5.4, LapGyn OOD result"}],"minor_comments":[{"comment":"There is a dangling reference 'as shown in Table ??' in the paragraph discussing per-category performance fluctuations; this should point to a specific table or figure.","section":"Section 5.1"},{"comment":"The manuscript states that A VOS, SimSurgSkill2021, and AIxSuture cannot be redistributed and are used only for pretraining, but Table 2 lists 'Pre-train Frames' and 'Evaluation Frames' for these sources as well. Please clarify whether the 'Evaluation Frames' column for these sources is used only for internal validation or is actually released as part of SurgBench-E.","section":"Section 3.1 and Appendix A"},{"comment":"The conclusion refers to 'SurgBench-P, a diverse pretraining dataset comprising 53 million frames,' but Table 6 has a typo in its caption ('SurBennch-P'). Please correct the typo.","section":"Section 6 and Appendix C"},{"comment":"The phrase 'enrish task diversity' in the introduction appears to be a typo for 'enrich task diversity.'","section":"Section 1"},{"comment":"The caption of Figure 2 says the accuracy curves 'validate the reliability and data quality of SurgBench-E,' but the figure shows fine-tuning accuracy on individual source datasets; it would be clearer to state that this is a sanity check of label quality rather than a validation of the benchmark's generalization properties.","section":"Section 3.3 and Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The core concern is the possibility of pretraining/evaluation overlap. The Table 2 arithmetic (Pre-train + Evaluation = total frames) makes the omission of an exclusion statement particularly serious. If the authors can confirm that no SurgBench-E test clip appears in SurgBench-P and specify the stage-4 rebalancing criterion, the paper would be a solid empirical contribution. If they cannot, the central comparison is invalid and the appropriate outcome would be rejection. The LapGyn experiment provides some independent support but is too thin to carry the generalization claim alone. I would ask the editor to treat this as a load-bearing methodological point rather than a presentation fix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about surgical video benchmarks. The contribution is real: a unified 16-source corpus spanning 11 specialties and a 72-task evaluation taxonomy is the kind of artifact the field needs, comparable in role to Kinetics for general video. The fine-tuning protocol is also sane: 5:5 split with no clips from the same video in both train and test, and a frozen backbone during fine-tuning so the classifier measures representation quality. The LapGyn out-of-distribution experiment is a good-faith attempt to show transfer beyond the pretraining distributions.\n\nBut there is a load-bearing problem. Section 3.3 explains how pretraining clips and evaluation clips are formed, and never states that evaluation videos are excluded from pretraining. Table 2 lists both \"Pre-train Frames\" and \"Evaluation Frames\" for the same sources, and the two columns sum to the 59,742,875 total frames in Section 3.4, implying that essentially all frames from each source are used somewhere. If the eval clips come from the same videos the masked autoencoder saw during pretraining, the +7% top-1 and +7.9% top-3 gains over Kinetics reflect test-content familiarity, not transferable surgical representations. The vague \"final precise IID-oriented stage\" in Section 4 rebalances pretraining data toward the downstream distribution, which compounds the risk if any statistics from SurgBench-E are used there. The LapGyn result (a single task, no error bars) provides partial independent support, but it cannot validate the main SurgBench-E numbers.\n\nOther soft spots: frame counts disagree across the abstract (53M), Section 3.4 (59M), and Section 4 (74.4M); Table 7 lists labels 35-43 as \"home,\" which looks like a mapping error; there are no error bars or multiple seeds; and the dataset and code are \"available upon request,\" which is a weak release for a benchmark.\n\nThis is a promising benchmark with a credibility problem that is probably fixable. If the authors can confirm the pretraining and evaluation splits are video-disjoint, or release the code and data so reviewers can check, the paper deserves serious engagement. As is, the headline claim is unproven. I'd send it to a serious referee, but with a clear request to verify the split and release the artifacts.","headline":"A genuinely useful surgical video benchmark, but the paper never rules out that SurgBench-P includes SurgBench-E test frames, which would dissolve the headline +7% claim.","tokens_in":19393,"tokens_out":2861,"would_cite":false,"duration_ms":33501,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pretraining on 53 million surgical video frames outperforms natural-video pretraining on surgical tasks.","keywords":["surgical video understanding","video foundation models","self-supervised pretraining","VideoMAE","continual pretraining","evaluation benchmark","domain generalization","surgical phase classification"],"falsifier":"A decisive test is to run the same four-stage continual pretraining twice, once with the final upsampling and downsampling decided before any SurgBench-E annotation is consulted and once informed by SurgBench-E label frequencies; if both reach the same accuracy on an unseen procedure such as LapGyn4, the generalization claim stands, and if only the SurgBench-E-informed run keeps the 7.9% top-3 gain, the central claim reduces to distribution matching.","tokens_in":18209,"feed_emoji":"🩺","tokens_out":11652,"duration_ms":122959,"temperature":0.7,"pith_summary":"The paper introduces SurgBench, a combined pretraining dataset (SurgBench-P) and evaluation benchmark (SurgBench-E) for surgical video understanding. SurgBench-P pools about 53 million frames from 16 sources spanning 11 specialties and 22 procedures, and SurgBench-E turns 23,004 clips into 72 fine-grained tasks across six categories: phase classification, camera motion, tool recognition, disease diagnosis, action classification, and organ detection. The paper's central claim is that continually pretraining a VideoMAE model on SurgBench-P, starting from its Kinetics-400 weights, improves downstream accuracy by 7% top-1 and 7.9% top-3 relative to the Kinetics-400-pretrained model, and that the benefit carries over to an unseen gynecologic procedure. If the claim holds, diverse surgical-video pretraining provides a practical route to video models that transfer across unfamiliar surgical procedures and imaging modalities, which matters for automated decision support, skill assessment, and postoperative review.","feed_headline":"53M surgical video frames beat Kinetics pretraining on surgical tasks","feed_subtitle":"A 72-task benchmark shows surgery-specific pretraining improves accuracy across six video understanding categories.","key_machinery":"The load-bearing mechanism is a four-stage continual pretraining protocol applied to VideoMAE, a self-supervised masked-autoencoder video model that reconstructs randomly masked spatiotemporal tubes of patches (masking ratio 0.9). The stages move from all 225,250 collected clips, to filtering over-dominant sources such as A VOS, to upsampling underrepresented data, and finally to a 'precise IID-oriented stage with both upsampling and downsampling' that aligns the pretraining distribution with SurgBench-E. The evaluation side is the 6C-10S-72T taxonomy, which maps six task categories and ten sub-categories onto 72 finetuning tasks, allowing one backbone to be scored across phase, motion, tool, disease, action, and organ understanding.","core_discovery":"The paper claims that surgical video understanding is limited less by model architecture than by pretraining data: a model that has seen a broad mix of surgical video can keep learning transferable representations beyond what general video data provides. SurgBench operationalizes that mix by standardizing 16 heterogeneous sources into uniform clips and organizing an evaluation hierarchy of 6 categories, 10 sub-categories, and 72 tasks. With the same VideoMAE backbone, SurgMAE (continually pretrained on SurgBench-P) reaches 44.8% average top-1 and 73.1% average top-3 on SurgBench-E, against 37.8% and 65.2% for the Kinetics-400 baseline; the large variant reaches 48.7% and 78.8%. The paper also reports a 2.69% gain on LapGyn4, a gynecologic event-recognition dataset not seen in pretraining, as evidence that the learned representations generalize to unseen procedures and domains.","pith_inferences":["Editorial caution: the paper does not state what information guides the final 'precise IID-oriented stage' of pretraining; if that stage rebalances toward SurgBench-E label or statistic frequencies, part of the reported gain over Kinetics-400 would be test-set distribution matching rather than transferable surgical knowledge.","Reproduction will be harder than the abstract suggests: the appendix notes that 330 A VOS video URLs have expired (leaving 296 annotated videos), three restricted-license sources used in pretraining are not redistributed, and one private hospital dataset appears only on request.","The paper gives different SurgBench-P frame counts in different sections (53M, 56M, and 74.4M), so the exact preprocessing definition needs pinning down before the headline numbers can be compared across future work.","The authors' own Section 7 limitations, long-tail class distribution, no language-supervision integration, and unoptimized architectures, suggest that the benchmark's hardest open problems are rare-class recognition and multimodal understanding, not the core pretraining pipeline."],"forward_implications":["A single SurgBench-P-pretrained backbone can be fine-tuned into competitive models across six distinct surgical task families, reducing the need for task-specific pretraining pipelines.","The 72-task hierarchy gives researchers a common yardstick, so future surgical foundation models can be compared on the same sub-categories rather than on scattered single-task datasets.","The reported LapGyn4 gain indicates that continual pretraining on SurgBench-P transfers to procedures and video distributions not present in the pretraining corpus.","Model scale compounds the benefit: the large SurgMAE variant improves average top-1 by 3.9% and top-3 by 5.7% over the standard variant, so the benchmark can support scaling studies."],"supporting_citations":[{"why":"Supplies the VideoMAE masked-autoencoder architecture and the Kinetics-400 pretrained weights that serve as the baseline for continual pretraining.","marker":"Tong et al. (2022)"},{"why":"Provides Endo-FM, an earlier endoscopy-specific self-supervised foundation model that SurgBench compares against in coverage and evaluation.","marker":"Wang et al. (2023b)"},{"why":"Provides GSViT, an earlier general-surgery video pretraining model that motivates the broader specialty coverage of SurgBench-P.","marker":"Schmidgall et al. (2024)"},{"why":"Provides Surg-3M, a recent surgical pretraining dataset that Table 1 uses as a comparison for specialty and procedure coverage.","marker":"Che et al. (2025)"},{"why":"Supplies Cholec80, the source of the phase-classification and tool-presence tasks in SurgBench-E.","marker":"Yu et al. (2018)"},{"why":"Supplies CholecT45 and CholecT50, the source of the action-classification, phase, and instrument labels in SurgBench-E.","marker":"Nwoye et al. (2022)"},{"why":"Supplies AutoLaparo, the source of the hysterectomy phase-classification and camera-motion tasks.","marker":"Wang et al. (2022)"},{"why":"Supplies Hyper-Kvasir, the source of gastrointestinal lesion-diagnosis labels in SurgBench-E.","marker":"Borgli et al. (2020)"},{"why":"Supplies Kvasir-Capsule, the source of the capsule-endoscopy pathology labels used in the gastrointestinal diagnosis category.","marker":"Smedsrud et al. (2021)"},{"why":"Supplies JIGSAWS, the source of the robotic teaching-surgery action-recognition tasks.","marker":"Ahmidi et al. (2017)"}],"fun_headline_variants":["Surgical video pretraining beats Kinetics on 72-task benchmark","53M surgery frames outperform Kinetics for surgical video tasks","72-task surgical benchmark shows domain-specific pretraining wins","Surgical pretraining with 53M frames beats Kinetics on 72 tasks","Surgery-specific video pretraining improves accuracy on 72 tasks and unseen data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's load-bearing assumption is that the last pretraining stage, which adds and removes data to make the distribution look more balanced, is chosen using general data-quality rules rather than the evaluation benchmark's own labels or statistics; if that assumption fails, the reported gains over Kinetics-400 would mostly come from matching the test set instead of learning transferable surgical video knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Surgical video pretraining beats Kinetics on 72-task benchmark","53M surgery frames outperform Kinetics for surgical video tasks","72-task surgical benchmark shows domain-specific pretraining wins","Surgical pretraining with 53M frames beats Kinetics on 72 tasks","Surgery-specific video pretraining improves accuracy on 72 tasks and unseen data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001732,"raw_usage":{"total_tokens":6838,"prompt_tokens":925,"completion_tokens":5913,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":5822}},"tokens_in":541,"tokens_out":5913,"duration_ms":43005,"temperature":1.0,"reasoning_tokens":5822,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:30:02.973153+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test is to run the same four-stage continual pretraining twice, once with the final upsampling and downsampling decided before any SurgBench-E annotation is consulted and once informed by SurgBench-E label frequencies; if both reach the same accuracy on an unseen procedure such as LapGyn4, the generalization claim stands, and if only the SurgBench-E-informed run keeps the 7.9% top-3 gain, the central claim reduces to distribution matching.","supporting_citations":[{"cited_title":"I., Yu, T., Gonzalez, C., Seeliger, B., Mascagni, P., Mutter, D., Marescaux, J., and Padoy, N","cited_arxiv_id":null,"evidence_quote":"Supplies CholecT45 and CholecT50, the source of the action-classification, phase, and instrument labels in SurgBench-E."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies AutoLaparo, the source of the hysterectomy phase-classification and camera-motion tasks."},{"cited_title":"H., Hicks, S., Jha, D., Eskeland, S","cited_arxiv_id":null,"evidence_quote":"Supplies Hyper-Kvasir, the source of gastrointestinal lesion-diagnosis labels in SurgBench-E."},{"cited_title":"H., Thambawita, V., Hicks, S","cited_arxiv_id":null,"evidence_quote":"Supplies Kvasir-Capsule, the source of the capsule-endoscopy pathology labels used in the gastrointestinal diagnosis category."},{"cited_title":"B., Zappella, L., Khudanpur, S., Vidal, R., and Hager, G","cited_arxiv_id":null,"evidence_quote":"Supplies JIGSAWS, the source of the robotic teaching-surgery action-recognition tasks."}],"review_version":1}