{"id":"e29fb848-d7b0-49a0-96d8-104d49690db5","arxiv_id":"2501.15326","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RASO recognizes surgical instruments and anatomy in images and video using a weakly supervised training pipeline built from automatically generated tag-image-text pairs from surgical lecture videos.","lead":"RASO is a surgical object recognition model trained on weakly supervised tags automatically extracted from 2,200 surgical lecture videos, and it reports higher zero-shot accuracy than prior methods on four surgical benchmarks. The paper also adds a temporal attention layer that makes video action recognition faster and more accurate.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot claim rests on unverified deduplication: WebSurg pretraining may contain the same IRCAD cholecystectomy videos used in CholecT50/Cholec80, so reported mAP gains could reflect memorization rather than generalization.","rationale":"The reader's formal weakest assumption was voiceover-tag visual grounding, which is a real label-noise concern but not the most load-bearing threat to the central claim. The reader's rationale, however, already mentions 'unexamined overlap between the WebSurg pretraining data and the cholecystectomy benchmarks.' I am elevating that overlap to the primary concern. The zero-shot claim is falsified only if test videos are absent from pretraining; the paper provides no evidence of this, and the institutional connection between WebSurg and IRCAD makes overlap plausible. This is more decisive than tag grounding because even perfectly grounded tags would not rescue the claim if the model has already seen the test frames. The appropriate verdict remains conditional: the paper should either provide a deduplication analysis or retrain after removing overlapping videos. I do not move the verdict because the concern is not yet proven, but it is the first condition to check.","tokens_in":13756,"tokens_out":6558,"duration_ms":61643,"concrete_test":"Run a video-level near-duplicate search between all WebSurg pretraining clips and all frames in the CholecT50, Cholec80, RSS, and GraSP test sets: extract one frame per second from each pretraining clip, embed with pHash or a DINOv2 feature, and flag any test video whose frames match a pretraining clip above a strict similarity threshold (e.g., mean cosine similarity > 0.95). Report the overlap count per benchmark. If any overlap exists, retrain RASO after excluding the overlapping clips and recompute Table 2; if mAP on CholecT50 or Cholec80 drops materially, the zero-shot claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that pretraining on 2,200 weakly supervised surgical lecture videos enables zero-shot gains on CholecT50, Cholec80, RSS, and GraSP. For that claim to hold, the pretraining corpus must not contain frames from the test videos. Section 4.1 states the data was 'sourced from platforms like WebSurg,' and WebSurg is the IRCAD e-surgical platform; Cholec80 and CholecT50 are laparoscopic cholecystectomy datasets produced by the IRCAD group. The paper reports no video-level deduplication, no exclusion of benchmark videos, and no analysis of overlap between the 901K pretraining frames and the benchmark test frames. If even a subset of WebSurg clips are the same recordings as CholecT50 or Cholec80 videos, the model has already seen those frames and their voiceover-derived tags during pretraining, so the 'zero-shot' evaluation becomes a memorization test, not a generalization test. This threatens the headline improvements directly on the two cholecystectomy benchmarks, and indirectly undermines the claim that weakly supervised data alone produces transferable surgical recognition. The concern is concrete and falsifiable: it is resolved by a deduplication check, not by an untestable appeal to annotation quality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RASO, a foundation model for open-set surgical object recognition built on the RAM architecture, trained with weakly supervised tag-image-text triplets automatically generated from 2,200 surgical lecture videos. The pipeline uses WhisperX ASR, a biomedical NER model, scene-graph parsing, and GPT-4o VLM annotations to produce 3.6 million tag annotations across 2,066 unique tags. RASO reports zero-shot mAP improvements of 2.9, 4.5, 10.6, and 7.2 points over prior methods on CholecT50, Cholec80, RSS, and GraSP respectively, and state-of-the-art supervised results on CholecT50 action-triplet recognition, including a temporal-attention fusion mechanism for video input.","tokens_in":14043,"tokens_out":5289,"duration_ms":44703,"significance":"If the zero-shot results hold under scrutiny, the paper makes a substantial contribution: it offers a scalable, low-annotation data-generation recipe, an open-vocabulary surgical recognition model, and a temporal fusion mechanism with a reported speed/accuracy advantage. The authors commit to releasing code, model, and data, which would aid reproducibility. However, the validity of the central zero-shot claim currently depends on two unverified points: the absence of overlap between the WebSurg pretraining corpus and the IRCAD cholecystectomy benchmarks, and the use of a genuinely zero-shot evaluation protocol rather than test-set-optimized thresholds. The supervised results and ablations are informative, but the headline claim needs these analyses before the paper can be accepted.","major_comments":[{"comment":"The zero-shot claim on CholecT50 and Cholec80 is threatened by possible corpus overlap. Section 4.1 states that pretraining data was sourced from 'platforms like WebSurg', and WebSurg is the e-surgical platform of IRCAD, the same group that produced the Cholec80 and CholecT50 laparoscopic cholecystectomy benchmarks. The paper reports no video-level deduplication, no exclusion of benchmark videos, and no analysis of frame-level overlap between the 901K pretraining frames and the benchmark test frames. If any WebSurg clips are the same recordings as the Cholec80 or CholecT50 videos, the reported zero-shot gains on those datasets would reflect memorization rather than generalization. This is directly testable: the authors should perform video- and frame-level near-duplicate detection between the pretraining corpus and the benchmark test sets, exclude overlapping content, and re-report the zero-shot results after exclusion.","section":"Section 4.1, Table 2"},{"comment":"The reported Precision, Recall, and F-beta values are not zero-shot because the decision threshold is fitted to the test set. Section 5.1 states: 'we apply the same rule to find the best threshold for each method which maximizes the Fβ=0.5 score.' This means each method's threshold is chosen using the test labels, so the F-beta and Precision/Recall numbers in Table 2 are oracle-threshold results, not predictions under a fixed zero-shot protocol. The threshold is a free parameter that is tuned on the evaluation data. The authors should fix thresholds on a validation set or report mAP, which is threshold-free, as the primary zero-shot metric; the current F-beta comparisons should be presented as upper bounds.","section":"Section 5.1, Table 2"},{"comment":"The text misstates the Cholec80 comparison for the no-fine-tuning variant. The paper says: 'even without fine-tuning with GPT-4o annotations, RASO (w/o FT) outperforms all other approaches across datasets, except for Cholec80, where it is marginally outperformed by SurgVLP.' Table 2 shows that on Cholec80, SurgVLP achieves mAP 33.3 and Fβ=0.5 31.1, whereas RASO (w/o FT) achieves mAP 25.0 and Fβ=0.5 27.6. This is a substantial gap, not a marginal one. The sentence should be corrected to accurately describe the quantitative results in the table.","section":"Section 5.2, Table 2"},{"comment":"The paper does not provide any evaluation of whether the automatically extracted voiceover tags are visually grounded in the frames they are paired with. The only quantitative annotation-quality check is for GPT-4o-generated annotations in Appendix A.1, Table 5, which reports agreement rates between 0.19 and 0.35 against clinician annotations; the definition of this agreement rate and the chance baseline are not stated. Since the entire pretraining signal comes from these tags, the authors should either include a sampled human evaluation of tag-image alignment for the speech-derived tags, or provide an analysis showing that the model's predictions are not driven by lexical shortcuts (e.g., tag co-occurrence with transcript words). Without this, the paper's central mechanism remains unverified.","section":"Sections 4.1, 4.2, Appendix A.1"}],"minor_comments":[{"comment":"There is a typo in the subsection heading: 'Evaluaiton Metrics' should be 'Evaluation Metrics'.","section":"Section 5.1"},{"comment":"The text says 'we choose Fβ=0.5 to balance the precision and recall', but β=0.5 weights precision more heavily than recall. The wording should say 'to emphasize precision over recall'.","section":"Section 5.1"},{"comment":"Figure 3 shows a tag with the note 'helmet trunk (manually removed)', indicating that manual filtering is applied, but the paper does not describe the manual filtering procedure, the number of tags removed, or the qualification of the people performing it. Please provide these details in the data generation section.","section":"Section 4.1 and Figure 3"},{"comment":"There are duplicate references for Nwoye et al. 2020 (a, b, c) that all point to the same paper. These should be consolidated to avoid citation confusion.","section":"References"},{"comment":"The abstract reports 3.6 million tag annotations and 2,066 unique tags, while Section 4.1 states 1,455 tags for pretraining and 611 for fine-tuning (totaling 2,066). It should be clarified whether the 3.6 million count includes both stages and how the fine-tuning tag set is constructed.","section":"Section 4.1"},{"comment":"The paper does not report variance or confidence intervals for the mAP improvements. Given the small test sizes (e.g., RSS has 997 images), the authors should provide error bars or significance tests for the main zero-shot comparisons.","section":"Table 2, Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper header indicates it was published at ICLR 2025; the journal submission should clarify the relationship to the conference version and what additional contributions are made. The most important risk is the WebSurg/IRCAD overlap issue, which can be resolved with a deduplication analysis; the test-set threshold fitting is also a serious protocol issue. The misstatement about the Cholec80 no-fine-tuning result should be corrected regardless of the outcome of the other revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper's real contribution is the weakly-supervised data generation pipeline: 2,200 surgical lecture videos, WhisperX transcripts, NER and scene graph parsing to extract tags, GPT-4o for additional annotations. That pipeline is genuinely new and the scale (901K frames, 2,066 tags) is impressive. It's also broadly applicable to other verticals with narrated video. The temporal fusion layer is a modest but sensible improvement over RAM. The supervised results on CholecT50 (57.5 mAP vs. Rendezvous 56.4, with far fewer epochs) are believable and in line with pretraining helping.\n\nThe soft spots are real. The biggest one: the pretraining data comes from WebSurg, which is the IRCAD e-surgical platform. Cholec80 and CholecT50 are IRCAD cholecystectomy videos. The paper reports no deduplication, no exclusion, no overlap analysis between the 901K pretraining frames and the benchmark test frames. If those videos overlap, the 'zero-shot' gains on the two cholecystectomy benchmarks are partly memorization, not generalization. This is a concrete, falsifiable concern, and it hits the headline numbers directly.\n\nSecond, the zero-shot evaluation selects the decision threshold per method and per benchmark on the test set to maximize F-beta. That inflates the F-beta columns and makes the comparison less clean. mAP is threshold-free, but for a paper claiming zero-shot, per-test-set threshold fitting is not the right protocol.\n\nThird, the text misstates the Cholec80 w/o FT comparison. Table 2 shows SurgVLP at 33.3 mAP vs. RASO w/o FT at 25.0. Calling that 'marginally outperformed' is wrong; it's a substantial gap. Also, there are no error bars, so the smaller gains (e.g., CholecT50 22.6 vs. 20.6) may be within noise.\n\nThe tag-frame alignment assumption is plausible but unvalidated. Frames paired with voiceover sentences may discuss structures that are not visible. The appendix evaluates GPT-4o tags against clinicians (agreement 0.19–0.35, which is modest), but not the core pretraining tag extraction.\n\nOverall the central architecture holds up; the data pipeline is the contribution, not the model. The paper deserves serious peer review because the pipeline is valuable and the issues are addressable. A deduplication check against CholecT50/Cholec80 and a proper zero-shot protocol (fixed threshold or classification-based evaluation) would settle the main question. I'd engage with it, but I wouldn't cite the zero-shot numbers until the overlap question is answered.","headline":"A genuinely useful data pipeline, but the zero-shot results on cholecystectomy benchmarks are not trustworthy until the WebSurg/IRCAD overlap is checked.","tokens_in":14535,"tokens_out":2984,"would_cite":true,"duration_ms":25089,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RASO claims that a weakly supervised pipeline, trained on tags mined from 2,200 surgical lecture videos, improves zero-shot mAP on four benchmarks by up to 10.6 points over prior methods.","keywords":["surgical object recognition","weakly supervised learning","zero-shot recognition","vision-language pretraining","temporal attention fusion","surgical lecture videos","open-set recognition","image tagging"],"falsifier":"Sample a set of lecture frames, have clinicians mark which tags are visibly present, and compare those against the pipeline's tags; if visible-grounding accuracy is low, or if shuffling the transcript-to-frame pairing leaves zero-shot mAP nearly unchanged, the reported gains would be explained by text-side shortcuts rather than visual recognition.","tokens_in":1666,"feed_emoji":"🏥","tokens_out":3493,"duration_ms":75083,"temperature":0.7,"pith_summary":"This paper claims that a surgical object recognizer can be pretrained entirely on weakly supervised data mined from narrated surgery videos, removing the need for manual annotation at scale. The model, RASO, is trained on 901K frames paired with 3.6 million tags extracted from voiceover transcripts and automated vision-language annotations. In zero-shot tests on four standard benchmarks, it improves mean average precision over prior surgical vision-language models by 2.9, 4.5, 10.6, and 7.2 points respectively. If the claim holds, large collections of instructional medical videos become a viable training source for open-set surgical recognition, which matters in a domain where annotated data is scarce and expensive to produce.","feed_headline":"Lecture-video tags lift zero-shot surgical mAP by up to 10.6","feed_subtitle":"Trained on 901K frames plus 3.6M tags mined from surgery lectures, it beats prior models on four benchmarks without manual annotations.","key_machinery":"The load-bearing object is the weakly supervised data generation pipeline: a data engine that collects surgery lecture videos, transcribes voiceovers with WhisperX, filters out slides, and samples frames; and a label engine that extracts surgical tags via biomedical named-entity recognition, parses verb-noun action triplets from sentences, and augments them with automated vision-language annotations, yielding tag-image-text triplets. Training uses a RAM-style architecture (RAM is a general image-tagging model) composed of a Swin image encoder, a frozen CLIP text encoder that embeds the tag list, a tag decoder, and a training-only text decoder that reconstructs captions to align visual and textual representations. The temporal-attention fusion layer aggregates frame features along time, which the paper shows improves verb recognition (74.5 vs 68.5 mAP) and overall video mAP, and the open-vocabulary mechanism generates embeddings for unseen tags on the fly.","core_discovery":"RASO establishes that weakly supervised tag-image-text triplets, generated automatically from 2,200 unannotated surgical lecture videos, are sufficient to train a foundation model that recognizes surgical objects in an open-set manner. The pipeline produces 2,066 unique surgical tags across 901K images, and a temporal-attention fusion layer extends the image tag decoder to video inputs. On zero-shot recognition, RASO reaches 22.6 mAP on CholecT50, 37.8 on Cholec80, 58.8 on RSS, and 20.0 on GraSP, exceeding the strongest prior baseline in each case. In the supervised CholecT50 triplet task, RASO reaches 57.5 mAP, above the Rendezvous baseline of 56.4, while the video mode is roughly four times faster than frame-by-frame inference. The paper presents these results as evidence that a weakly supervised data engine can substitute for manual annotation in a specialized visual domain.","pith_inferences":["If the transcript-to-frame alignment is reliable, scaling the pipeline to a much larger lecture-video corpus could push zero-shot accuracy further, and the resulting tags could seed a reusable surgical taxonomy for captioning, question answering, or grounded segmentation.","A direct testable extension would be to apply the same data and label engines to other video-rich clinical settings, such as endoscopy or interventional radiology, to see whether the weakly supervised gains transfer across procedure types.","The paper leaves open whether the zero-shot gains come primarily from the lecture-video pretraining or from the automated vision-language annotations added during fine-tuning; a per-tag breakdown of the fine-tuning contribution would isolate that effect."],"forward_implications":["Zero-shot surgical object recognition improves on four public benchmarks without manual annotations, so RASO can be applied directly to new surgical scenes and unseen classes.","The weakly supervised data engine lowers annotation cost: pretraining and fine-tuning complete in about eight hours on eight A6000 GPUs.","The temporal-attention fusion layer makes video recognition more accurate and faster than per-frame processing, reporting 61.3 vs 58.8 mAP and roughly four times lower latency.","Open-vocabulary inference lets the model recognize tags outside its 2,066-tag training vocabulary by embedding them with the frozen CLIP text encoder.","The same tag-image-text generation recipe is claimed to transfer to other vertical domains where narrated videos exist but manual annotations are scarce."],"supporting_citations":[{"why":"Base architecture: RASO builds on the RAM image-tagging model, inheriting its tag decoder and open-vocabulary design.","marker":"Zhang et al., 2024"},{"why":"Supplies WhisperX, the speech-to-text tool that produces time-aligned transcripts from surgical lecture videos for the data engine.","marker":"Bain et al., 2023"},{"why":"Defines the CholecT50 benchmark and the Rendezvous baseline that RASO surpasses in supervised triplet recognition.","marker":"Nwoye et al., 2022"},{"why":"Provides SurgVLP, the closest prior weakly supervised surgical vision-language baseline, which RASO outperforms in zero-shot tests.","marker":"Yuan et al., 2023"},{"why":"Supplies the frozen CLIP text encoder used to embed the tag list, including unseen tags at inference.","marker":"Radford et al., 2021"},{"why":"Defines the Cholec80 benchmark used for zero-shot phase and tool recognition evaluation.","marker":"Twinanda et al., 2016"},{"why":"Defines the GraSP benchmark used for multi-level surgical scene understanding in zero-shot evaluation.","marker":"Ayobi et al., 2024"},{"why":"Defines the EndoVis18/RSS robotic scene segmentation benchmark used for zero-shot evaluation.","marker":"Allan et al., 2020"}],"fun_headline_variants":["Surgical AI learns from lecture tags, boosts mAP by 10.6","Zero-shot surgical detection from 3.6M lecture-mined tags","Weak supervision from videos trains open-set surgical model","RASO: lecture videos replace manual annotation for surgery AI","2,200 surgery videos yield tags for open-set recognition"],"cache_read_input_tokens":16768,"weakest_assumption_plain":"The load-bearing premise is that each tag extracted from a surgeon's voiceover actually names something visible in the frame it is paired with, so the model learns visual recognition rather than lexical correlations.","fun_headline_variants_meta":{"raw":{"variants":["Surgical AI learns from lecture tags, boosts mAP by 10.6","Zero-shot surgical detection from 3.6M lecture-mined tags","Weak supervision from videos trains open-set surgical model","RASO: lecture videos replace manual annotation for surgery AI","2,200 surgery videos yield tags for open-set recognition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1415,"prompt_tokens":922,"completion_tokens":493,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":405}},"tokens_in":538,"tokens_out":493,"duration_ms":4408,"temperature":1.0,"reasoning_tokens":405,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:22:52.593972+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample a set of lecture frames, have clinicians mark which tags are visibly present, and compare those against the pipeline's tags; if visible-grounding accuracy is low, or if shuffling the transcript-to-frame pairing leaves zero-shot mAP nearly unchanged, the reported gains would be explained by text-side shortcuts rather than visual recognition.","supporting_citations":[{"cited_title":"Whisperx: Time-accurate speech transcription of long-form audio","cited_arxiv_id":null,"evidence_quote":"Supplies WhisperX, the speech-to-text tool that produces time-aligned transcripts from surgical lecture videos for the data engine."}],"review_version":1}