{"id":"263adfdf-1b37-4316-a4df-fda2537a602c","arxiv_id":"2502.09653","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SASVi uses a Mask2Former overseer to automatically re-prompt SAM2 during surgical videos, improving temporal consistency of segmentations from scarce annotations on Cholec80, CATARACTS, and Cataract1k.","lead":"SASVi couples the SAM2 video foundation model with a lightweight frame-wise object detection overseer that automatically re-prompts SAM2 whenever surgical instruments enter or leave the scene, producing temporally smoother segmentations on cholecystectomy and cataract videos. It matters because it offers a way to turn sparse frame-level annotations into full-video segmentation labels without additional manual labeling.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal consistency metrics in Table 2 measure self-consistency, not accuracy; the claim that SASVi's full-video masks are usable annotations is unvalidated because no generated mask is compared to ground truth on held-out frames.","rationale":"I examined whether the central claim can be true without assuming the temporal consistency metrics equal quality. It cannot. The paper has no ground-truth evaluation on the generated full-video masks, and the metrics are explicitly designed to measure smoothness, not correctness. The authors' own SAM2 (GT) comparison shows a ground-truth-prompted oracle can be worse on DiceOF than SASVi while better on CDT, which strongly suggests the headline metric is not monotonically related to accuracy. This is a correctness risk rather than a novelty dispute; the method itself is plausible and the appendices give useful compute and data-scarcity analyses. I agree with the reader's weakest assumption and see no need to move the verdict: CONDITIONAL remains the right call, with the condition being direct validation of sampled full-video masks against human annotations (plus the existing requests for code and error bars).","tokens_in":10700,"tokens_out":5453,"duration_ms":46491,"concrete_test":"Run SASVi on the held-out videos containing the test frames of CholecSeg8k, CaDISv2, and Cataract1k (split as in Section 4.2), and compare the SASVi masks at those exact frame indices against the ground-truth masks using per-class and macro Dice/IoU. Also report the same accuracy for the frame-wise Mask2Former and nnUNet baselines. If SASVi's accuracy is within a few points of the frame-wise overseer's semantic Dice (Table 1), the consistency gains are probably not bought by omission; if it is substantially lower, or if accuracy drops sharply on frames where a new tool enters, the full-video annotations require human validation before release.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the unstated identification of temporal consistency with annotation quality. Table 2's only metrics are self-consistency measures: DiceOF/IoUOF warp the previous predicted mask with optical flow and compare it to the next predicted mask; CDT/IoUT compare adjacent predicted masks directly. Neither checks whether the predicted masks match the true objects, and both can be inflated by a method that silently drops a newly entered tool and continues predicting static background, since the warped background then agrees with the next background prediction. The paper never reports a direct comparison of SASVi's output masks to the human ground-truth masks available in the held-out frames of CholecSeg8k, CaDISv2, and Cataract1k, despite splitting these datasets for evaluation in Section 4.2. The internal comparison worsens the concern: on Cholec80, SAM2 re-prompted with ground-truth masks achieves DiceOF 0.730 and CDT 2.879, while SASVi (Mask2Former) achieves DiceOF 0.754 and CDT 3.291; if a ground-truth-prompted oracle is not the best on the headline metric, the metric is not a reliable surrogate for accuracy. Hence the public full-video annotations, a central contribution, are not validated as usable ground truth.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SASVi, an inference-time re-prompting mechanism that couples SAM2 with a frame-wise object-detection and segmentation Overseer model (Mask R-CNN, DETR, or Mask2Former) trained on scarce annotated frames. When the detected class composition in a video changes, SASVi tracks back to the change point, extracts anchor prompts from the Overseer's mask, and re-prompts SAM2 so that the video continues to be segmented with temporally smooth and semantically complete masks. The authors evaluate per-frame detection and segmentation on CholecSeg8k, CaDISv2, and Cataract1k, report temporal-consistency results for full videos in Table 2, and describe publicly released full-video masks that they generate for Cholec80, CATARACTS, and Cataract1k.","tokens_in":10949,"tokens_out":4042,"duration_ms":34448,"significance":"If the validation were complete, the contribution would be practically valuable: it would reduce annotation cost for video segmentation, provide a reusable baseline for deploying SAM2 in surgical domains, and enlarge the available annotations for three surgical video datasets. The per-frame Overseer results in Table 1 are clearly specified and support the choice of Mask2Former as the main Overseer, and the pseudocode plus appendices on compute and annotation sparsity make the method easy to reproduce once code is released. However, the headline temporal-consistency claim rests on self-consistency metrics that are never validated against ground-truth masks on the held-out frames that were explicitly set aside in Section 4.2, so the significance of the claimed full-video annotations is currently conditional on additional evidence.","major_comments":[{"comment":"The headline improvement is not statistically supported. All entries in Table 2 are single point estimates with no confidence intervals, per-video error bars, or significance tests, and the abstract's claim of improvement 'by at least 2.4%' is not tied to any specific compared pair in the table. Given that the test portion consists of only a few videos (Section 4.2), the variability is likely substantial, so the statement that re-prompting 'significantly improves' temporal consistency cannot be evaluated from the reported results.","section":"Section 4.4, Table 2"},{"comment":"The temporal-consistency metrics measure self-consistency, not accuracy. DiceOF/IoUOF compare predictions across adjacent frames after optical-flow warping, and CDT/IoUT compare adjacent predictions directly; none of these metrics compares a predicted mask to a human annotation. The paper defines held-out annotated frames in the test splits in Section 4.2 but never reports SASVi's full-video output masks evaluated against these ground-truth masks. Consequently, the central contribution—that the released full-video masks for Cholec80, CATARACTS, and Cataract1k are usable annotations—is not validated.","section":"Sections 4.4 and 5"},{"comment":"The internal evidence contradicts the use of DiceOF as an accuracy surrogate. On Cholec80, SAM2 (GT), which is re-prompted with ground-truth masks whenever they are available, achieves DiceOF 0.730 and CDT 2.879, while SASVi (Mask2Former) achieves DiceOF 0.754 and CDT 3.291. If ground-truth prompting is not the best method on the headline metric, the metric cannot be relied on to rank methods by true segmentation quality; this reinforces the need for direct accuracy evaluation of the generated masks.","section":"Section 4.4, Table 2"}],"minor_comments":[{"comment":"The anchor sampling size na is required by Algorithm 1 but is never specified or ablated, and the temporal change threshold nt is given only as an empirical value (nt = 4) with no sensitivity analysis; both should be documented or experimentally justified.","section":"Section 3.3, Algorithm 1"},{"comment":"The phrase 'by at least 2.4%' should state explicitly which metric and which method pair it refers to; as written, it is not traceable from Table 2.","section":"Section 4.4"},{"comment":"The paper correctly acknowledges that SASVi's performance depends on the Overseer quality, and Table C1 shows that this dependence is strong, but no error-propagation analysis or comparison with direct accuracy on held-out frames is provided; this limitation should be discussed in the main text or addressed experimentally.","section":"Appendix C"},{"comment":"The dataset name 'CadISv2' should be spelled 'CaDISv2' for consistency with Section 4.1 and the reference list.","section":"Section 4.4, paragraph 2"},{"comment":"Reporting only macro-averaged Semantic Dice hides per-class behavior; adding per-class Dice or a confusion analysis would help identify which semantic classes drive the reported improvements.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The main missing evidence is a direct comparison of SASVi's generated full-video masks against the held-out ground-truth frames that the paper already sets aside. If the authors add that evaluation, together with error bars or significance tests on the Table 2 results, I would be comfortable with the revised manuscript. The released annotations should be clearly described as model-generated and validated before they are promoted as large-scale public annotations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe re-prompting loop is a genuine incremental idea, but the paper's headline evidence for it is weaker than the writing suggests. The authors have SAM2 re-prompted by a Mask2Former overseer that detects when the set of visible instrument classes changes, and re-prompts with masks and anchor points, backtracking a few frames. That specific mechanism is not in Surgical-DeSAM or Surgical SAM2, and it is clearly described. Table 1's per-frame results are solid, and the compute analysis shows the overhead is modest.\n\nThe soft spot is the validation of temporal consistency and the released full-video annotations. Table 2's DiceOF/IoUOF/CDT/IoUT are self-consistency metrics: they compare adjacent predicted masks, not predictions to ground truth. The internal check is damning: on Cholec80, SAM2 prompted with ground-truth masks scores 0.730 DiceOF while SASVi scores 0.754. If a ground-truth-prompted oracle is not best on the headline metric, the metric is not a reliable surrogate for accuracy. And the paper never compares SASVi's output masks to the human masks available in the held-out frames, despite splitting these datasets for evaluation. That is load-bearing because the public full-video annotations are a central contribution; they are generated by the same method being proposed and could be systematically wrong.\n\nThere are smaller problems: no error bars or significance tests for Table 2, the abstract claims 'at least 2.4%' while another version says 1.5%, the code is a placeholder, and the sensitivity of the re-prompt threshold nt is not explored. None of these are fatal by themselves, but combined they mean the paper currently overstates what is shown.\n\nWhat the paper does well: the idea is simple, the pipeline is reproducible in principle, the per-frame numbers are decent, and the data scarcity analysis in Appendix E is useful. The authors are honest in Appendix C about dependence on the overseer.\n\nMy bottom line: this deserves a serious referee, but the referee should ask for validation of a sample of the generated masks against human annotations, error bars on Table 2, and a corrected abstract. The method is a reasonable baseline for surgical video segmentation with scarce labels; the release of full-video masks could be valuable if validated. For my own work, I would not cite it until those numbers are checked.","headline":"Sensible incremental re-prompting idea, but temporal consistency is validated with self-consistency metrics that fail an oracle check and the released annotations are never compared to ground truth.","tokens_in":11494,"tokens_out":3131,"would_cite":false,"duration_ms":25721,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SASVi couples SAM2 with a frame-wise Overseer that automatically re-prompts the model whenever surgical tools enter or leave the scene, yielding temporally consistent full-video segmentations from scarce annotations.","keywords":["Surgical video segmentation","Foundation models","SAM2","Re-prompting mechanism","Temporal consistency","Mask2Former","Semantic segmentation","Cholec80"],"falsifier":"Annotate a random sample of frames from the released full-video masks, especially frames immediately before and after the Overseer triggers a re-prompt, and compare human semantic segmentation to SASVi's masks with standard Dice/IoU. If the masks score high on optical-flow-warped consistency but low on expert-verified semantic accuracy at those transition points, the central claim that SASVi yields complete and accurate temporally consistent segmentations would be falsified; if expert Dice matches the reported consistency gains, the claim is supported.","tokens_in":10511,"feed_emoji":"🎥","tokens_out":7373,"duration_ms":56547,"temperature":0.7,"pith_summary":"The paper proposes SASVi, a way to make SAM2, a foundation model for video object segmentation, usable on surgical videos without per-frame manual prompting. The key idea is to run a lightweight frame-wise detector/segmenter, called the Overseer, alongside SAM2; when the set of visible instruments changes for a few consecutive frames, SASVi backtracks to the first changed frame, samples prompt points from the Overseer's mask, and re-prompts SAM2 to keep tracking. The authors report that this re-prompting improves temporal consistency by at least 2.4 percentage points over single-frame prompting and frame-wise baselines on Cholec80, CATARACTS, and Cataract1k, and they release full-video segmentations derived from sparse annotations. If correct, the method turns a scarce-annotation setting into dense video annotations for an entire domain with no extra manual labelling.","feed_headline":"Overseer re-prompts SAM2 to keep surgical masks temporally smooth","feed_subtitle":"A lightweight detector trained on sparse labels yields full-video masks for three surgical datasets.","key_machinery":"The central mechanism is the re-prompting loop: an Overseer model (Mask R-CNN, DETR, or Mask2Former, with Mask2Former chosen as the default) is trained on scarce segmentation labels and runs in parallel with SAM2 on every frame, predicting classes, boxes, and masks. A buffer stores recently predicted class sets; when the set of classes stays different from the initial buffer for more than $n_t = 4$ frames, SASVi goes back to the first changed frame $t' - n_t$, samples anchor points from the Overseer's mask for each present entity, and uses those prompts plus the mask to re-initialise SAM2, continuing forward. The backtrack-and-re-prompt step is what lets the pipeline correct SAM2's drift when objects leave the scene or new tools appear, and the frame-wise Overseer supplies the semantic class identity that SAM2 itself lacks.","core_discovery":"SASVi claims that a cheap, frame-wise detection model trained on a small annotated subset can act as an automated re-prompting supervisor for SAM2, so that the foundation model's memory-based temporal smoothing is combined with domain-specific semantic knowledge of which instruments are present. Whenever the detected class set changes for more than $n_t = 4$ consecutive frames, the pipeline tracks back to the frame where the change began, samples anchor points from the Overseer's segmentation mask, and restarts SAM2 from there; this corrects drift and picks up newly entered or departed tools. On three surgical datasets the resulting masks achieve higher optical-flow-warped Dice/IoU and lower contour distances than frame-wise nnUNet, Mask2Former, Surgical-DeSAM, and SAM2 prompted only once, and the authors use the pipeline to generate and publicly release complete masks for the full videos behind CholecSeg8k, CaDISv2, and the Cataract-1k segmentation subset.","pith_inferences":["If the temporal-consistency metrics are faithful, the released masks could serve as pseudo-labels for training video segmentation models on rare instruments, potentially mitigating class imbalance in surgical data science.","The re-prompting loop is model-agnostic on the foundation side: the same Overseer-triggered backtrack-and-re-prompt pattern could be applied to other video tracking or segmentation models beyond SAM2, and possibly to non-surgical scenes with similar composition changes.","A testable extension is to validate the generated full-video masks against a small set of human-annotated frames sampled at re-prompting moments, to separate genuine tracking quality from metric artifacts."],"forward_implications":["SASVi produces full-video semantic masks for the Cholec80, CATARACTS, and Cataract1k counterparts of the sparse annotated subsets, and the authors release those masks publicly.","A lightweight Overseer re-prompting SAM2 yields temporally smoother and more complete segmentations than frame-wise models, one-time prompting, and similar re-prompting baselines on the three datasets.","The method needs only a small number of annotated frames per domain (around 1% of the video frames) to generate dense annotations, making it a practical annotation-extension tool.","The authors position SASVi as a baseline for future temporally consistent surgical video segmentation work."],"supporting_citations":[{"why":"SAM2 is the foundation model whose memory-based temporal segmentation SASVi re-prompts; its behavior with scene changes motivates the whole method.","marker":"[12]"},{"why":"Mask2Former is the chosen Overseer backbone, supplying frame-wise class, box, and mask predictions used for re-prompting.","marker":"[16]"},{"why":"CholecSeg8k provides the scarce segmentation annotations used to train the cholecystectomy Overseer.","marker":"[22]"},{"why":"CaDISv2 provides the scarce segmentation annotations used to train the cataract Overseer on CATARACTS videos.","marker":"[23]"},{"why":"Cataract-1k segmentation subset provides the scarce annotations for the third domain and the target full videos for annotation extension.","marker":"[24]"},{"why":"Introduces the optical-flow warping approach used to define the Dice_OF and IoU_OF temporal consistency metrics.","marker":"[26]"},{"why":"Applies the same optical-flow warping for temporal consistency evaluation in surgical sequence translation, supporting the metric choice.","marker":"[27]"},{"why":"Surgical-DeSAM is a frame-wise SAM+DETR re-prompting baseline that SASVi compares against.","marker":"[7]"},{"why":"Cholec80 is the large-scale counterpart video dataset whose full annotations SASVi generates.","marker":"[3]"},{"why":"CATARACTS is the large-scale counterpart video dataset whose full annotations SASVi generates.","marker":"[2]"}],"fun_headline_variants":["SASVi: auto re-prompt SAM2 on scene change for smooth surgical masks","SASVi: lightweight Overseer re-prompts SAM2 for surgical video consistency","Re-prompting on scene shifts: SASVi makes SAM2 track surgical tools smoothly","Temporal smoothness from sparse labels: SASVi re-prompts SAM2 per scene change","SASVi: scene-change re-prompting keeps SAM2 masks consistent in surgery"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The temporal-consistency metrics (optical-flow-warped Dice/IoU and contour distance) are treated as proxies for segmentation quality on full videos where no ground-truth masks exist; if a mask is smooth but semantically wrong, for example silently dropping a newly entered instrument, those metrics can still score it well, so the reported advantage over frame-wise baselines would not prove that the generated full-video annotations are accurate.","fun_headline_variants_meta":{"raw":{"variants":["SASVi: auto re-prompt SAM2 on scene change for smooth surgical masks","SASVi: lightweight Overseer re-prompts SAM2 for surgical video consistency","Re-prompting on scene shifts: SASVi makes SAM2 track surgical tools smoothly","Temporal smoothness from sparse labels: SASVi re-prompts SAM2 per scene change","SASVi: scene-change re-prompting keeps SAM2 masks consistent in surgery"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00073,"raw_usage":{"total_tokens":3298,"prompt_tokens":1008,"completion_tokens":2290,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":2179}},"tokens_in":624,"tokens_out":2290,"duration_ms":15009,"temperature":1.0,"reasoning_tokens":2179,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T11:04:02.743160+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate a random sample of frames from the released full-video masks, especially frames immediately before and after the Overseer triggers a re-prompt, and compare human semantic segmentation to SASVi's masks with standard Dice/IoU. If the masks score high on optical-flow-warped consistency but low on expert-verified semantic accuracy at those transition points, the central claim that SASVi yields complete and accurate temporally consistent segmentations would be falsified; if expert Dice matches the reported consistency gains, the claim is supported.","supporting_citations":[{"cited_title":"In: CVPR, pp","cited_arxiv_id":null,"evidence_quote":"Mask2Former is the chosen Overseer backbone, supplying frame-wise class, box, and mask predictions used for re-prompting."},{"cited_title":"MedIA 71, 102053 (2021)","cited_arxiv_id":null,"evidence_quote":"CaDISv2 provides the scarce segmentation annotations used to train the cataract Overseer on CATARACTS videos."},{"cited_title":"In: ICCV, pp","cited_arxiv_id":null,"evidence_quote":"Introduces the optical-flow warping approach used to define the Dice_OF and IoU_OF temporal consistency metrics."},{"cited_title":"IJCARS 18(7), 1217–1224 (2023)","cited_arxiv_id":null,"evidence_quote":"Applies the same optical-flow warping for temporal consistency evaluation in surgical sequence translation, supporting the metric choice."},{"cited_title":"IJCARS, 1–5 (2024)","cited_arxiv_id":null,"evidence_quote":"Surgical-DeSAM is a frame-wise SAM+DETR re-prompting baseline that SASVi compares against."},{"cited_title":"IEEE transactions on medical imaging 36(1), 86–97 (2016)","cited_arxiv_id":null,"evidence_quote":"Cholec80 is the large-scale counterpart video dataset whose full annotations SASVi generates."},{"cited_title":": Cataracts: Challenge on automatic tool annotation for cataract surgery","cited_arxiv_id":null,"evidence_quote":"CATARACTS is the large-scale counterpart video dataset whose full annotations SASVi generates."}],"review_version":1}