{"id":"8b750531-e8a3-41bf-964f-2aa8daa619cf","arxiv_id":"2411.16794","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A new MSICS surgical video dataset and a phase-conditioned segmentation model that substantially improves tool segmentation, especially for rare and small instruments.","lead":"This paper introduces Sankara-MSICS, the first video dataset for manual small-incision cataract surgery, with pixel-level labels for 13 surgical tools and phase labels for 18 surgical steps. It also proposes ToolSeg, a method that uses surgical phase information and semi-supervised labels to improve tool segmentation by up to 38% over standard models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pseudo-label evaluation may leak test videos into training: §4.4 and Supp. §3.3 generate 20,878 pseudo-labeled frames from all 3,527 GT frames without fold-wise restriction, so SAM2-propagated labels from test videos can enter the training set and inflate v1/v4/v7.","rationale":"The reader's condition on predicted-phase accuracy is real and is acknowledged in the paper's Limitations, but it affects the practical deployment variant, not the internal validity of the method. The pseudo-label split issue is more load-bearing because it bears on whether the reported numbers themselves are trustworthy. The manuscript never states that the 20,878 pseudo-labels were generated separately for each fold; the Supplementary splits are defined only at the evaluation stage (Table 7), and Section 4.4 describes pseudo-label generation from the full base dataset. In the absence of an explicit statement, the manuscript as written does not rule out that pseudo-labels from test videos are available during training. This is a known leakage failure mode in video segmentation with temporal propagation. A clean audit would leave the reader's predicted-phase concern as the main residual issue; an unclean audit would invalidate the central improvement. I therefore recommend keeping the conditional verdict but making the leakage audit a required condition for acceptance.","tokens_in":17623,"tokens_out":5878,"duration_ms":57301,"concrete_test":"Request from the authors a per-fold audit of pseudo-label provenance: for each of the five folds, list any pseudo-labeled training frame whose source expert frame lies in a video assigned to that fold's test set. If any exist, rerun Table 1 variants v1, v4, and v7 with pseudo-labels generated only from that fold's train/validation videos (or with all test-video-derived pseudo-labels removed) and report mean DSC; a material drop in v7 (e.g., to near or below v6's 64.76) would confirm leakage. If the audit is impossible because code/data are not released, the semi-supervised claim should be marked unverified rather than accepted.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that SAM2 pseudo-labels improve tool segmentation (v0→v1, v6→v7 in Table 1) rests on an evaluation-protocol detail the manuscript never specifies. Section 4.4 and Supplementary §3.3 describe generating 20,878 pseudo-labeled frames 'from our base dataset of 3,527 annotated frames' by propagating each expert mask forward and backward 90 frames and sampling every 30th frame. Cross-validation is video-level (Supplementary Table 7), but nothing restricts pseudo-label generation to the training split of each fold, and the base 3,527 annotated frames include the test frames. For a typical test video, an expert mask in that video seeds pseudo-labels in neighboring unlabeled frames of the same video; if those pseudo-labels are used in training, the model has seen test-video content during training. This would inflate the pseudo-label contribution: v1 gains +7.63 DSC over v0 with no phase conditioning, and v7 gains +5.20 DSC over v6, both headline improvements. The manuscript must report fold-wise pseudo-label provenance before the semi-supervised results can be interpreted.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Sankara-MSICS, a new dataset of 53 manual small-incision cataract surgery videos with phase labels for 18 surgical phases and pixel-level annotations of 13 surgical tools in 3,527 frames. It proposes ToolSeg, a U-Net-like segmentation network with a phase-conditioned decoder using PAFT, DFBF, and CGate modules, and a semi-supervised extension that propagates ground-truth masks with SAM 2 to create 20,878 pseudo-labeled frames. On Sankara-MSICS, ToolSeg v7 (ground-truth phases + pseudo-labels) reaches 69.96 DSC versus 50.66 DSC for the U-Net baseline, and the method also improves over baselines on CaDIS. The authors benchmark several baselines, report ablations in Table 1, and provide per-tool results in Table 4.","tokens_in":17906,"tokens_out":3537,"duration_ms":184973,"significance":"If the reported results hold, this is a useful contribution to surgical video analysis. Sankara-MSICS appears to be the first pixel-level tool-segmentation dataset for MSICS, a procedure with high relevance in low-resource settings, and the phase-conditioned decoder is a sensible inductive bias given the strong phase-tool co-occurrence shown in Figure 2. The paper also provides a fairly complete ablation study, five-fold cross-validation at the video level, and a cross-dataset test on CaDIS. The SAM 2 pseudo-labeling pipeline is a reasonable way to use unlabeled frames without manual annotation. However, the central quantitative claims rest on a semi-supervised protocol whose evaluation details are not fully specified, and the practical benefit of phase conditioning depends on the relatively weak phase predictor (50.10% accuracy), which the paper reports but does not foreground. These issues need to be addressed before the reported margins can be taken at face value.","major_comments":[{"comment":"The pseudo-labeling procedure as described may leak test-video content into the training set. Section 4.4 generates 20,878 pseudo-labeled frames 'from our base dataset of 3,527 annotated frames' by propagating each expert mask forward and backward 90 frames and sampling every 30th frame. The five-fold cross-validation is video-level (Supplementary Table 7), but the text never states that pseudo-labels generated from annotated frames belonging to a fold's test videos are excluded from that fold's training set. Since the base 3,527 frames include the test frames, a SAM 2 propagation seeded by a test-video annotation can place frames from that same test video into training, even if the exact test frames are not duplicated. This would inflate the pseudo-label contributions in v1, v4, and v7. The manuscript must report fold-wise pseudo-label provenance, and ideally rerun the ablations with pseudo-labels restricted to training/validation videos, before the semi-supervised results can be interpreted.","section":"§4.4 and Supplementary §3.3, Table 7"},{"comment":"The headline '38.10% increase in mean Dice scores' refers to ToolSeg v7, which uses ground-truth phases as input. In a real deployment the phase source is MS-TCN++ predictions, and the corresponding variant with pseudo-data is v4, which reaches 62.70 DSC, a 23.77% relative gain over the v0 baseline. The paper does report v4 and v3 in Table 1, and the Limitations section acknowledges a gap between predicted and ground-truth phases, but the abstract and introduction emphasize the ground-truth-phase result without stating that it is an upper bound. I recommend making the predicted-phase variant the primary reported result, or at least clearly labeling the GT-phase variants as an oracle upper bound in the abstract and contribution list.","section":"Abstract and Table 1"},{"comment":"The comparison to ISINet and MATIS-Frame may be confounded by model capacity and training protocol. ISINet and MATIS are instance-segmentation methods trained for different settings, and MATIS-Frame in particular obtains only 18.24 DSC, which is far below U-Net. The paper attributes this to the ocular domain, but it is not clear whether these baselines were given the same training budget, learning rate, or data augmentation as ToolSeg, or whether they were fine-tuned from their original checkpoints. The text should state the exact training setup for each baseline so the comparison is fair and reproducible.","section":"§5.2, Table 3"}],"minor_comments":[{"comment":"The standard deviation for ToolSeg v4 is printed as '± 34.4' in the IoU column; this is almost certainly a typo for '± 4.4', matching Table 1.","section":"Table 3"},{"comment":"The sentence 'followed by U-Net with an IoU of 40.90 and DSC of 52.03' gives U-Net a DSC of 52.03, but Table 3 reports U-Net DSC as 50.66. The value 52.03 belongs to TernausNet.","section":"§5.2, text after Table 3"},{"comment":"The CaDIS experiment uses '13 tools across 18 surgical phases' without specifying how the 29 CaDIS classes were reduced to 13 or how the 18 phases were obtained for CaDIS. This selection should be described explicitly so the cross-dataset comparison is reproducible.","section":"§5.2, CaDIS paragraph"},{"comment":"The pseudo-label quality check reports an average DSC of 66.06 and IoU of 55.79 against expert labels on 654 common frames. This is a useful sanity check, but it would be helpful to report whether these 654 frames overlap with the training folds and whether low-quality propagations (e.g., the reflection artifact in Figure 8) are filtered before training.","section":"Supplementary §3.3"},{"comment":"There are several typos: 'complimentary' should be 'complementary', 'gasterointestinal' should be 'gastrointestinal', and the phrase 'with a notable boost for tools that are less prevalent and small' would read better as 'small and less prevalent tools'.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The pseudo-label leakage concern is, in my reading, a real ambiguity in the evaluation protocol rather than an artifact of the review pipeline: the manuscript's Section 4.4 and Supplementary Section 3.3 describe generating pseudo-labels from the full base dataset, while Table 7 makes clear that test videos are held out at the video level. The authors may have in fact restricted pseudo-labels by fold, but they do not say so. Because the v0→v1 and v6→v7 improvements are headline results, the omission is load-bearing. If the authors can confirm the fold-wise restriction or rerun with it, the paper would likely be acceptable after minor revision; as written, I cannot recommend acceptance until this is clarified. I would also encourage the authors to make the predicted-phase variant the main reported result, since the GT-phase numbers are an oracle upper bound that may mislead readers about deployment performance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the MSICS tool segmentation paper. The dataset is the real news: the first labeled MSICS dataset, 53 videos, 18 phases, 3,527 pixel-annotated frames, 13 tools, with a documented SAM-assisted annotation pipeline and video-level five-fold splits. That alone is worth having. The ToolSeg phase-conditioned decoder (PAFT/DFBF/CGate) is a reasonable extension of existing conditional segmentation ideas, and the CaDIS transfer results suggest the phase prior and the pseudo-labeling both generalize. The ablations in Table 1 are informative, and the per-tool table shows the expected pattern: phase helps phase-specific tools, pseudo-labels help rare tools.\n\nNow the soft spots, in proportion. The biggest one is the pseudo-label protocol. Section 4.4 says 20,878 pseudo-labeled frames are generated from the full base set of 3,527 annotated frames, with no statement that generation is restricted to the training split of each fold. Since the splits are video-level, a test video's expert mask can seed pseudo-labels for neighboring frames of that same video; if those pseudo-labels enter training, v1 and v7 are inflated. The manuscript must report fold-wise provenance. I don't see this as evidence of bad faith, but it is a load-bearing detail that the current text simply does not pin down.\n\nAlso worth noting: the phase predictor runs at 50.10% accuracy, so the practical gains with predicted phases (v2/v3) are smaller than the ground-truth-phase headline numbers. The paper does acknowledge this in Limitations, but the abstract's 23.77%-38.10% range is partly driven by GT phases. And a single-hospital dataset with no release plan limits external validation; the authors say code/data are not available, which is a real constraint for a dataset paper.\n\nThe math and comparisons look fine. The percentage improvements are relative, but the absolute DSC numbers are reported clearly enough that a careful reader can convert. The citation pattern is appropriate, including the self-cited systematic review that documents the MSICS gap.\n\nBottom line: this deserves a serious referee. The dataset contribution is solid and the method is a plausible advance, but the pseudo-label leakage question has to be resolved, ideally with a supplementary table showing fold-wise pseudo-label counts and a re-run of v1/v7 without test-video propagation. I'd also push for at least a data-access statement. If the leak is real, the headline gains shrink; if it isn't, the paper is a good addition to the surgical video literature.\n\nRecommendation: send to peer review with a request for clarification on the pseudo-label protocol and data availability.","headline":"Sankara-MSICS is a genuine and useful dataset contribution; the ToolSeg gains are plausible but the pseudo-label protocol needs fold-wise clarification before the headline numbers carry weight.","tokens_in":18470,"tokens_out":2009,"would_cite":true,"duration_ms":18301,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Phase information as a prior improves cataract tool segmentation by 38 percent.","keywords":["surgical tool segmentation","cataract surgery","MSICS","phase-conditioned segmentation","semi-supervised learning","SAM 2 pseudo-labels","surgical phase recognition","dataset"],"falsifier":"Evaluate ToolSeg v3 (predicted phases) and v6 (ground-truth phases) on a large, realistically distributed MSICS test set and compare their DSC to a plain U-Net; if the predicted-phase model does not beat the U-Net by a substantial margin, the practical benefit of phase conditioning under realistic phase errors is not established.","tokens_in":17430,"feed_emoji":"👁️","tokens_out":5564,"duration_ms":70772,"temperature":0.7,"pith_summary":"The paper introduces Sankara-MSICS, the first dataset of Manual Small-Incision Cataract Surgery (MSICS) videos, with 18 surgical phases and 13 pixel-level tool classes across 53 videos. It argues that knowing the surgical phase improves tool segmentation because phases and tools are strongly correlated, and proposes ToolSeg, a U-Net with a phase-conditional decoder, plus a semi-supervised step that uses SAM 2 to propagate expert masks to unlabeled frames. On Sankara-MSICS, ToolSeg raises mean Dice from 50.66 (U-Net baseline) to 69.96, a 38.10% relative gain, and also improves on the CaDIS dataset. The authors claim the gains are largest for small and less frequent tools.","feed_headline":"First MSICS dataset and phase-aware ToolSeg beat U-Net by 38%","feed_subtitle":"Surgical phase as a prior plus SAM 2 pseudo-labels lift mean Dice from 50.7 to 70.0 on the new Sankara-MSICS dataset.","key_machinery":"The central object is the Phase-informed Conditional Decoder (PCD) layer, inserted at each decoder level of a U-Net. It has three components: Phase-aware Affine Feature Transform (PAFT) applies per-phase channel-wise shift and scale ($f' = \\gamma_p \\odot f + \\beta_p$); Dynamic Feature Blending Factor (DFBF) computes a spatial blending factor $\\alpha = (H(f) + \\eta_p) / 2$; Context-Aware Adaptive Gating (CGate) fuses phase-modulated and original features as $f'' = f' \\cdot \\alpha + f \\cdot (1 - \\alpha)$. The other load-bearing mechanism is the semi-supervised pipeline: SAM 2, prompted with points sampled from expert masks, propagates masks to neighboring frames (every 30th frame up to 90 frames away), generating 20,878 pseudo-labeled frames from 3,527 manually labeled ones.","core_discovery":"On its own terms, the paper claims that surgical phase information is a usable prior for tool segmentation in cataract surgery: the same-looking tool can appear in different phases, and the network can disambiguate it when told which phase it is in. The evidence is a set of ablation variants on the new Sankara-MSICS dataset: conditioning the decoder on ground-truth phases (v6) lifts mean IoU from 40.90 to 56.13 and DSC from 50.66 to 64.76; adding SAM 2 pseudo-labels (v7) lifts IoU to 61.62 and DSC to 69.96. The paper further claims the method transfers to a different cataract setting (CaDIS), where ToolSeg v6 reaches 60.73 IoU / 68.63 DSC, outperforming the same baselines. The authors are careful to state that with predicted (not ground-truth) phases the gain is smaller, with v3 reaching 57.52 DSC, and that the phase predictor itself is only 50.10% accurate.","pith_inferences":["Because the practical gain with MS-TCN++-predicted phases (v3) is only 57.52 DSC versus 64.76 with ground-truth phases, the real-world benefit of phase conditioning hinges on phase-prediction accuracy; an end-to-end model that jointly predicts phases and segments tools could close part of this gap.","The phase–tool co-occurrence prior is likely to be weaker in complex or anomalous surgeries where tools are used out of their typical phase; testing ToolSeg on videos of complications would show whether phase conditioning degrades gracefully.","The SAM 2 pseudo-label pipeline with point prompting could be reused for other sparsely annotated surgical video datasets, but its quality depends on temporal coherence and the absence of specular reflections; the paper's own error examples (e.g., reflections from the surgeon lamp) suggest the propagation quality varies."],"forward_implications":["ToolSeg's phase-conditioned decoder plus SAM 2 pseudo-labels yields a 38.10% relative gain in mean DSC over a U-Net baseline on Sankara-MSICS, with the largest gains for rare and small tools, e.g., Rhexis Needle DSC jumps from 13.98 to 47.22.","Ablations show that both components contribute independently: phase conditioning with ground-truth phases raises DSC to 64.76, and pseudo-labels alone raise it to 58.29; together (v7) they reach 69.96.","Using only 50% of the manually annotated frames plus pseudo-labeled frames, the model reaches 65.92 DSC, exceeding the fully supervised baseline (64.76), indicating that the pseudo-labeling pipeline can halve annotation effort.","ToolSeg transfers to the CaDIS dataset (Phaco cataract surgery), where v6 achieves 68.63 DSC, outperforming U-Net (62.84) and other baselines."],"supporting_citations":[{"why":"U-Net encoder–decoder that serves as the base architecture and the primary baseline (v0).","marker":"[38]"},{"why":"MS-TCN++ supplies the predicted surgical phases used in ToolSeg variants v2–v4.","marker":"[27]"},{"why":"SAM 2 provides the mask propagation and prompting for pseudo-labeled frames.","marker":"[36]"},{"why":"CaDIS dataset used to test generalization of ToolSeg to Phaco cataract surgery.","marker":"[18]"},{"why":"MATIS transformer baseline benchmarked on both datasets.","marker":"[3]"},{"why":"ISINet instance segmentation baseline benchmarked on both datasets.","marker":"[17]"}],"fun_headline_variants":["Phase-aware tool segmentation boosts Dice by 38% on first MSICS dataset","New MSICS dataset plus phase-conditional decoder lifts tool segmentation","Surgical phase prior and SAM 2 pseudo-labels improve tool Dice by 38%","Phase-conditioned segmentation outperforms U-Net by 38% on Sankara-MSICS","ToolSeg: phase-informed network beats baselines on new cataract surgery data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's headline gains assume that each frame's surgical phase is known correctly; with the paper's own phase predictor (which is only 50.10% accurate), the improvement over a plain U-Net shrinks from 19.3 to 6.9 DSC points.","fun_headline_variants_meta":{"raw":{"variants":["Phase-aware tool segmentation boosts Dice by 38% on first MSICS dataset","New MSICS dataset plus phase-conditional decoder lifts tool segmentation","Surgical phase prior and SAM 2 pseudo-labels improve tool Dice by 38%","Phase-conditioned segmentation outperforms U-Net by 38% on Sankara-MSICS","ToolSeg: phase-informed network beats baselines on new cataract surgery data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000294,"raw_usage":{"total_tokens":1742,"prompt_tokens":1008,"completion_tokens":734,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":631}},"tokens_in":624,"tokens_out":734,"duration_ms":6970,"temperature":1.0,"reasoning_tokens":631,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:21:26.154361+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate ToolSeg v3 (predicted phases) and v6 (ground-truth phases) on a large, realistically distributed MSICS test set and compare their DSC to a plain U-Net; if the predicted-phase model does not beat the U-Net by a substantial margin, the practical benefit of phase conditioning under realistic phase errors is not established.","supporting_citations":[{"cited_title":"U- net: Convolutional networks for biomedical image segmen- tation","cited_arxiv_id":null,"evidence_quote":"U-Net encoder–decoder that serves as the base architecture and the primary baseline (v0)."},{"cited_title":"Ms-tcn++: Multi-stage temporal convolu- tional network for action segmentation","cited_arxiv_id":null,"evidence_quote":"MS-TCN++ supplies the predicted surgical phases used in ToolSeg variants v2–v4."},{"cited_title":"Cadis: Cataract dataset for surgical rgb-image segmentation","cited_arxiv_id":null,"evidence_quote":"CaDIS dataset used to test generalization of ToolSeg to Phaco cataract surgery."},{"cited_title":"Matis: Masked-attention transformers for surgical instrument segmentation","cited_arxiv_id":null,"evidence_quote":"MATIS transformer baseline benchmarked on both datasets."},{"cited_title":"Isinet: an instance-based approach for surgical instru- ment segmentation","cited_arxiv_id":null,"evidence_quote":"ISINet instance segmentation baseline benchmarked on both datasets."}],"review_version":1}