{"id":"896babab-506b-4ab5-8cb8-c4804f95fc74","arxiv_id":"2507.04317","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A CLIP-based encoder with RL residual refinement and curriculum learning reaches 81% mIoU on EndoVis 2018 and 74.12% on EndoVis 2017 surgical segmentation.","lead":"This paper combines a frozen CLIP image encoder with a reinforcement-learning refinement module and curriculum learning to segment surgical instruments and tissues in endoscopic video. The authors report state-of-the-art mean IoU scores on the EndoVis 2018 and 2017 benchmarks, but the mechanism, code, and evaluation details are incompletely specified.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim depends on benchmark rows whose provenance is broken: reference [13] points to a relighting S3Net, and the asterisked U-Net and SegFormer rows are reported without protocol details.","rationale":"The reader's weakest assumption correctly identifies the most load-bearing condition: the benchmark comparison must be valid for the abstract's claim 'CLIP-RL achieved a mean IoU of 81%, outperforming state-of-the-art models' to hold. My independent reading finds no internal contradiction that would overturn the reported numbers, but the provenance gap is concrete and checkable. The citation mismatch for S3Net is visible in the reference list, and the asterisked baselines lack any protocol description, so the comparative claim is not verifiable from the preprint. I considered whether the undefined RL reward and policy-gradient loss constitute a stronger concern; they make the method unreproducible, but they do not by themselves falsify the empirical claim. The benchmark provenance issue is more directly load-bearing because a single mismatched baseline row or protocol difference would invalidate the central comparison. Because the reader already assigned CONDITIONAL with low confidence and identified the same weakest assumption, no verdict adjustment is needed; the conditional verdict appropriately holds until code or detailed comparison protocols are released.","tokens_in":10146,"tokens_out":3386,"duration_ms":40434,"concrete_test":"Independently reproduce the comparison: locate the actual surgical-segmentation S3Net paper and verify whether Table I's 71.99 mIoU matches its reported EndoVis 2017 result; then rerun U-Net and SegFormer with the same train/validation split, 224x224 resolution, and evaluation code. If the S3Net row is from a relighting paper, or if reproduced baseline mIoUs differ by more than 1 point, the outperformance claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract; Tables I and II) that CLIP-RL outperforms state-of-the-art models rests on the validity of the comparison rows. Two conditions are insecure. First, Table I's S3Net row (mIoU 71.99) is cited to reference [13], arXiv:2105.00681, which is 'S3Net: A Single Stream Structure for Depth Guided Image Relighting' -- not a surgical segmentation network. The source of the 71.99 value is therefore untraceable, so the second-best specialist baseline cannot be checked. Second, the asterisked rows in Table II (U-Net*, SegFormer*, and CLIP-RL*) are described only as 'reported by us,' with no training protocol stated: the validation split is only '20% of the dedicated training dataset, split randomly,' no image resolution or class-averaging detail is given for the baselines, and no evaluation code is released. Since the conclusion that 0.81 beats 0.75, 0.65, and 0.61 depends on these rows being produced under the same training/test protocol, validation split, and evaluation code as CLIP-RL, a mismatch in any of these would change the headline. The other weaknesses noted in the paper -- undefined RL reward and advantage, single-run results without error bars, and missing code or data -- are real, but they do not independently break the comparative claim as sharply as the missing provenance of the comparison numbers themselves. If either provenance gap is real, the 'outperforms SOTA' claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes CLIP-RL, a surgical scene segmentation framework that combines a frozen CLIP encoder with a lightweight decoder, a reinforcement-learning-based residual refinement module, and curriculum learning. The method is evaluated on the EndoVis 2017 and EndoVis 2018 benchmarks, reporting a mean IoU of 74.12% and 81%, respectively, and the authors claim that these results outperform existing state-of-the-art models. The paper also includes an ablation study showing incremental gains from curriculum learning and the RL module. The central methodological ingredients are described in Section III, with the architecture and loss defined in Equations (1)-(4), and results are presented in Tables I-IV.","tokens_in":10455,"tokens_out":4595,"duration_ms":45406,"significance":"If the reported results are reproducible, the idea of combining a frozen CLIP backbone with RL-based residual refinement and curriculum learning is a potentially useful contribution to surgical scene segmentation, and the per-class improvements on instruments and soft tissue are noteworthy. However, the current manuscript does not provide sufficient evidence for the central 'outperforms SOTA' claim. The comparison rows in Tables I and II lack provenance or protocol details, the RL module is under-specified, and all results are single-run point estimates without error bars. The paper would be substantially strengthened by releasing code and data, specifying the evaluation protocol, correcting the S3Net citation, and reporting variance over multiple runs.","major_comments":[{"comment":"The S3Net row in Table I (mIoU 71.99) is cited to reference [13], arXiv:2105.00681, which is 'S3Net: A Single Stream Structure for Depth Guided Image Relighting,' not a surgical segmentation method. The provenance of the 71.99 value is therefore untraceable, and because this row is one of the strongest specialist baselines on EndoVis 2017, the comparison claim is not verifiable. Please either correct the citation to the actual surgical segmentation S3Net paper and report the source of the number, or re-run the baseline under the same protocol and report that protocol.","section":"Table I / Reference [13]"},{"comment":"The asterisked rows in Table II (U-Net*, SegFormer*, CLIP-RL*) are described only as 'reported by us.' The text gives no training or evaluation protocol for these baselines: no image resolution, optimizer, number of epochs, loss function, validation split details, or evaluation code. Since the headline result (0.81 vs 0.75 for SegFormer, 0.65 for AdaptiveSAM, 0.61 for nn-UNet) depends on these rows being produced under identical train/validation/test splits and evaluation metrics, the comparison is currently unsupported. Please specify the full protocol for all self-reported rows and release the evaluation script.","section":"Table II / Section IV.A"},{"comment":"The RL component is not fully specified. Equation (2) defines the refined output as O = sL + α·r, but neither the reward function nor the baseline used in the policy-gradient loss is defined anywhere; the text only says that the advantage is 'the reward minus a running baseline.' Moreover, the action space is only given as an example ({−0.1, 0.0, +0.1}), and it is unclear whether α is sampled from a categorical distribution or a continuous one. Without these definitions, the claim that the RL module 'dynamically refines predictions' is not reproducible. Please provide the exact reward, baseline, action distribution, and how gradients flow through the sampling step.","section":"Section III.B-C / Equations (2)-(4)"},{"comment":"There are internal inconsistencies in the architectural description. Section III.B states the input resolution is 244×244 and that upsampling starts from a 14×14 feature map, while Section III.C states images are resized to 224×224 and upsampling starts from 16×16. The introduction (Section I) calls the encoder a 'ResNet-based CLIP model,' whereas Section III.B/C describes a CLIP vision transformer. Also, Section III.C says the softmax yields 'voxel-wise probability distributions' for what are 2D images. Please reconcile these statements so the architecture is unambiguous.","section":"Section III.B vs III.C"},{"comment":"All quantitative results are reported as single-run point estimates with no error bars or significance tests. Given that the reported gap to SegFormer on EndoVis 2018 is 6 mIoU points and the per-class gaps are often smaller, run-to-run variance could affect the conclusions. Please report results over multiple seeds (mean ± std) and, if possible, statistical significance. In addition, Table IV does not state which dataset the ablation was performed on, and Table III's claim of 'highest mIoU in 8 out of 11 classes' is inconsistent with the table, which shows 9 out of 11 classes.","section":"Tables I-IV / Section IV"}],"minor_comments":[{"comment":"The notation UPL in Equation (1) is not defined and conflicts with the later use of UPl; please standardize the notation for the upsampling operators.","section":"Equation (1)"},{"comment":"The comparison between CLIP and DINOv2 is attributed to references [7] and [9], which appear to be self-citations or unrelated to the specific benchmark claim; please cite primary sources and verify the numbers.","section":"Section II.D"},{"comment":"Figure 2 is described qualitatively but no quantitative details are given for the selected examples; please specify which frames are shown and whether they are representative or adversarial cases.","section":"Section IV.A"},{"comment":"The reference list contains formatting inconsistencies, including incomplete entries (e.g., [22]) and potentially incorrect author names (e.g., [27]); please check all references against the original sources.","section":"References"},{"comment":"The phrase 'This two-task setting is critical' is unclear because the paper evaluates on two datasets rather than two joint tasks; please clarify what the two tasks are and how they relate.","section":"Section IV"}],"recommendation":"major_revision","confidential_remarks":"The provenance issue with reference [13] is serious and should be resolved before publication; the authors should be asked to provide the exact source of the S3Net number. The self-reported baselines in Table II also need a complete protocol and, ideally, shared evaluation code. If these provenance gaps cannot be closed, the 'outperforms SOTA' claim should be weakened or removed. The RL component, as currently specified, is not reproducible and needs a precise definition of reward, baseline, and action distribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The architecture is a reasonable engineering combination, and the ablation suggests each added component helps. But the headline 'outperforms SOTA' claim is currently unverifiable: the S3Net baseline is cited to a depth-guided relighting paper, and the asterisked baseline rows are self-reported without protocol details. That is a load-bearing problem for the main empirical claim.\n\nWhat's new is the specific integration of a frozen CLIP ViT encoder, a lightweight decoder, a small RL-based residual scaling module (action set like {-0.1, 0, +0.1}), and a curriculum schedule on the loss weights. None of these components is new in itself, so the novelty is incremental, but the combination is plausible and the paper describes it with enough clarity to re-implement. The ablations (72.4 → 76.8 → 81.0 mIoU) provide a useful internal control, and the qualitative failure case on the Thread class is honestly reported.\n\nThe soft spots are mostly in the evidence. Reference [13] points to 'S3Net: A Single Stream Structure for Depth Guided Image Relighting,' not the surgical segmentation S3Net whose 71.99 mIoU appears in Table I; that number is untraceable. The asterisked rows (U-Net*, SegFormer*) are described only as 'reported by us,' with no training protocol, random split details, resolution, or evaluation code. Without that, the 0.81 vs 0.75 margin cannot be checked. The RL reward and advantage baseline are never specified, so the policy-gradient loss is not reproducible. There are also minor internal inconsistencies: input resolution is 224×224 in the implementation section but 244×244 in the methods and figure; the intro says ResNet-based CLIP while the methods describe a ViT; and 'voxel-wise probabilities' is odd for 2D image segmentation. All results are single runs with no error bars. These issues are addressable, but they compound the reproducibility problem.\n\nWho is this for? Someone deciding whether CLIP+RL residual scaling is worth trying on surgical video segmentation. If the numbers reproduce, it is a solid incremental result, not a paradigm shift. The paper deserves a serious referee, but with a major-revision expectation: release code or a detailed protocol, define the reward, correct the references, and provide multi-seed statistics. I would not cite it as a SOTA reference in its current form.","headline":"Promising architecture, but the 'outperforms SOTA' claim rests on comparison rows with broken provenance; worth a careful major revision, not a reject.","tokens_in":10991,"tokens_out":4051,"would_cite":false,"duration_ms":39312,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLIP-RL combines a frozen CLIP encoder with an RL-based residual-correction step and reports 81% mean IoU on EndoVis 2018 surgical scenes.","keywords":["CLIP-RL","surgical scene segmentation","semantic segmentation","reinforcement learning","curriculum learning","EndoVis 2018","EndoVis 2017","contrastive language-image pretraining"],"falsifier":"Rerun U-Net and SegFormer on the official EndoVis 2018 test set using the same $224 \\times 224$ resolution, the same 20% validation split, and the same evaluation code as CLIP-RL, and check the source of the S3Net row in Table I. If either baseline reproduces above 0.81 mIoU, or if the cited S3Net reference is not the model in the table, the 'outperforms state-of-the-art' claim fails.","tokens_in":9940,"feed_emoji":"🩺","tokens_out":7996,"duration_ms":75006,"temperature":0.7,"pith_summary":"CLIP-RL is an attempt to make surgical scene segmentation cheap and fully automatic by replacing prompt-dependent vision-language models with a frozen CLIP vision encoder, a lightweight decoder, an RL-based residual-refinement module, and a curriculum loss schedule. The paper claims that on EndoVis 2018 this combination reaches 81% mean IoU and 0.88 Dice, outperforming prompt-based and specialist baselines, and 74.12% mean IoU on EndoVis 2017. If that is right, a text-free segmentation pipeline built on pretrained contrastive features can beat methods that need manual prompt engineering, which would be directly useful for analyzing long minimally invasive surgery videos. The central problem the paper tries to solve is labor-intensive annotation and the impracticality of prompting every frame; its proposed fix is to let a learned residual correction refine the decoder output without any text input.","feed_headline":"CLIP-RL hits 81% mean IoU in surgical scene segmentation","feed_subtitle":"A frozen CLIP encoder plus an RL residual-correction step beats prompt-based baselines on EndoVis 2018, at 0.81 mIoU and 0.88 Dice.","key_machinery":"The load-bearing object is the RL-based residual refinement defined by $O = s_L + \\alpha r$: a lightweight agent samples a scalar residual scaling factor $\\alpha$ after the decoder has produced its softmax map, and an auxiliary module derives a residual term $r$ from the same logits. This makes the whole pipeline act as a self-correcting segmentation network in which even small boundary corrections are learned rather than prompted. The second mechanism is the curriculum weighting $f_{\\text{epoch}} = (1 - \\text{epoch}/\\text{epoch}_{\\text{total}})^2$, which stabilizes training by focusing on segmentation loss early and on RL loss later, and the frozen CLIP vision transformer supplies multi-resolution semantic features after the CLS token is discarded.","core_discovery":"The paper's central claim is that the combination of a frozen CLIP vision transformer, a multi-stage upsampling decoder, and a reinforcement-learning-inspired refinement module produces state-of-the-art surgical scene segmentation. On EndoVis 2018 the reported mean IoU is $0.81$ and Dice is $0.88$, with 8 of 11 classes best among the compared models; on EndoVis 2017 the mean IoU is $74.12\\%$, best in 5 of 7 instrument classes. The RL agent is a small fully connected network that maps a globally pooled encoder feature to a distribution over discrete residual scaling factors $\\{-0.1, 0.0, +0.1\\}$, and the final prediction is $O = s_L + \\alpha r$, where $s_L$ is the softmax output of the decoder and $r$ is a residual correction computed from those logits. A curriculum schedule $f_{\\text{epoch}} = (1 - \\text{epoch}/\\text{total})^2$ weights the segmentation loss heavily at first and gradually shifts weight to the policy-gradient RL loss, which the ablation study attributes to the jump from 72.4% to 81.0% mIoU.","pith_inferences":["The language side of CLIP is never used at inference; only the frozen image encoder is present. This suggests the same architecture could work with any contrastively trained vision backbone, and a fair test would swap in a pure-vision encoder to see how much of the gain is actually from language alignment.","The RL module is effectively a learned global residual calibration of the segmentation logits rather than a full sequential decision process; treating it as such makes it directly comparable to lightweight post-processing layers and opens a cheaper way to test the contribution.","A natural testable extension is to enlarge the action space from three discrete scalars to spatially varying or multi-scale residual maps, and to add temporal consistency across video frames; this could specifically target the thread and covered-kidney errors the paper reports.","Because validation was a random 20% split of the training set and checkpoints were selected by validation mIoU, the reported numbers should be re-checked against the official EndoVis test protocols before relying on the clinical claim."],"forward_implications":["If the reported numbers reproduce, CLIP-RL would be the first prompt-free CLIP-based surgical segmentation pipeline to beat prompt-dependent SAM variants on EndoVis 2018.","The ablation table (72.4 to 76.8 to 81.0 mIoU) indicates that curriculum learning and RL refinement each contribute independently, so both components are worth keeping when the method is transferred.","Because the model works without text prompts during inference, it could be applied to long surgical videos without per-frame interaction.","The per-class results suggest that thin, low-contrast structures (thread and covered kidney on EndoVis 2018) are the residual failure mode, pointing to where the next version of the refinement module should concentrate."],"supporting_citations":[{"why":"Supplies the frozen CLIP vision encoder that is the backbone of the segmentation network.","marker":"[33]"},{"why":"Provides the EndoVis 2017 dataset and challenge protocol used for instrument segmentation.","marker":"[44]"},{"why":"Provides the EndoVis 2018 dataset and challenge protocol used for holistic scene segmentation.","marker":"[45]"},{"why":"Is the SurgicalSAM baseline that the paper compares against and the source of the SAM-based model rows in Table I.","marker":"[19]"},{"why":"Is cited as the S3Net baseline in Table I, though the reference text describes an image relighting method rather than surgical segmentation.","marker":"[13]"},{"why":"Is the U-Net baseline that CLIP-RL is compared against on both datasets.","marker":"[15]"},{"why":"Is the SegFormer baseline that CLIP-RL is compared against on EndoVis 2018.","marker":"[32]"},{"why":"Is the TransUNet baseline that CLIP-RL is compared against on EndoVis 2017 and 2018.","marker":"[30]"},{"why":"Supplies the reinforcement learning formulation and policy gradient method used by the refinement module.","marker":"[38]"},{"why":"Supplies the curriculum learning strategy that schedules the transition from segmentation loss to RL loss.","marker":"[39]"}],"fun_headline_variants":["RL refines CLIP's surgical vision to 81% mIoU","Frozen CLIP + RL correction lifts surgical segmentation to 81%","CLIP-RL merges contrastive and RL learning to reach 81% mIoU","Surgical scene parsing at 81% mIoU: CLIP-RL pairs CLIP with RL","81% mIoU: CLIP-RL's RL residual correction sharpens surgery"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the benchmark comparison is valid: the 'S3Net' row in Table I must be the surgical segmentation S3Net the text describes, not the image-relighting S3Net that reference [13] points to, and the starred U-Net and SegFormer results must come from the same training protocol, image resolution, validation split, and evaluation code used for CLIP-RL.","fun_headline_variants_meta":{"raw":{"variants":["RL refines CLIP's surgical vision to 81% mIoU","Frozen CLIP + RL correction lifts surgical segmentation to 81%","CLIP-RL merges contrastive and RL learning to reach 81% mIoU","Surgical scene parsing at 81% mIoU: CLIP-RL pairs CLIP with RL","81% mIoU: CLIP-RL's RL residual correction sharpens surgery"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000529,"raw_usage":{"total_tokens":2597,"prompt_tokens":1042,"completion_tokens":1555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":1443}},"tokens_in":658,"tokens_out":1555,"duration_ms":14932,"temperature":1.0,"reasoning_tokens":1443,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:50:22.590897+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun U-Net and SegFormer on the official EndoVis 2018 test set using the same $224 \\times 224$ resolution, the same 20% validation split, and the same evaluation code as CLIP-RL, and check the source of the S3Net row in Table I. If either baseline reproduces above 0.81 mIoU, or if the cited S3Net reference is not the model in the table, the 'outperforms state-of-the-art' claim fails.","supporting_citations":[{"cited_title":"Learning Transferable Visual Models From Natural Language Supervision,","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen CLIP vision encoder that is the backbone of the segmentation network."},{"cited_title":"S3Net: A Single Stream Structure for Depth Guided Image Relighting","cited_arxiv_id":"2105.00681","evidence_quote":"Is cited as the S3Net baseline in Table I, though the reference text describes an image relighting method rather than surgical segmentation."},{"cited_title":"SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers,","cited_arxiv_id":null,"evidence_quote":"Is the SegFormer baseline that CLIP-RL is compared against on EndoVis 2018."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the reinforcement learning formulation and policy gradient method used by the refinement module."},{"cited_title":"Bengio, J","cited_arxiv_id":null,"evidence_quote":"Supplies the curriculum learning strategy that schedules the transition from segmentation loss to RL loss."}],"review_version":1}