{"id":"5d30bc12-4e69-4702-9a0f-ea9386c1f256","arxiv_id":"2501.10957","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MARIO combines pixel, box, polygon, scribble, and point supervision in a single transformer model, reaching 85.8% Dice on polyp segmentation benchmarks.","lead":"MARIO is a polyp segmentation model that trains on a mix of five annotation types: pixel, box, polygon, scribble, and point. It reportedly beats fully supervised models on five benchmark datasets, but the comparison is biased because MARIO uses many more training images.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed benefit of mixed supervision is confounded by training-data volume: no baseline is trained with the same weak labels, and the only ablation varies losses, not data.","rationale":"The central contribution of the paper is the mixed-supervision framework, and the headline evidence is the benchmark comparison in Table 1. A benchmark comparison only supports the framework if the training corpus is controlled across methods. Here the comparison is confounded: MARIO sees roughly 80,000 additional weakly annotated samples, while the fully supervised baselines see only 1,451 pixel-level images. The observed improvement is therefore equally consistent with a pure data-scaling effect. The ablation in Table 2 does not resolve this because it varies loss terms while keeping the full mixed corpus fixed; it also omits several of the proposed supervision-specific losses. This is a genuine experimental-design gap, not a matter of disagreement with community consensus. The reader's weakest-assumption analysis identifies the same load-bearing concern, and the conditional verdict is appropriate. If the matched-data comparisons requested above were supplied, the causal claim could be meaningfully evaluated. No adjustment to the reader's verdict is needed beyond what was already stated.","tokens_in":5021,"tokens_out":3021,"duration_ms":36180,"concrete_test":"Reproduce Table 1 under a matched-data design: (A) train MARIO using only the 1,451 pixel-level images with the same training schedule; (B) train MARIO with the full mixed corpus as submitted; (C) train the strongest baseline, e.g., CAFE-Net, on the same 1,451 images plus the weak-label corpus converted to dense pseudo-masks. If (A) already reaches or exceeds CAFE-Net's 84.9% Dice, or if (B) and (C) are statistically indistinguishable with repeated runs, the mixed-supervision claim is not supported. If (B) exceeds both (A) and (C) by more than 1% Dice under matched compute, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim (Section 3.2, Table 1) is that MARIO's mixed-supervision design outperforms fully supervised polyp segmenters. For that claim to hold, the comparison must hold training signal fixed except for the annotation-mixing mechanism. This is not satisfied: Section 3.1 reports 1,451 pixel-level images for the fully supervised training set, while MARIO additionally trains on 33,884 box-annotated LDPolypVideo frames, 49,136 SUN-SEG frames split into polygon and scribble annotations, and 1,412 PolypGen point-annotated images. Any of these extra 80,000+ samples could explain the higher weighted-average Dice and IoU. The Table 2 ablation only adds loss terms while the full mixed data volume is present; it never runs MARIO on the pixel-only training set, and it does not include the box, polygon, or point losses described in Section 2.2. Consequently, the central causal claim that mixed supervision, rather than simply more data, drives the 85.8% Dice result is untested. The absence of error bars is secondary; the missing matched-data comparison is the decisive gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MARIO, a polyp segmentation framework that trains a PVTv2-based model with five annotation types: pixel, polygon, box, scribble, and point. Each annotation type is paired with a tailored loss; the total loss is a sum of the individual losses. The authors report that MARIO achieves a weighted-average Dice of 85.8% and IoU of 78.3% across five polyp benchmarks, surpassing fully supervised baselines such as CAFE-Net. A small ablation in Table 2 shows that adding the uncertainty and consistency losses improves performance. The paper concludes that mixed supervision reduces labeling cost and improves segmentation accuracy.","tokens_in":5180,"tokens_out":3115,"duration_ms":35045,"significance":"If the central claim is supported, MARIO would be a useful practical contribution: it would show that a single model can exploit heterogeneous, cheap annotations from multiple public datasets and outperform models trained only on pixel-level labels. The paper also proposes a simple uncertainty loss for scribble supervision and a rotation-based consistency loss for point supervision. However, the paper's significance is contingent on demonstrating that the improvement comes from the mixed-supervision mechanism rather than from the much larger training set. The evaluation is on standard public benchmarks and the implementation details are mostly reproducible, but the missing controlled comparison is a serious weakness that must be addressed before the claim can be accepted.","major_comments":[{"comment":"The central claim that MARIO's mixed-supervision design outperforms fully supervised models is confounded by training-data volume. According to §3.1, the fully supervised baselines are trained on 1,451 pixel-level images, while MARIO additionally trains on 33,884 box-annotated LDPolypVideo frames, 49,136 SUN-SEG frames (polygon and scribble), and 1,412 PolypGen point-annotated images, totaling over 80,000 extra weakly labeled samples. No baseline is trained with the same weak data, and no experiment isolates the effect of the annotation-mixing mechanism from the effect of simply having more training data. Without a controlled comparison—for example, (a) MARIO trained only on the 1,451 pixel-level images, (b) a fully supervised baseline trained with the same additional weak data through pseudo-labels or auxiliary losses, or (c) subsampling the weak data to match the baseline's data volume—the reported 1.0% Dice improvement over CAFE-Net cannot be attributed to mixed supervision. This is a load-bearing gap.","section":"§3.2, Table 1"},{"comment":"The paper states that PolypGen provides 1,412 images with point annotations (5 foreground and 5 background points), but the proposed point-supervision loss is a rotation-consistency MSE loss that does not use the point labels at all. The described loss constrains predictions on an image to be consistent with predictions on its 90-degree-rotated version, which is a generic consistency regularizer that can be applied to any image, labeled or not. As written, the point-level labels are never used to supervise the model; thus the claim that MARIO learns from point annotations is unsupported. Either the point labels are used in a way not described in the paper (e.g., a cross-entropy term on the annotated pixels), or the point-annotation data contributes nothing beyond unlabeled images. The authors must clarify or correct this.","section":"§2.2, Point Supervision"},{"comment":"The ablation study only varies loss components while keeping the full mixed dataset fixed. It never runs MARIO on the pixel-only training set, nor does it remove individual annotation types (box, polygon, scribble, point) one at a time. Consequently, the ablation cannot distinguish the contribution of the mixed-supervision mechanism from the contribution of the additional data volume. A proper ablation should include at least a row with only dense supervision on the 1,451 pixel-level images, and ideally rows with each weak annotation type added while holding the total number of training images constant or controlling for data volume.","section":"§3.3, Table 2"}],"minor_comments":[{"comment":"The second-best model is referred to as 'CAFA-Net' in the text but 'CAFE-Net' in the table caption and in reference [9]; the naming should be made consistent.","section":"Table 1 and §3.2"},{"comment":"The statement that SUN-SEG's 49,136 samples were 'evenly split for polygon and scribble annotations' is ambiguous; please specify the exact numbers for each annotation type.","section":"§3.1"},{"comment":"The ablation table does not state which test set or aggregation the Dice and IoU values correspond to; presumably they are the weighted-average metrics over the five test sets, but this should be stated explicitly.","section":"§3.3, Table 2"},{"comment":"The total loss is a simple sum of the five losses with equal weights; the paper does not discuss whether the loss scales are comparable or whether any weight balancing was considered.","section":"§2.2, Eq. (2)"},{"comment":"The datasets SUN-SEG, LDPolypVideo, and PolypGen are used in training but no references are provided for them; citations should be added.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The core issue is experimental rather than theoretical: the paper's central claim is currently unsupported because the training-data volume is not controlled. In my view this is fixable with additional experiments, so I recommend major revision rather than rejection. The authors should also double-check the point-supervision description, because as written it appears that point labels are not used at all, which is a substantive flaw. The paper is otherwise reasonably written but would benefit from a more precise statement of what 'outperforming fully supervised methods' is supposed to demonstrate given the intentionally larger and cheaper training set."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a genuinely plausible idea: instead of forcing every dataset into pixel-level masks, train one polyp segmenter on whatever annotation type is available (box, polygon, scribble, point, and pixel). That is new for this specific task, and the engineering is sensible. Each loss is borrowed from prior work—mask-to-box, uncertainty for scribbles, rotation consistency—but combining them in one framework with five data sources is a reasonable contribution. The benchmark numbers are also in the right ballpark; the model does beat fully supervised methods on average, and the gain on ETIS is notable. The problem is that the central claim is not actually tested. Section 3.1 says the fully supervised baselines train on 1,451 pixel-level images. MARIO additionally trains on 33,884 box-annotated frames, 49,136 SUN-SEG frames split into polygon and scribble, and 1,412 point-annotated images. That is roughly 85,000 extra weakly labeled samples. With that much additional data, a 1.0% Dice improvement could come from sheer volume, not from the mixed-supervision mechanism. The paper never compares MARIO against a fully supervised model trained on the same weak data, nor does it run MARIO with only the 1,451 pixel images. The ablation in Table 2 only toggles loss terms while keeping all the weak data present, so it cannot separate data volume from the method. That is a load-bearing gap, not a minor omission. The small effect size also matters: 1.0% Dice on a weighted average, with no error bars, is a thin margin to hang a causal story on. There are also smaller issues: the paper names the second-best method \"CAFA-Net\" in the text but \"CAFE-Net\" in Table 1, and the notation for the total loss is a simple sum without weighting, which is fine but under-discussed. What is good: the writing is clear, the experimental setup is mostly described with enough detail to reproduce (datasets, splits, optimizer, iterations), and the idea of expanding usable annotation types is practically motivated. If the authors add a matched-data baseline—say, training MARIO on the pixel-only set and training a strong fully supervised baseline on the same weak data—the result would be much more convincing. The missing comparison is fixable, which makes this a solid borderline paper rather than a dead one. For a reading group, it is a useful case study in confounding in medical imaging benchmarks. I would not cite the headline claim until the data-volume issue is resolved. But I would send it to peer review: the core idea is relevant, the evaluation is extensive for the claim it makes, and the key weakness is something reviewers can reasonably demand be fixed. A serious referee should ask for the controlled experiment and probably some error bars, but the paper deserves the round.","headline":"Mixed-supervision idea is real and the benchmark table is decent, but the headline comparison is confounded by training-data volume and the paper needs a matched-data baseline before the central claim is credible.","tokens_in":802,"tokens_out":1146,"would_cite":false,"duration_ms":25953,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MARIO mixes pixel, box, polygon, scribble, and point annotations in one transformer, reporting 85.8% weighted Dice on five polyp benchmarks, beating the best fully supervised rival by 1.0%.","keywords":["polyp segmentation","mixed supervision","weakly supervised learning","colonoscopy","transformer","annotation efficiency","medical image segmentation"],"falsifier":"Run a fully supervised baseline (for example, CAFE-Net or Polyp-Pvt) with the same backbone and the same training protocol as MARIO, but give it both the 1,451 pixel-level images and the weakly labeled images MARIO uses. If its weighted-average Dice on the five test sets reaches or exceeds 85.8%, the claim that mixed-supervision design is the cause is falsified.","tokens_in":4775,"feed_emoji":"🩺","tokens_out":5747,"duration_ms":55976,"temperature":0.7,"pith_summary":"Polyp segmentation normally requires pixel-level masks, which are expensive to produce and limit dataset size. MARIO is a single transformer model that trains on five annotation types at once — pixel, box, polygon, scribble, and point — each paired with a loss designed to handle its noise. The paper reports that this mixed-supervision approach reaches the highest weighted-average performance on five benchmarks, with 85.8% Dice and 78.3% IoU, beating the best fully supervised model by 1.0% Dice. The intended payoff is that clinics can label data with cheaper annotation styles and still obtain competitive segmentation.","feed_headline":"Mixed annotations beat full supervision in polyp segmentation","feed_subtitle":"MARIO uses five label types to outscore CAFE-Net by 1.0% Dice on five benchmarks.","key_machinery":"The engine of MARIO is the combined loss $L_{total}=L_{pixel}+L_{polygon}+L_{box}+L_{scribble}+L_{points}$, with a distinct mechanism per annotation type. Pixel- and polygon-level annotations use the standard BCE plus Dice loss. Box annotations are supervised through a mask-to-box ($M2B$) transform, which projects a predicted mask into row and column vectors and then back-projects a box-shaped mask, letting the model learn location without being misled by box shape. Scribble annotations use an uncertainty loss $\\min(-\\log P_s, -\\log(1-P_s))$ that keeps the model from becoming overconfident in unlabeled pixels. Point annotations use a consistency loss computed as the MSE between predictions on an image and on its 90-degree rotated version. The backbone is a Pyramid Vision Transformer (PVTv2-B2).","core_discovery":"MARIO claims that a transformer-based segmentation model can learn effectively from a mix of five annotation types, and that this mixed supervision outperforms full pixel-level supervision. On the five test datasets (ColonDB, Kvasir, ClinicDB, EndoScene, ETIS), its weighted-average Dice is 85.8% and IoU is 78.3%, exceeding the best fully supervised competitor CAFE-Net by 1.0% and 0.6%, respectively. The paper attributes this to the complementary strengths of the annotation types and to specially designed losses: dense supervision for pixel and polygon labels, a mask-to-box transformation for box labels, an uncertainty loss for scribble labels, and a rotation-consistency loss for point labels. The conclusion is that mixed-annotation training can simultaneously improve accuracy, expand usable data, and reduce labeling cost.","pith_inferences":["The paper does not isolate the mixed-supervision mechanism from the larger training set: MARIO adds more than 80,000 weakly labeled images on top of the 1,451 pixel-level images the baselines use, so the reported gain could stem from data volume alone.","A direct test would train CAFE-Net or another fully supervised baseline with the same pool of weakly labeled images; if the gap disappears, the claim that mixed supervision causes the improvement would be falsified.","The rotation-consistency loss for point labels suggests a general equivariance principle; using several random rotations and flips might further stabilize point-supervised training.","The five-benchmark weighted average treats the five test sets by image count; a per-dataset analysis would show whether the headline gain is concentrated in one dataset (e.g., ETIS) rather than uniform."],"forward_implications":["If the results hold, large weakly labeled polyp datasets that are currently discarded become usable training material.","Clinicians could annotate with boxes, scribbles, or points instead of pixel masks and still expect segmentation accuracy on par with fully supervised models.","The per-annotation loss design offers a recipe for other medical segmentation problems where annotation types are heterogeneous.","Low-cost annotation could expand polyp screening programs in settings where expert pixel annotation is unavailable."],"supporting_citations":[{"why":"CAFE-Net: the strongest fully supervised baseline MARIO claims to beat; central comparison in Table 1.","marker":"[9]"},{"why":"Polyp-Pvt: transformer-based polyp segmentation baseline that MARIO's transformer design extends and competes with.","marker":"[4]"},{"why":"PraNet: source of the dense BCE-plus-Dice supervision loss and a fully supervised baseline.","marker":"[2]"},{"why":"SANet: another fully supervised baseline and contributor of deep supervision ideas.","marker":"[3]"},{"why":"PVTv2-B2: the backbone architecture MARIO uses for feature extraction.","marker":"[10]"},{"why":"WeakPolyp: prior box-supervision method that motivates MARIO's mixed-supervision approach.","marker":"[5]"}],"fun_headline_variants":["Mixed annotations beat full supervision for polyp segmentation","Five annotation types outperform one in polyp segmentation","MARIO beats fully supervised polyp segmentation with mixed labels","Cheap mixed annotations better than full labels in polyp segmentation","Polyp segmentation wins with mixed weak supervision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central premise is that mixed supervision itself drives the improvement, but it is never isolated from the much larger volume of weakly labeled training data used only by MARIO; without a same-data baseline, the gain could be entirely due to more training images.","fun_headline_variants_meta":{"raw":{"variants":["Mixed annotations beat full supervision for polyp segmentation","Five annotation types outperform one in polyp segmentation","MARIO beats fully supervised polyp segmentation with mixed labels","Cheap mixed annotations better than full labels in polyp segmentation","Polyp segmentation wins with mixed weak supervision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000438,"raw_usage":{"total_tokens":2191,"prompt_tokens":877,"completion_tokens":1314,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":1243}},"tokens_in":493,"tokens_out":1314,"duration_ms":10702,"temperature":1.0,"reasoning_tokens":1243,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:46:33.730516+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a fully supervised baseline (for example, CAFE-Net or Polyp-Pvt) with the same backbone and the same training protocol as MARIO, but give it both the 1,451 pixel-level images and the weakly labeled images MARIO uses. If its weighted-average Dice on the five test sets reaches or exceeds 85.8%, the claim that mixed-supervision design is the cause is falsified.","supporting_citations":[{"cited_title":"Polyp-pvt: Polyp segmen- tation with pyramid vision transformers,","cited_arxiv_id":null,"evidence_quote":"CAFE-Net: the strongest fully supervised baseline MARIO claims to beat; central comparison in Table 1."},{"cited_title":"MARIO unifies five annotation types: pixel-level, polygon-level, box-level, scribble-level, and point-level","cited_arxiv_id":null,"evidence_quote":"Polyp-Pvt: transformer-based polyp segmentation baseline that MARIO's transformer design extends and competes with."},{"cited_title":"MARIO: A Mixed Annotation Framework For Polyp Segmentation","cited_arxiv_id":"2501.10957","evidence_quote":"PraNet: source of the dense BCE-plus-Dice supervision loss and a fully supervised baseline."},{"cited_title":"Dataset and Implementation Detail This study leverages eight datasets: Kvasir, CVC-ClinicDB, CVC-ColonDB, EndoScene, ETIS, SUN-SEG, LDPolypVideo, and PolypGen","cited_arxiv_id":null,"evidence_quote":"SANet: another fully supervised baseline and contributor of deep supervision ideas."},{"cited_title":"Weakpolyp: You only look bounding box for polyp segmentation,","cited_arxiv_id":null,"evidence_quote":"PVTv2-B2: the backbone architecture MARIO uses for feature extraction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"WeakPolyp: prior box-supervision method that motivates MARIO's mixed-supervision approach."}],"review_version":1}