{"id":"ae0cc3e8-e4c7-49b7-8103-6351691f347e","arxiv_id":"2412.18738","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HELPNet uses hierarchical jigsaw consistency, entropy-weighted pseudo-labels, and structural priors to bring scribble-supervised medical segmentation close to fully supervised performance.","lead":"This paper introduces HELPNet, a weakly supervised segmentation framework that uses different-density jigsaw puzzles, entropy-weighted pseudo-labels, and structural priors to train from scribble annotations. On three public MRI datasets it reports near-fully-supervised segmentation accuracy, which could reduce the cost of creating training labels for medical imaging.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 89.6%-vs-90.2% comparison with nnUNet lacks proof of a common protocol: hyperparameters are tuned on the same dataset, and nnUNet is reported without the error bars given for all other methods.","rationale":"The reader's weakest_assumption focuses on the reliability of entropy-weighted pseudo-labels. That concern is legitimate but secondary: the ablation (Table 4) shows EGPL alone contributes modestly (77.5→82.1) and HPC+EGPL (85.5) is not better than HPC alone (85.6); SPR and HPC drive most of the gain. Thus the empirical headline does not hinge on pseudo-labels being perfectly calibrated. The headline does hinge on the two reported numbers being measured under the same protocol, which is not demonstrated. The missing standard deviation for nnUNet and the ambiguous hyperparameter-selection protocol are concrete, checkable gaps that directly affect whether scribble supervision can be said to 'nearly match' full supervision. The reader's overall CONDITIONAL verdict remains appropriate; the protocol concern reinforces it rather than moving it.","tokens_in":17754,"tokens_out":9536,"duration_ms":94492,"concrete_test":"Inspect the released repository (https://github.com/IPMI-NWU/HELPNet) to verify: (1) the hyperparameter sweep in §4.6.2 was scored on the 5-case validation split, not the 15-case test split; and (2) the nnUNet row in Table 1 was produced by training nnUNet on the same 25/5/15 split with the same preprocessing, rather than quoted from literature. Then rerun the chosen configuration with ≥3 seeds on the test split and retrain nnUNet the same way. If HELPNet's mean stays ≥88.6% and the nnUNet mean is reproduced within ~1%, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that scribble supervision nearly matches full supervision—depends on the two numbers 89.6% (HELPNet) and 90.2% (nnUNet) being measured under the same protocol on the same MSCMRseg split. This is not established. In Table 1, every method except nnUNet is reported with a standard deviation; nnUNet is a bare point estimate, which suggests it is quoted from the literature rather than re-evaluated on the 25/5/15 split used for all other rows. If the nnUNet number comes from a different test set or a different preprocessing pipeline, the 0.6 percentage-point gap is not an apples-to-apples comparison. Second, Sections 4.2 and 4.6.2 state that the loss weights λ1, λ2, λ3 and temperature τ were selected 'on the MSCMRseg dataset' through ablation experiments. The paper never states that this selection used only the 5-case validation split; if the 15-case test split informed the choice, the reported 89.6% is optimistically selected. Because the gap to nnUNet (0.6%) is much smaller than the reported improvement over the previous weakly supervised method (4.4%), a small optimistic bias could change the conclusion. The available code would make this checkable, but the manuscript itself does not specify the protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"HELPNet is a scribble-supervised medical image segmentation framework combining three modules: a hierarchical perturbation consistency (HPC) module that applies intensity augmentation and two densities of jigsaw puzzles (2x2 and 4x4) to create global, local, and focal views and enforces cosine-similarity consistency between their predictions; an entropy-guided pseudo-label ensemble (EGPL) that weights the three prediction maps by softmaxed negative entropy to generate pseudo-labels and supervises all branches with Dice loss; and a structural prior refinement (SPR) module that filters pseudo-labels via connected-component analysis and refines boundaries using a pretrained edge detector (PiDiNet). The model is trained with partial cross-entropy on scribbles plus the three auxiliary losses. The authors report Dice scores on MSCMRseg, ACDC, and CHAOS T1/T2, claiming state-of-the-art performance among scribble-supervised methods and performance comparable to fully supervised nnUNet.","tokens_in":18013,"tokens_out":4692,"duration_ms":41145,"significance":"If the reported results are protocol-sound, the paper provides valuable evidence that scribble supervision can approach full supervision on cardiac MRI: on MSCMRseg the method reaches 89.6% average Dice versus 90.2% for nnUNet, and on ACDC it reaches 90.6% versus 91.7%. The framework is modular, the three components are clearly motivated, and the authors have made the code publicly available, which strengthens reproducibility. However, the significance of the central claim depends on the evaluation being apples-to-apples with the fully supervised baseline and on the hyperparameter-selection protocol not leaking test information; those points are not yet established in the manuscript.","major_comments":[{"comment":"The central comparison with nnUNet is not apples-to-apples as reported. In Table 1, nnUNet is a bare point estimate (0.902 average) with no standard deviation, while all other rows report mean ± std, and the text does not state that nnUNet was re-run on the same 25/5/15 split and preprocessing pipeline. If the nnUNet number is quoted from prior literature, the 0.6-point gap is not evidence that the weakly supervised method is comparable to full supervision. Please either re-run nnUNet under the identical protocol or explicitly soften the claim to 'comparable to published nnUNet results.'","section":"§4.3, Table 1"},{"comment":"The hyperparameters λ1, λ2, λ3 and τ were selected by ablations described as performed 'on the MSCMRseg dataset,' without stating that only the 5-case validation split was used. Since the headline 89.6% is measured on the 15-case test split, selecting hyperparameters on the test set would introduce optimistic bias, and this bias is large relative to the 0.6% gap to nnUNet. Please clarify explicitly that the test split was never used for model selection, or re-run the ablation on the validation split and report the corresponding test results for the chosen configuration.","section":"§4.2 and §4.6.2, Tables 5–6"},{"comment":"The main tables are not accompanied by significance tests. The phrase 'significantly outperformed' in §4.3 and §4.5 is not supported by paired tests or confidence intervals; the only Wilcoxon tests appear in the ablation study (Table 4). Please add paired significance tests or confidence intervals against the strongest baselines on each dataset, and report standard deviations for all quoted baselines, including those currently given as bare numbers.","section":"§4.3, Tables 1–3"},{"comment":"The core motivation of EGPL is that entropy-weighted ensembling yields better pseudo-labels than equal-weight averaging, but this specific design choice is never tested. The ablation shows EGPL on/off, yet does not compare entropy weighting against uniform weighting or random weighting of the same three prediction maps. Without that comparison, the 'entropy-guided' contribution is not empirically validated, and the improvement attributed to EGPL could come from ensembling itself. Please add such an ablation.","section":"§3.2 and Table 4"},{"comment":"The structural prior loss is written as LSPR = Lcos(yPL, yCCA) + LEdge(ŷPL, ye), but yPL is defined in Eq. (5) as an argmax label map, while Lcos in Eq. (4) is a cosine similarity between probability vectors. Cosine similarity on hard label maps is not well defined. Please clarify whether yPL in this loss is the soft probability map before argmax, or provide the appropriate loss formulation for label maps.","section":"§3.3, Eq. (10)"}],"minor_comments":[{"comment":"The text in the T1 paragraph says 'Compared to PacingPseudo, HELPNet shows superior performance in terms of the average Dice score,' but on T2 PacingPseudo achieves 0.737 average versus 0.735 for HELPNet. Please restrict the claim to T1 or report a combined average across modalities.","section":"§4.5, Table 3"},{"comment":"The sentence 'Perturbed images can be enhanced as follows' is misleading because xS2 and xS4 are jigsaw-perturbed, not intensity-enhanced. Please rephrase to distinguish the intensity augmentation of xS1 from the jigsaw perturbations of xS2 and xS4.","section":"§3.1, Eq. (1)"},{"comment":"The statement 'Except for some unreported experimental results, other methods are presented as mean ± standard deviation' is vague. Please mark in the tables which numbers are re-run in this paper and which are quoted from prior publications, so that the reader can judge protocol comparability.","section":"Tables 1–3"},{"comment":"There are minor wording issues: the abstract contains 'enhance the precision and reliability and pseudo-labels' (likely 'of pseudo-labels'), and §4.4 says 'Our method achieved the best results in RV and Myo segmentation, ranks second in RV segmentation,' which is self-contradictory and should be corrected.","section":"Abstract and §4.4"}],"recommendation":"major_revision","confidential_remarks":"The empirical claim is plausible and the code is available, so the protocol issues can likely be resolved with additional experiments. The most serious risk is test-set leakage in hyperparameter selection, since the margin over nnUNet is small. I would recommend asking the authors to clarify the validation protocol, re-run or explicitly qualify the nnUNet comparison, and add significance tests before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"HELPNet is a solid, well-engineered scribble-supervised segmentation method, and the empirical work is strong enough to justify a serious referee. The genuinely new idea is combining jigsaw perturbations at three densities (global, local, focal) with consistency regularization, and then using entropy to weight the resulting prediction maps when ensembling pseudo-labels. Each component is familiar from the SSL/PL literature, but the combination for scribble supervision is original and the ablations show each piece earns its keep.\n\nThe headline claim — 89.6% vs 90.2% nnUNet on MSCMRseg — is plausible but not air-tight. The nnUNet row in Table 1 has no standard deviation, which suggests it is quoted from the literature rather than re-run on the same 25/5/15 split. With a 0.6-point gap, a small protocol difference can change the conclusion. Relatedly, the hyperparameters λ and τ were selected through ablations 'on the MSCMRseg dataset' (Tables 5–6), and the paper does not explicitly say only the 5-case validation split was used. The ablation curves are flat within about 1 point, which mitigates the risk of optimistic selection, but the protocol should be stated.\n\nTwo smaller issues. On CHAOS T2, Table 3 shows PacingPseudo (73.7) slightly above HELPNet (73.5), while a sentence in Section 4.5 says HELPNet shows superior performance to PacingPseudo; that sentence appears to be about T1 only, but it reads as a general claim and should be reworded. Also, the comparisons omit recent scribble methods that are cited in the references (S2ME, DMSPS, PCLMix), so 'state-of-the-art' is a bit strong.\n\nThe data and citations are otherwise in order, the code is available, and the internal logic holds together. This is a within-subfield improvement rather than a fundamental advance, but it is a useful one for anyone working on scribble-supervised medical image segmentation. I would send it to peer review with a request to pin down the nnUNet protocol, confirm the hyperparameter selection used only the validation split, and add the missing baselines.","headline":"HELPNet is a solid scribble-supervision method with strong results, but the near-nnUNet claim rests on protocol details that need to be pinned down.","tokens_in":18551,"tokens_out":4457,"would_cite":true,"duration_ms":35882,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that scribble-only supervision, powered by hierarchical jigsaw consistency and entropy-guided pseudo-label ensembling, nearly matches fully supervised cardiac MRI segmentation.","keywords":["scribble supervision","weakly supervised segmentation","medical image segmentation","cardiac MRI","jigsaw perturbation","entropy-guided pseudo-labels","consistency regularization","pseudo-label refinement"],"falsifier":"Train HELPNet and two ablations on the same datasets and scribbles: one with equal weights instead of entropy weights in Eq. 6, one with random weights. If the entropy-weighted version does not beat both on average Dice across MSCMRseg, ACDC, and CHAOS, the entropy mechanism is not what carries the gain; a further ablation using oracle weights derived from the ground-truth labels would bound the best possible improvement from confidence weighting.","tokens_in":17543,"feed_emoji":"🫀","tokens_out":7141,"duration_ms":53239,"temperature":0.7,"pith_summary":"HELPNet sets out to prove that scribble annotations—thin strokes marking a few pixels per organ—can train a medical image segmenter almost as well as full pixel-level masks. Its three-module recipe perturbs the input with jigsaw puzzles at two densities (none, 2×2, and 4×4), enforces consistency among the three restored predictions, then uses per-pixel entropy to weight those predictions into a pseudo-label that supervises the whole image. A structural-prior step cleans the label with connected-component analysis and a pretrained edge detector. On MSCMRseg cardiac MRI the method reports 89.6% average Dice, 4.4 points above the previous best scribble method and within 0.6 points of fully supervised nnUNet, with similar margins on ACDC and CHAOS. If the claim holds, scribble supervision becomes a practical substitute for dense annotation in cardiac MRI segmentation.","feed_headline":"Scribble-only cardiac MRI training nears full-supervision accuracy","feed_subtitle":"HELPNet hits 89.6% Dice with scribbles against 90.2% for fully supervised nnUNet.","key_machinery":"The central mechanism is the entropy-guided pseudo-label ensemble built on a three-view hierarchical perturbation scheme. The three views are the original image, a 2×2 jigsaw, and a 4×4 jigsaw; after the jigsaw views are unshuffled, a cosine similarity loss (Eq. 3) forces all three predictions to agree. Per-pixel entropy (Eq. 7) acts as a confidence meter for each view, and a temperature-scaled softmax (Eq. 8) converts negative entropies into weights that bias the ensemble toward the most confident view at each location (Eq. 5). This pseudo-label supervises all three views through a Dice loss (Eq. 9), while the SPR module applies connected-component filtering and PiDiNet edge alignment (Eq. 10–11) as a final denoising step.","core_discovery":"On the paper's own terms, the central discovery is that hierarchical jigsaw perturbations make a single weight-shared segmenter attend to a target organ at three complementary granularities, and that entropy-weighted ensembling of the three views yields pseudo-labels strong enough to supervise most of the image. The entropy weights upweight whichever view is most confident at each pixel (Eq. 6), the cosine consistency loss ties the views together (Eq. 3), and the structural-prior refinement cleans the label with connected-component analysis and PiDiNet edges (Eq. 10–11). The experimental core is the reported 89.6% mean Dice on MSCMRseg with scribble labels, 4.4 points above the best prior weakly supervised method, ShapePU, and 0.6 points below fully supervised nnUNet; the same pattern appears on ACDC and CHAOS. The authors conclude that scribble annotations, augmented with multi-scale consistency and entropy-guided pseudo-labeling, offer a viable alternative to full annotation.","pith_inferences":["If entropy is not a calibrated confidence signal, the method's edge could shrink or reverse on low-contrast or out-of-distribution images; a reliability-diagram study on a held-out imaging site would test this directly.","The 0.6-point gap to fully supervised nnUNet on MSCMRseg could widen on more anatomically variable or pathology-rich cohorts, since parity is demonstrated only on three public benchmarks.","A natural extension is to learn the confidence weights with a small network instead of using raw entropy, and to search the jigsaw densities per organ; both changes could push scribble performance closer to full supervision."],"forward_implications":["Scribble annotations could replace dense masks for cardiac MRI segmentation in clinical pipelines without a substantial drop in Dice.","Hierarchical jigsaw consistency can serve as a generic regularization strategy for other scribble- or point-supervised segmentation tasks.","Entropy-weighted ensembling of multiple prediction views may improve pseudo-label quality beyond medical imaging, in any weakly supervised setting where several perturbed views exist.","The reported margins on ACDC, MSCMRseg, and CHAOS suggest the method transfers across cardiac and abdominal organs and across MRI sequences."],"supporting_citations":[{"why":"Defines the UNet+ backbone and CycleMix baseline plus scribble annotations on MSCMRseg and ACDC used for training and comparison.","marker":"(Zhang and Zhuang, 2022a)"},{"why":"ShapePU, the previous best scribble-supervised method that HELPNet claims to beat by 4.4% on MSCMRseg.","marker":"(Zhang and Zhuang, 2022b)"},{"why":"nnUNet, the fully supervised method HELPNet claims to match on MSCMRseg and ACDC.","marker":"(Isensee et al., 2021)"},{"why":"DMPLS, the dynamically mixed pseudo-label baseline that EGPL is designed to outperform.","marker":"(Luo et al., 2022)"},{"why":"ScribFormer, a strong scribble baseline (also source of UNet++ numbers) that HELPNet surpasses on ACDC and MSCMRseg.","marker":"(Li et al., 2024)"},{"why":"Introduces the jigsaw-puzzle pretext task that HPC adapts into two-density consistency learning.","marker":"(Noroozi and Favaro, 2016)"},{"why":"Provides PiDiNet, the pretrained edge detector used in SPR for boundary refinement.","marker":"(Su et al., 2021)"},{"why":"Supplies the ACDC dataset with scribble annotations on which the second benchmark is measured.","marker":"(Bernard et al., 2018)"},{"why":"Supplies the MSCMRseg dataset with gold standard and scribbles used for the main benchmark.","marker":"(Zhuang, 2016, 2019)"},{"why":"Supplies the CHAOS dataset with scribble annotations on which the abdominal-organ results are measured.","marker":"(Kavur et al., 2021)"}],"fun_headline_variants":["Scribble annotations now match full-label MRI segmentation accuracy","Entropy-weighted pseudo-labels close the scribble supervision gap","Multi-scale perturbations lift scribble segmentation to full-supervision level","Three-view ensemble turns scribbles into near-full-supervision labels","Scribble-trained model hits 89.6% Dice vs 90.2% fully supervised"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that low-entropy predictions are trustworthy predictions, so the entropy-weighted pseudo-labels are reliable enough to supervise regions far from the scribbles without amplifying errors.","fun_headline_variants_meta":{"raw":{"variants":["Scribble annotations now match full-label MRI segmentation accuracy","Entropy-weighted pseudo-labels close the scribble supervision gap","Multi-scale perturbations lift scribble segmentation to full-supervision level","Three-view ensemble turns scribbles into near-full-supervision labels","Scribble-trained model hits 89.6% Dice vs 90.2% fully supervised"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000877,"raw_usage":{"total_tokens":3825,"prompt_tokens":1012,"completion_tokens":2813,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":2718}},"tokens_in":628,"tokens_out":2813,"duration_ms":17173,"temperature":1.0,"reasoning_tokens":2718,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:31:03.754407+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train HELPNet and two ablations on the same datasets and scribbles: one with equal weights instead of entropy weights in Eq. 6, one with random weights. If the entropy-weighted version does not beat both on average Dice across MSCMRseg, ACDC, and CHAOS, the entropy mechanism is not what carries the gain; a further ablation using oracle weights derived from the ground-truth labels would bound the best possible improvement from confidence weighting.","supporting_citations":[{"cited_title":", author Jaeger, P.F","cited_arxiv_id":null,"evidence_quote":"nnUNet, the fully supervised method HELPNet claims to match on MSCMRseg and ACDC."},{"cited_title":", author Hu, M","cited_arxiv_id":null,"evidence_quote":"DMPLS, the dynamically mixed pseudo-label baseline that EGPL is designed to outperform."},{"cited_title":", author Favaro, P","cited_arxiv_id":null,"evidence_quote":"Introduces the jigsaw-puzzle pretext task that HPC adapts into two-density consistency learning."},{"cited_title":", author Lalande, A","cited_arxiv_id":null,"evidence_quote":"Supplies the ACDC dataset with scribble annotations on which the second benchmark is measured."},{"cited_title":", year 2016","cited_arxiv_id":null,"evidence_quote":"Supplies the MSCMRseg dataset with gold standard and scribbles used for the main benchmark."},{"cited_title":", author Gezer, N.S","cited_arxiv_id":null,"evidence_quote":"Supplies the CHAOS dataset with scribble annotations on which the abdominal-organ results are measured."}],"review_version":1}