{"id":"19a32dd4-58c1-4469-991d-9df7d4c741ef","arxiv_id":"2412.06610","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using nnU-Net V2 with a 15-fold ensemble, the authors report aggregated Dice scores of 0.81 (pre-RT) and 0.70 (mid-RT) for head and neck tumor segmentation on MRI.","lead":"This paper applies the nnU-Net V2 framework with a 15-fold cross-validation ensemble to segment head and neck tumors on T2-weighted MRI. The method achieved aggregated Dice scores of 0.81 on pre-treatment scans and 0.70 on mid-treatment scans in blind testing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own validation comparison shows 5-fold and 15-fold both reach DSC agg 0.74, contradicting the conclusion that 15-fold has a positive effect; the central methodological claim is unsupported.","rationale":"The reader's verdict is CONDITIONAL, and their rationale explicitly identifies the contradiction between the 5-fold vs 15-fold validation comparison (both 0.74) and the conclusion's claim of a positive effect. However, the reader's 'weakest_assumption' field focuses on the unablated three-channel input for Task 2, which is a different concern. My stress-test pass finds the 15-fold overclaim to be the single most load-bearing issue because it attacks the paper's stated methodological novelty directly and relies on an internal inconsistency rather than a gap in external comparison. The three-channel input concern is real but secondary: even if the extra channels are not beneficial, the reported test scores still stand; whereas the 15-fold claim, if false, undermines the title, abstract, and conclusion. I agree with the reader's CONDITIONAL verdict but identify a different primary weak point, hence 'partial' agreement. The concrete test would settle the contradiction by reproducing the comparison; if the 15-fold and 5-fold performances are equal, the paper's central methodological claim fails and the conclusion must be revised.","tokens_in":10255,"tokens_out":6041,"duration_ms":62622,"concrete_test":"Run the published nnU-Net V2 code (Zenodo, version hntsmrg) on Task 2 with 5-fold and 15-fold cross-validation, keeping all other settings identical (1250 epochs, same data splits, same preprocessing). Compute the validation DSC agg for each fold and compare the ensembles. If the 15-fold ensemble does not significantly outperform the 5-fold (e.g., paired test p>0.05 or absolute difference <0.01), the paper's central claim of a positive effect is contradicted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central methodological contribution is the 15-fold cross-validation ensemble, which the title, abstract, and conclusion tout as providing 'increased robustness and variability' and a 'positive effect on the segmentation performance' for both tasks. However, Section 2.5 reports: 'A comparison of training 5-folds and 15-folds resulted in a similar validation DSC agg of 0.74 for both models, both trained for 1250 epochs.' If both configurations yield the same validation DSC agg, then the claimed benefit of 15-fold over 5-fold is not demonstrated by the paper's own evidence. The conclusion's claim of a 'positive effect' is internally contradicted by this result. This is not a matter of disagreement with external consensus; it is an inconsistency within the manuscript. The 15-fold ensemble is the paper's stated novelty, so this contradiction directly weakens the central claim that the proposed 15-fold approach is beneficial. A standard 5-fold nnU-Net V2 may perform equally well, reducing the paper's contribution to a routine challenge submission with acceptable but not uniquely justified design choices.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports the RUG_UMCG team's submission to the HNTSMRG-24 challenge, whose two tasks are segmentation of the primary gross tumor volume (GTVp) and metastatic lymph nodes (GTVn) in T2-weighted MRI volumes acquired before (Task 1) and during (Task 2) radiotherapy. The proposed method is the nnU-Net V2 framework in its 3d_fullres configuration, modified from the default 5-fold to a 15-fold cross-validation ensemble; for Task 1, the training set is augmented with mid-RT volumes from the same patients, and for Task 2, a three-channel input is used that stacks the mid-RT volume, the registered pre-RT volume, and the pre-RT label mask. The paper reports validation DSC agg of 0.74 for both the 5-fold and the 15-fold ensemble on Task 2, and blind test results on 50 patients of DSC agg 0.81 (GTVp 0.77, GTVn 0.85) for Task 1 and 0.70 (GTVp 0.54, GTVn 0.86) for Task 2. The conclusion claims that the 15-fold ensemble had a positive effect on segmentation performance in both tasks.","tokens_in":10509,"tokens_out":18357,"duration_ms":159852,"significance":"If the blind test scores are taken at face value, the proposed pipeline is competitive on both tasks, and the per-class breakdown highlights an important open problem: mid-RT GTVp segmentation (DSC agg 0.54) is substantially worse than pre-RT GTVp segmentation (0.77). The paper has several genuine strengths: the aggregated Dice metric is clearly defined (Eq. 1); the null result of the 5-fold versus 15-fold comparison is reported openly rather than hidden; the abandoned approaches (a custom MONAI U-Net and fine-tuned SAM) are described transparently; and the source code is publicly released on Zenodo (§2.5, ref. [21]), which makes the empirical measurements reproducible. These factors support the credibility of the blind test results. At the same time, the stated methodological novelty — the 15-fold ensemble — is not supported by any comparison in the manuscript, and the Task 2 multi-channel input is not ablated, so the contribution as framed is substantially overstated.","major_comments":[{"comment":"The conclusion that the 15-fold ensemble 'displayed to have a positive effect on the segmentation performance' is internally contradicted by §2.5, which reports that 'A comparison of training 5-folds and 15-folds resulted in a similar validation DSC agg of 0.74 for both models, both trained for 1250 epochs.' Because the fold count is the paper's only methodological innovation, this null result leaves the central claim of the title, abstract, and conclusion unsupported. The comparison was also performed only for Task 2, so the conclusion's assertion that the benefit holds 'in both pre-RT and mid-RT T2-weighted MRI volumes' has no experimental basis; the related assertion in §3.1 that the 15-fold ensemble provides a larger quality gain than test-time augmentation is likewise unsupported. The authors should either supply evidence of a benefit of 15 folds (for example, per-class and per-fold validation scores for both fold counts, a paired per-patient comparison, or an analysis of prediction variability across folds) or remove the superiority claim and reframe the paper as a challenge-participation report.","section":"§2.5 and §4"},{"comment":"The three-channel input for Task 2 (mid-RT MRI, registered pre-RT MRI, and the pre-RT label mask) was adopted without an ablation study; §3.2 states only that the authors 'opted for including' these channels. Because of this, the contribution of each channel to the final DSC agg is unknown, and the sharp GTVp degradation in Task 2 (DSC agg 0.54 versus 0.77 in Task 1) could plausibly be caused by misregistration of the pre-RT channel, by a learned reliance on the pre-RT mask, or by the intrinsic difficulty of the mid-RT data; the manuscript does not test any of these explanations. The use of the expert pre-RT label mask as an inference-time input is also a notable design choice that deserves explicit discussion, since it means the method depends on the availability of expert annotations at test time. A minimal ablation (for example, training with only the mid-RT volume) and a failure-case analysis for mid-RT GTVp would make the design defensible and would directly address the paper's largest performance drop.","section":"§3.2 and §4"},{"comment":"For Task 1, no fold-wise validation DSC agg is reported; the results section provides only a two-patient preliminary score (0.89) and the final 50-patient test score (0.81), so the reader cannot assess model selection for Task 1 or compare it with Task 2, for which validation results are reported. Relatedly, the paper does not specify whether the cross-validation splits are patient-level, which matters for Task 1 because each fold's 280 training samples consist of pre-RT and mid-RT volumes from the same 140 patients; if the split were volume-level, a validation patient's mid-RT volume could leak into the training set. Please report fold-wise validation scores for both tasks and state explicitly how the folds were constructed.","section":"§3.1"}],"minor_comments":[{"comment":"The keyword 'HNTSMR24' should read 'HNTSMRG-24' to match the challenge name used in the abstract and in Section 2.1.","section":"Keywords"},{"comment":"In the sentence 'This framework configures a U-Net architecture [26], , alongside the hyper-parameters and data processing steps,' the doubled comma after the citation should be removed.","section":"§4, first paragraph"},{"comment":"In the sentence 'which reduces the the large impact of small GTV volumes,' the doubled 'the' should be removed.","section":"§2.4"},{"comment":"In the sentence reporting GTVn statistics, '14,001 (259 voxels, with 20 patients having no GTVn)' is confusing; the parenthetical appears to be intended as a count of GTVn volumes rather than voxels, and the sentence should be rewritten to match the GTVp reporting format immediately above it.","section":"§2.1"},{"comment":"The selection of 1250 over 1000 epochs is justified by improved DSC agg in 11 of 15 and 12 of 15 folds for GTVp and GTVn, respectively, with an aggregate improvement from 0.73 to 0.74; a two-sided sign test on 11 of 15 is not significant at the 0.05 level (p ≈ 0.12), so this justification should be presented as marginal or supported by an appropriate statistical test.","section":"§2.5"},{"comment":"The abstract states that the pre-RT training data (150 pre-RT volumes and masks) was augmented with mid-RT data, whereas Section 3.1 reports per-fold training on 140 pre-RT and 140 mid-RT scans; the abstract should clarify that the stated augmentation is applied per fold of the 15-fold cross-validation.","section":"Abstract and §3.1"},{"comment":"The phrase 'e.g.99.8% of the voxels is background' should read 'e.g., 99.8% of the voxels are background.'","section":"§2.2"},{"comment":"The paper does not discuss the gap between the Task 2 validation DSC agg of 0.74 and the blind test DSC agg of 0.70, nor the gap between the two-patient preliminary scores (0.89 for Task 1 and 0.75 for Task 2) and the corresponding final test scores (0.81 and 0.70); brief comments on these gaps would help the reader judge generalization.","section":"§3"},{"comment":"For the 5-fold versus 15-fold comparison, only the aggregate validation DSC agg of 0.74 is reported for both configurations; reporting the per-class GTVp and GTVn scores would be more informative, since the two structures show very different behavior on the final test set.","section":"§2.5"}],"recommendation":"major_revision","confidential_remarks":"This is essentially a challenge short paper, and its headline claim is contradicted by its own validation data. For this journal, I would ask the authors to either provide a genuine head-to-head comparison supporting the 15-fold choice or drop the superiority claim from the title, abstract, and conclusion. The two scientifically most valuable next analyses are an ablation of the Task 2 input channels and a failure analysis of the mid-RT GTVp result, which shows a dramatic drop to 0.54. If the journal's scope includes challenge-participation reports, the revised version could fit after these changes; if a methodological advance is required, the current framing is too thin."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: this is a straightforward HNTSMRG-24 challenge write-up, and the blind-test DSC agg numbers (0.81 for Task 1, 0.70 for Task 2) are useful reference points for MRI-based head and neck tumor segmentation. Code is on Zenodo, the evaluation metric is externally defined, and the results are credible. But the paper's central methodological claim—that 15-fold cross-validation improves performance—is not supported by the evidence in the paper itself.\n\nWhat is actually new: the blind-test results for this challenge, plus specific data-augmentation choices (using mid-RT scans to augment the pre-RT task, and a three-channel input for mid-RT). Those are incremental variations on nnU-Net V2. The paper does what a challenge submission should do: it describes the method, reports validation and test metrics, and releases code.\n\nSoft spots, in proportion:\n\n(1) The 15-fold claim is the paper's title, abstract, and conclusion, but Section 2.5 reports that 5-fold and 15-fold both reach validation DSC agg 0.74. The authors frame 15-fold as 'increased robustness and variability,' yet their own comparison shows no gain. The conclusion sentence overstates. This is not an external attack; it is an internal contradiction.\n\n(2) No ablation for the three-channel input. The mid-RT model uses the registered pre-RT MRI and its label mask as extra input channels. The mask is available at test time, so it is not leakage, but without an ablation the contribution of each channel is unknown. Poor registration could be one reason the GTVp score drops from 0.77 (Task 1) to 0.54 (Task 2). The authors mention volume reduction and contrast, but they do not analyze the failure cases.\n\n(3) The comparisons to SAM and the custom MONAI pipeline are cursory. That is acceptable in a challenge report, but it means the paper does not really test alternatives with equal rigor.\n\nThe empirical measurements themselves look sound: the test set is blind, the metric is defined by the organizers, and the reported numbers are plausible. The paper is worth refereeing for the benchmark value alone, but the overclaimed 15-fold advantage needs to be corrected or removed.\n\nWho this is for: researchers working on HNTSMRG or on MRI-based adaptive radiotherapy segmentation. They will want the blind-test numbers. I would send this to a serious referee, but the referee should push for an honest revision of the central claim.","headline":"A competent challenge report with real blind-test numbers, but the headline claim about 15-fold cross-validation is directly contradicted by the paper's own validation results.","tokens_in":11010,"tokens_out":1890,"would_cite":false,"duration_ms":21663,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 15-fold nnU-Net ensemble segments head and neck tumors on MRI with a blind-test Dice score of 0.81 pre-treatment and 0.70 mid-treatment.","keywords":["head and neck cancer","MRI segmentation","nnU-Net","gross tumor volume","cross-validation ensemble","adaptive radiotherapy","deep learning","HNTSMRG-24"],"falsifier":"Train the same 15-fold nnU-Net ensemble on the mid-RT task using only the single mid-RT MRI channel, keeping everything else identical, and compare the aggregated Dice on the same blind test set; if the score does not drop, the two extra input channels are not responsible for the reported performance.","tokens_in":10128,"feed_emoji":"🩻","tokens_out":4402,"duration_ms":39152,"temperature":0.7,"pith_summary":"This paper reports an automatic segmentation method for head and neck tumors on T2-weighted MRI, developed for the HNTSMRG-24 challenge. The authors show that a 15-fold cross-validation ensemble of the nnU-Net V2 framework, rather than the standard 5-fold ensemble, reaches a blind-test aggregated Dice coefficient of 0.81 for pre-radiotherapy volumes (0.77 for primary tumors, 0.85 for lymph nodes) and 0.70 for mid-radiotherapy volumes (0.54 and 0.86). For the pre-treatment task, they doubled the training set by adding the corresponding mid-treatment scans; for the mid-treatment task they fed the network a three-channel input of the mid-treatment MRI, the registered pre-treatment MRI, and its label mask. The work matters because reliable automatic contouring on MRI could support adaptive radiotherapy, where target volumes change during treatment.","feed_headline":"Tumor MRI segmentation hits 0.81 Dice with 15-fold nnU-Net","feed_subtitle":"Blind test of 50 patients shows pre-treatment contours are accurate, while mid-treatment primary tumors remain the hard case at 0.54.","key_machinery":"The central machinery is the nnU-Net V2 framework in its 3D full-resolution configuration, an automated pipeline that determines preprocessing, architecture, and hyperparameters from the training data. On top of this, the authors replace the default 5-fold cross-validation ensemble with a 15-fold one, so each fold trains on 140 of 150 patients and the final prediction averages 15 models. For the mid-treatment task, the input is a three-channel volume: the mid-RT MRI, the pre-RT MRI registered onto it, and the corresponding label mask of the pre-treatment volumes; the label mask is treated as an image channel, providing prior anatomical context.","core_discovery":"Using the nnU-Net V2 3D full-resolution configuration, the authors trained a 15-fold cross-validation ensemble for both challenge tasks. On the blind test set of 50 patients, the method achieved an aggregated Dice coefficient of 0.81 for Task 1 (pre-RT GTVp and GTVn segmentation) and 0.70 for Task 2 (mid-RT), with lymph-node segmentation consistently strong (0.85 and 0.86) and primary-tumor segmentation the limiting factor, dropping from 0.77 pre-RT to 0.54 mid-RT. The authors attribute the mid-RT drop to tumor shrinkage and reduced contrast, and they conclude that increasing the ensemble from 5 to 15 folds improves robustness and variability.","pith_inferences":["The paper does not report an ablation on the registered pre-RT MRI and mask channels for the mid-RT task, so whether these channels actually help, or whether a simpler single-channel model performs the same, remains open.","The reported validation Dice was the same (0.74) for 5-fold and 15-fold ensembles in the mid-RT task, so the paper's claim that more folds improve performance rests on a single blind-test comparison rather than a controlled experiment.","A testable extension would be to feed the mid-RT network with the mid-RT mask from the previous planning day (or the registered pre-RT mask as done here) and measure how sensitive the GTVn accuracy is to registration errors."],"forward_implications":["If the method generalizes, it offers a concrete recipe for automatic GTVp and GTVn contouring on T2-weighted MRI for both treatment-planning and mid-treatment adaptive workflows.","The 15-fold ensemble, at 15 forward passes per case, still fits within the 20-minute inference limit on a T4 GPU, so the stability gain is practically affordable.","The mid-treatment primary-tumor score of 0.54 marks the hardest part of the problem, so future work aimed at low-contrast, shrunken GTVp has an immediate target.","The authors' decision to train the pre-RT model on both pre-RT and mid-RT scans shows that combining time-point data can help when the test distribution is the pre-RT domain."],"supporting_citations":[{"why":"Supplies the nnU-Net self-adapting framework that plans the preprocessing, architecture, and hyperparameters used in both tasks.","marker":"[14]"},{"why":"Provides the updated nnU-Net V2 version that the authors adopt as the base pipeline.","marker":"[15]"},{"why":"Defines the aggregated Dice Similarity Coefficient used as the evaluation metric for the challenge and for all reported scores.","marker":"[3]"},{"why":"Introduces the U-Net architecture that the nnU-Net framework configures and that the 3D approach builds upon.","marker":"[26]"}],"fun_headline_variants":["15-fold nnU-Net ensemble scores 0.81 Dice on head and neck MRI","Mid-RT primary tumor hardest for MRI segmentation: 0.54 Dice","nnU-Net with 15-fold CV hits 0.81 pre-RT, 0.70 mid-RT Dice","Lymph nodes segmented well (0.86), primary tumor lags (0.54) in mid-RT","More folds, better robustness: 15-fold nnU-Net for MRI segmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that adding the registered pre-RT MRI volume and its label mask as extra input channels improves mid-RT segmentation is the load-bearing premise; the authors chose these channels without an ablation study, so poor registration or mask errors could be hurting the 0.54 GTVp score.","fun_headline_variants_meta":{"raw":{"variants":["15-fold nnU-Net ensemble scores 0.81 Dice on head and neck MRI","Mid-RT primary tumor hardest for MRI segmentation: 0.54 Dice","nnU-Net with 15-fold CV hits 0.81 pre-RT, 0.70 mid-RT Dice","Lymph nodes segmented well (0.86), primary tumor lags (0.54) in mid-RT","More folds, better robustness: 15-fold nnU-Net for MRI segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000364,"raw_usage":{"total_tokens":2034,"prompt_tokens":1091,"completion_tokens":943,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":707,"completion_tokens_details":{"reasoning_tokens":819}},"tokens_in":707,"tokens_out":943,"duration_ms":8009,"temperature":1.0,"reasoning_tokens":819,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:28:03.441138+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 15-fold nnU-Net ensemble on the mid-RT task using only the single mid-RT MRI channel, keeping everything else identical, and compare the aggregated Dice on the same blind test set; if the score does not drop, the two extra input channels are not responsible for the reported performance.","supporting_citations":[],"review_version":1}