{"id":"56af47b0-e048-4631-b529-04004ec72166","arxiv_id":"2411.15872","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A MedNeXt ensemble with finetuning and validation-set-tuned postprocessing achieved average Dice 0.896 on BraTS-2024 SSA and 0.830 on BraTS-2024 Pediatric validation sets.","lead":"This paper describes a MedNeXt-based deep learning pipeline for brain tumor segmentation in the BraTS 2024 Africa and Pediatrics tasks, reporting validation Dice scores of 0.896 and 0.830. It combines model ensembling, finetuning, and postprocessing thresholds tuned partly on the validation set.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation-set tuning of postprocessing thresholds (Sec 2.3) makes the reported 0.896/0.830 leaderboard scores biased as generalization estimates; the Pediatric 0.830 is the best of six validation-selected runs.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: the validation set was used to manually tune postprocessing thresholds (Sec 2.3), so the reported validation scores are not an unbiased measure of generalization. I agree with that assessment and with the conditional verdict. The concern is concrete: Table 1 quantifies a 0.0033 Dice gain and 0.69 HD95 gain from the manual threshold adjustment for SSA, and Table 2 shows the Pediatric average is selected from six validation-queried configurations, with a much larger spread (~0.03 Dice). Without a held-out evaluation or confidence intervals, the headline numbers overstate certainty about new-data performance. The issue is addressable by locked-pipeline validation or test-set reporting, so rejection is not warranted. A secondary reproducibility issue is the empty citation in Section 2.3 ('we followed the approach outlined in .'), which should be filled but does not affect the central claim. Because my concern is the same one the reader already identified, the verdict remains conditional.","tokens_in":9131,"tokens_out":8837,"duration_ms":76937,"concrete_test":"Freeze the full pipeline (ensemble composition, finetuning checkpoint, and binarization thresholds) using only the 5-fold CV sweep, without any validation leaderboard queries. Submit one validation evaluation with these locked settings and compare with row 6 of Table 1 (0.8963 SSA) and the Pediatric 0.830. If locked-pipeline validation Dice is outside the best-of-N spread (e.g., below 0.89 for SSA or below 0.81 for Peds), the manual validation adjustment is material and the headline numbers should be labeled as leaderboard-tuned rather than unbiased estimates. Alternatively, if per-case validation scores are available, compute a paired bootstrap 95% CI for the row 5 vs row 6 difference in Table 1; a CI excluding 0 for Dice or HD95 would confirm the threshold tuning landed outside noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is the validation Dice/HD95 performance, but Section 2.3 states: 'We tuned postprocessing hyperparameters using a Wandb sweep on the 5-fold CV and then manually adjusted them to fit the unseen validation set.' That breaks the independence of the validation set as an unbiased estimate of generalization. Table 1 shows the effect for SSA: with the CV-default threshold configuration (row 5, reinitialized finetune) validation Dice is 0.8930/HD95 15.37; after manual validation-set adjustment to 0.7/0.7/0.5 thresholds (row 6), it is 0.8963/14.68. For Pediatrics, Table 2 shows the abstract's 0.830 is the best among six learning-rate and minimum-size-threshold combinations; the LR=0.0027 runs average around 0.79-0.80. Selecting the best row after seeing validation scores inflates expected performance on any truly unseen test set, and no confidence intervals or per-case variability are reported, so the reader cannot separate genuine improvement from selection noise. The method is otherwise internally consistent and follows standard MedNeXt/ensemble practice; this is a validity-of-evidence concern rather than an algorithmic contradiction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript describes a MedNeXt-based segmentation pipeline for the BraTS 2024 SSA and Pediatric tumor tasks. The authors train MedNeXt-B and MedNeXt-M models on combined adult glioma and SSA data, use the schedule-free AdamW optimizer, 5-fold cross-validation ensembles, and postprocessing with minimum-size and binarization thresholds. They report average validation Dice scores of 0.896 (SSA) and 0.830 (Pediatric) with HD95 of 14.682 and 37.508, and they ablate patch size, finetuning strategy, learning rate, and postprocessing thresholds.","tokens_in":9367,"tokens_out":4514,"duration_ms":39714,"significance":"The paper provides a practical, reproducible challenge pipeline with a public code repository and a recent optimizer, and its ensembles and finetuning are standard and internally consistent. If the reported validation numbers were unbiased estimates of generalization, the results would be competitive for the two BraTS 2024 tasks. However, the central evidence is weakened by explicit tuning of postprocessing hyperparameters on the validation set and by selecting the best of several validation-set runs, so the quantitative claims need to be reframed or supplemented with cross-validation or held-out test results before they can be taken at face value.","major_comments":[{"comment":"The sentence in Section 2.3, 'We tuned postprocessing hyperparameters using a Wandb sweep on the 5-fold CV and then manually adjusted them to fit the unseen validation set,' breaks the independence of the validation set as a generalization estimate. Because the binarization thresholds and minimum tumor sizes were adjusted using validation-set performance, the reported Dice of 0.896 and 0.830 are not unbiased estimates for truly unseen data. The effect is visible in Table 1: the average Dice improves from 0.8930 with CV-default thresholds (row 5) to 0.8963 after manual adjustment (row 6), and HD95 improves from 15.37 to 14.68. The authors should report the 5-fold cross-validation performance of the final configuration, or clearly label the validation numbers as leaderboard results obtained with validation-set-informed postprocessing and report an independent test-set result if available.","section":"Section 2.3; Tables 1 and 2"},{"comment":"The abstract's Pediatric Dice of 0.830 is the best among the six learning-rate and minimum-size-threshold combinations shown in Table 2; selecting the best row after observing validation scores inflates the expected performance on future data through selection bias. For example, the 0.0027 learning-rate runs in Table 2 have average Dice below 0.80, while the 0.0005 runs reach 0.853 in some rows. The authors should report all configurations considered, provide the selection rule, and give per-case variability (e.g., standard deviation across validation subjects) or confidence intervals for the chosen configuration so that the reader can separate genuine improvement from selection noise.","section":"Section 3.2, Table 2"},{"comment":"The conclusion calls the Africa results 'state-of-the-art average Dice scores above 0.895' without comparing to other BraTS 2024 submissions or published methods on the same tasks. Since 'state-of-the-art' is a comparative claim, the authors should provide a comparison table with leaderboard positions, other challenge entries, or prior published results on the BraTS SSA and Pediatrics validation sets, or temper the claim to 'competitive with' or 'among the top entries.'","section":"Section 4"},{"comment":"The text names configurations such as 'MedNeXt Finetuned False 0.7' (Dice 0.895) and 'MedNeXt Finetuned True 0.7,0.5' (Dice 0.896), but Table 1 does not show finetuning initialization and threshold values as explicit columns, so the mapping between table rows and the described models is ambiguous. This makes it difficult to verify which configuration produced the headline number. Please add explicit columns for finetuning initialization (True/False), binarization thresholds, and minimum-size thresholds, and align the table rows with the bullet descriptions.","section":"Section 3.1, Table 1"}],"minor_comments":[{"comment":"The sentence 'For the objective function (loss), deep supervision, brain MRI preprocessing, and data augmentation, we followed the approach outlined in .' contains an empty citation; please fill in the intended reference.","section":"Section 2.3"},{"comment":"The bullet 'MedNeXt Base achieved an average Dice of 0.889, which was the highest among the non-finetuned models. It performed well, especially in segmenting WT, with a Dice score of 0.932.' is duplicated verbatim in the list of MedNeXt models; please remove the duplicate.","section":"Section 3.1"},{"comment":"The column header for Table 2 is garbled ('Dice Scores HD 95ETTCWTETTCWTNETCCCED ET TC WT NETCCC ED'); please reformat the table so that the six tumor-region columns and their metric groupings are clearly labeled.","section":"Section 3.2, Table 2"},{"comment":"The text says sliding window inference uses 'aX% (X ∈ {50, 70}) overlap' but does not state which overlap value was used for each task or for the final submission; please specify this in the experimental setup.","section":"Section 2.4"},{"comment":"The final ensemble composition is not specified: the text mentions ensembling 'models derived from different training strategies' but does not state which models (Base, Medium, finetuned variants) were combined, in what number, and with what weighting for the reported validation numbers. Please describe the final ensemble explicitly.","section":"Section 2.4, Section 3"},{"comment":"For the SSA dataset, the three sub-regions are introduced as ET, NETC, and SNFH, but the results in Table 1 report ET, TC, and WT. Please clarify the relationship between these label sets (e.g., TC = ET + NETC and WT includes surrounding FLAIR hyperintensity) to avoid confusion.","section":"Section 2.1"},{"comment":"There are minor typographical issues, including 'accross' in Section 3.1 and 'LesionWise' in the Table 2 caption; please proofread for spelling and capitalization.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is essentially a BraTS challenge report. The main risk is that the headline validation scores are presented as generalization estimates despite the authors' own statement in Section 2.3 that postprocessing thresholds were manually adjusted to fit the validation set, and despite selecting the best of several validation-set runs in Section 3.2. This is fixable within the manuscript's scope by reporting cross-validation metrics, clearly separating tuning from evaluation, and either reporting test-leaderboard results or explicitly labeling the numbers as validation-set-informed. The paper would also benefit from a direct comparison with other BraTS 2024 entries before using the term 'state-of-the-art.' I recommend major revision rather than rejection because the underlying pipeline is sound and the issue is about evidence framing rather than an algorithmic contradiction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent BraTS 2024 challenge report that shows MedNeXt works on the SSA and pediatric tasks, but the headline numbers are not trustworthy as generalization estimates because the postprocessing thresholds were tuned on the validation set.\n\nWhat's actually new: the combination of MedNeXt-B/M, schedule-free AdamW, pretraining on adult glioma plus finetuning on SSA, and ensembling for these two new BraTS tasks. That's a legitimate engineering extension, not a new architecture. The paper does the work of ablating patch size, finetuning initialization, learning rates, and threshold choices, and it is honest about what was tried. The GitHub repo is a plus.\n\nSoft spots: the main one is Section 2.3: 'We tuned postprocessing hyperparameters using a Wandb sweep on the 5-fold CV and then manually adjusted them to fit the unseen validation set.' That breaks the independence of the validation set. Table 1 shows the effect: before manual threshold adjustment the SSA Dice is 0.8930; after adjustment it is 0.8963. Table 2 makes the same point for pediatrics: the reported 0.830 is the best of six validation-selected runs, and the LR=0.0027 runs sit around 0.79-0.80. No error bars or per-case variability are reported. So the abstract's numbers are optimistic as estimates of true unseen performance. This is addressable: hold out a separate test set, report nested CV, or at least show the threshold sweep as a bracketing analysis. Minor issues: a missing citation in Section 2.3 ('the approach outlined in .') and a duplicated bullet in Section 3.1.\n\nThe method itself is internally consistent and follows standard MedNeXt/ensemble practice. The central algorithmic claim holds: MedNeXt with schedule-free AdamW, adult pretraining, and finetuning does well on these populations. It's the evidence that is overstated, not the method.\n\nWho it's for: teams working on BraTS or clinical deployment in low-resource settings, and anyone who wants a concrete example of why validation-set tuning matters.\n\nRecommendation: send it to review, but with the expectation of revision. The evaluation protocol needs fixing before the numbers can be taken at face value.","headline":"Solid BraTS 2024 challenge report, but the headline Dice numbers are inflated by validation-set threshold tuning and should not be read as unbiased generalization.","tokens_in":9955,"tokens_out":2239,"would_cite":false,"duration_ms":20253,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using the MedNeXt architecture with ensembling, finetuning, and size-threshold postprocessing, this paper reports average Dice scores of 0.896 on the BraTS 2024 sub-Saharan Africa validation set and 0.830 on the pediatric validation set.","keywords":["brain tumor segmentation","glioma","MedNeXt","BraTS 2024","BraTS-SSA","BraTS-PEDs","distribution shift","3D MRI"],"falsifier":"Fix every training choice and postprocessing threshold using only the five cross-validation folds, then freeze the pipeline and recompute Dice and HD95 on the same 35 African and pediatric validation cases; if the scores fall clearly below 0.896 and 0.830, the gap is the validation-tuning contribution. Alternatively, run the released pipeline unchanged on a newly collected, never-seen cohort of African and pediatric MRIs and compare.","tokens_in":8924,"feed_emoji":"🧠","tokens_out":10337,"duration_ms":88130,"temperature":0.7,"pith_summary":"Automated brain-tumor segmentation models trained on adult glioma MRI data that dominates existing benchmarks often fail when applied to other populations, where image quality or anatomy differ. This paper claims that its MedNeXt-based approach—trained with a schedule-free optimizer, combining adult glioma and local training data, ensembled across five cross-validation folds, and finished with size-threshold postprocessing—can handle both the BraTS 2024 sub-Saharan Africa and pediatric tasks. On the challenge validation sets the pipeline reports an average Dice Similarity Coefficient, a standard overlap accuracy measure, of 0.896 for the African data and 0.830 for the pediatric data, with 95th-percentile Hausdorff distances, a boundary-error measure, of 14.682 and 37.508. If those numbers hold on genuinely unseen data, the approach is a practical recipe for deploying automatic tumor segmentation in settings with distribution shift.","feed_headline":"MedNeXt hits 0.896 Dice on Africa, 0.830 on pediatric tumors","feed_subtitle":"Trained on adult glioma plus small local cohorts, it handles MRI quality and anatomy shifts.","key_machinery":"The load-bearing mechanism is MedNeXt itself: a convolutional U-Net whose blocks replace standard convolutions with depthwise 3D convolutions that mimic the large attention windows of transformers, followed by channel expansion and compression, with deep supervision at each decoder stage. The paper uses the Base and Medium sizes and ensembles the five models from 5-fold cross-validation by averaging their output probability maps. A schedule-free AdamW optimizer with learning rates near 0.0027 replaces the usual cosine-annealed schedule. Postprocessing removes predicted tumor components smaller than experimentally chosen voxel thresholds, and for Africa the last ten decoder layers are finetuned on the target population after training on combined adult and African data. These components together carry the performance; the paper's ablations show each addition raises Dice or reduces boundary error.","core_discovery":"The paper's central claim is that MedNeXt, a U-Net-like network built from inverted-bottleneck convolutional blocks, can be adapted to segment gliomas under distribution shift, and that the adaptation comes mostly from data and training choices rather than architectural novelty. For the sub-Saharan Africa task the authors train MedNeXt-Base and MedNeXt-Medium on the 60 African training scans together with 1,470 adult glioma scans, finetune the last decoder layers on the African data alone, and ensemble the five cross-validation models by averaging probability maps; the best configuration reports average Dice 0.896 and HD95 14.682 on the 35-case validation set. For the pediatric task they train MedNeXt-Base on 464 pediatric high-grade glioma cases, predict enhancing tumor, tumor core, and whole tumor, and use minimum-size thresholds to remove small false-positive components, reporting average Dice 0.830 and HD95 37.508. The paper attributes the remaining performance gap to data heterogeneity and to children's different anatomy, and it treats postprocessing thresholds as an important lever, noting that thresholds were tuned partly against the validation set.","pith_inferences":["A natural test the paper does not run is to freeze the full pipeline, including thresholds, after cross-validation and evaluate on the eventual test set; if validation-tuned thresholds were a major source of the reported scores, the test-set gap would show it.","The same recipe could transfer to other underrepresented populations: combine a large adult benchmark, finetune decoder layers on a small local cohort, and tune thresholds per region; the paper's SSA results make this concrete enough to try.","The Base-plus-Medium ensemble scored lower Dice than either model alone in the Africa task, suggesting simple equal-weight probability averaging is not always beneficial; selecting or weighting ensemble members by validation performance might recover the lost margin.","For clinical deployment, the large pediatric HD95 implies some segmentations have gross boundary outliers; combining the size-threshold step with a shape prior or outlier-aware loss would be the next lever to test."],"forward_implications":["If the Africa ablations generalize, the recipe is transferable: pretrain on a large adult glioma set, finetune only the decoder on a small target-population cohort, and tune per-region binarization thresholds; each step moved average Dice upward, ending at 0.896.","A schedule-free optimizer near learning rate 0.0027 with no weight decay matched or beat the cosine-annealed AdamW baseline in both tasks, so explicit learning-rate schedules are not required for strong segmentation.","The best pediatric configuration used learning rate 0.0005 and voxel thresholds of 50, 75, and 250 for enhancing tumor, tumor core, and whole tumor, averaging Dice 0.830 and HD95 37.508; other thresholds degraded both metrics.","Pediatric HD95 of 37.5 is much larger than Africa HD95 of 14.7, showing boundary accuracy lags overlap accuracy when anatomy and image quality differ, and overlap-focused metrics alone would mask that gap."],"supporting_citations":[{"why":"Defines the MedNeXt architecture used as the backbone for all models.","marker":"[18]"},{"why":"Provides the BraTS-Africa dataset of 60 training and 35 validation sub-Saharan African MRI cases with radiologist annotations.","marker":"[2]"},{"why":"Provides the BraTS-PED dataset of 464 pediatric high-grade glioma cases and defines the six labeled regions, of which three are predicted.","marker":"[9]"},{"why":"Supplies the 1,470 BraTS-2021 adult glioma cases used to augment the African training set.","marker":"[3]"},{"why":"Introduces the schedule-free AdamW optimizer that replaces the cosine-annealed baseline in all training runs.","marker":"[6]"},{"why":"Supplies the region-based Dice-plus-focal loss and the general preprocessing and augmentation recipe followed by the authors.","marker":"[8]"},{"why":"Underpins the ensemble strategy by arguing that aggregating diverse models at probability level improves robustness.","marker":"[19]"}],"fun_headline_variants":["MedNeXt adapts to MRI shifts: 0.896 Dice on Africa, 0.830 on kids","Ensembled MedNeXt tackles distribution shift: 0.896 African, 0.830 pediatric Dice","MedNeXt robust to MRI quality: 0.896 Dice Africa, 0.830 in pediatric tumors","0.896 Dice on low-res MRI, 0.830 on children: MedNeXt adapts","MedNeXt handles MRI diversity: 0.896 Africa, 0.830 pediatric Dice"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The validation set is treated as an unbiased estimate of generalization even though the postprocessing thresholds were manually adjusted to fit it; if that adjustment leaks information, the reported Dice and HD95 numbers overstate performance on genuinely new scans.","fun_headline_variants_meta":{"raw":{"variants":["MedNeXt adapts to MRI shifts: 0.896 Dice on Africa, 0.830 on kids","Ensembled MedNeXt tackles distribution shift: 0.896 African, 0.830 pediatric Dice","MedNeXt robust to MRI quality: 0.896 Dice Africa, 0.830 in pediatric tumors","0.896 Dice on low-res MRI, 0.830 on children: MedNeXt adapts","MedNeXt handles MRI diversity: 0.896 Africa, 0.830 pediatric Dice"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000383,"raw_usage":{"total_tokens":2100,"prompt_tokens":1090,"completion_tokens":1010,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":706,"completion_tokens_details":{"reasoning_tokens":877}},"tokens_in":706,"tokens_out":1010,"duration_ms":7544,"temperature":1.0,"reasoning_tokens":877,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:46:13.997217+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix every training choice and postprocessing threshold using only the five cross-validation folds, then freeze the pipeline and recompute Dice and HD95 on the same 35 African and pediatric validation cases; if the scores fall clearly below 0.896 and 0.830, the gap is the validation-tuning contribution. Alternatively, run the released pipeline unchanged on a newly collected, never-seen cohort of African and pediatric MRIs and compare.","supporting_citations":[{"cited_title":"ArXiv (2023)","cited_arxiv_id":null,"evidence_quote":"Provides the BraTS-Africa dataset of 60 training and 35 validation sub-Saharan African MRI cases with radiologist annotations."},{"cited_title":"In: Crimi, A., Bakas, S","cited_arxiv_id":null,"evidence_quote":"Supplies the region-based Dice-plus-focal loss and the general preprocessing and augmentation recipe followed by the authors."},{"cited_title":"In: Bakas, S., Crimi, A., Baid, U., Malec, S., Pytlarz, M., Baheti, B., Zenk, M., Dorent, R","cited_arxiv_id":null,"evidence_quote":"Underpins the ensemble strategy by arguing that aggregating diverse models at probability level improves robustness."}],"review_version":1}