{"id":"ce91205a-2e8e-4f0d-83af-1402890ab8d1","arxiv_id":"2505.16561","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Auto-nnU-Net, an AutoML extension of nnU-Net, reports the highest mean test Dice score on the Medical Segmentation Decathlon while optimizing both accuracy and training runtime.","lead":"Auto-nnU-Net adds automatic hyperparameter tuning and neural architecture search to the standard nnU-Net segmentation pipeline, and reports improved Dice scores on six of ten Medical Segmentation Decathlon datasets. It also introduces a resource-aware search method, Regularized PriorBand, that considers training time as a second objective.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Concern: full nnU-Net baseline with ensembling/post-processing may erase the claimed 2 of 10 improvements, since only stripped baselines are compared.","rationale":"The reader's weakest_assumption is exactly the concern that the stripped baselines do not represent the nnU-Net whose performance the paper claims to improve. The manuscript itself admits this in the Limitations paragraph: results are 'based on the 3D U-Net architecture without post-processing or ensembling, which may not fully reflect the original nnU-Net's performance.' This is the load-bearing assumption because the central claim is comparative: AutoML 'improves nnU-Net' on most datasets. A comparison against a deliberately degraded baseline cannot establish that claim. Moreover, the per-dataset numbers show the improvements are small on most datasets, making the comparison sensitive to omitted pipeline components. This is an empirical concern, not a logical or methodological contradiction in the AutoML framework itself; the framework, Regularized PriorBand, and the evaluation across all 10 MSD datasets are valuable. My recommendation is CONDITIONAL, matching the reader's verdict: the paper should be published only if the claim is scoped to the stripped-pipeline setting or if the full nnU-Net baseline is added. I agree with the reader that this is the weakest assumption, and my proposed test would directly settle the concern.","tokens_in":24767,"tokens_out":1657,"duration_ms":11437,"concrete_test":"Run the official nnU-Net pipeline (all configurations: 2D, 3D full-resolution, 3D cascade where applicable) with default ensembling and post-processing on at least D01, D05, D07, and D10, using the same 5-fold splits and test-submission protocol; then compare official-pipeline nnU-Net test DSC against Auto-nnU-Net's Table 1 values. If official nnU-Net's per-dataset test DSC meets or exceeds Auto-nnU-Net on more than two of these datasets, the 'improves nnU-Net on 6 of 10' claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that Auto-nnU-Net 'substantially improves' nnU-Net on 6 of 10 MSD datasets. However, all nnU-Net baselines in Table 1 and Table 6 are the 3D U-Net without post-processing or ensembling, as Section 7 Limitations admits. Published nnU-Net results and its design show that ensembling and post-processing are standard, intended components that typically add meaningful DSC, especially on datasets with high variance or small structures (D01, D05, D10). Table 1 shows Auto-nnU-Net's improvements over the best stripped baseline are: D01 +0.24, D02 +0.10, D03 -0.30 (loss vs ResL), D04 +0.32, D05 +1.38, D06 -1.74 (loss vs ResL), D07 -0.30 (loss vs ResL), D08 -0.12 (loss vs Conv), D09 -0.10 (loss vs ResL), D10 +5.09 (vs Conv, but -0.31 vs ResL). Thus only D04, D05, and D10 show gains over the strongest stripped baseline, and on D10 the ResL baseline is the strongest, making Auto-nnU-Net worse than ResL. The '6 of 10' claim therefore depends on comparing against the weakest baseline family per dataset, and the whole claim could fail if full nnU-Net (ensemble + post-processing) recovers even 0.5-1.5 DSC on most datasets. The limitation is explicitly admitted, making the headline claim outside what the evidence supports.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Auto-nnU-Net, a framework that extends nnU-Net with hyperparameter optimization (HPO), neural architecture search (NAS), and hierarchical NAS (HNAS), jointly optimizing segmentation accuracy and training runtime via a new multi-objective method called Regularized PriorBand. The method is evaluated on all ten Medical Segmentation Decathlon (MSD) datasets using 5-fold cross-validation and official test-set submissions, comparing against three stripped nnU-Net 3D U-Net configurations (Conv, ResM, ResL) and a MedSAM2 baseline. The central claim, stated in the abstract and conclusion, is that Auto-nnU-Net 'substantially improves' nnU-Net on 6 of 10 datasets while remaining on par on the others and maintaining practical resource requirements.","tokens_in":25081,"tokens_out":5782,"duration_ms":42800,"significance":"If the central claim were supported, the paper would make a valuable contribution: it introduces a full AutoML pipeline for the widely used nnU-Net framework, extends PriorBand to multi-objective joint architecture and hyperparameter search, and provides an unusually comprehensive evaluation on all ten MSD datasets, including official test-set scores. The authors also release their code, and the study is large-scale (approximately 60k GPU hours). However, the significance is currently limited by a baseline-comparison issue that the authors themselves acknowledge in the Limitations paragraph: all nnU-Net baselines are 3D U-Nets without the ensembling and post-processing components that are integral to the original nnU-Net pipeline. Because the headline claim is framed as improving 'nnU-Net' rather than a stripped variant, the evidence does not yet support it.","major_comments":[{"comment":"The central claim that Auto-nnU-Net 'substantially improves the segmentation performance of nnU-Net' is not supported by the baselines used. All nnU-Net baselines are the 3D U-Net without ensembling or post-processing, as explicitly stated in the Limitations paragraph ('This study, including results for both nnU-Net and Auto-nnU-Net, is based on the 3D U-Net architecture without post-processing or ensembling, which may not fully reflect the original nnU-Net's performance'). The original nnU-Net pipeline includes both ensembling and post-processing as default inference components, and these are known to typically increase DSC. The observed improvements over the strongest stripped baseline (Table 1) are small (e.g., D01 +0.24, D02 +0.10, D04 +0.32) and could be erased if the full nnU-Net pipeline recovers even a fraction of a point. To support the abstract's claim, the authors must either add a full nnU-Net baseline (or cite published full-nnU-Net test-set results) or reframe the claim as improving the stripped 3D U-Net component of nnU-Net.","section":"Section 7, Table 1, Table 6"},{"comment":"The abstract's claim of improvement on '6 out of 10 datasets' is inconsistent with the official test-set results reported in Table 1, where Auto-nnU-Net is the best-performing method on only five datasets (D01, D02, D04, D05, D10); Section 6.1 correctly states 'ranking best on five out of ten datasets.' The '6 out of 10' figure appears to derive from Table 6 (5-fold cross-validation), where Auto-nnU-Net beats the best stripped baseline on D01, D02, D04, D05, D06, and D10. The abstract and main text should report the same numbers, or explicitly distinguish between validation and test-set claims, to avoid overstating the result.","section":"Abstract, Section 6.1, Table 1, Table 6"},{"comment":"The optimization is run with 'one random seed' (Section 5.3), and no statistical significance testing is reported. Most improvements over the best stripped baseline in Table 1 are small (e.g., D01 +0.24, D02 +0.10, D04 +0.32) relative to the reported standard deviations (e.g., D01 ±24.3, D10 ±32.0). Without multiple optimization seeds or paired significance tests, the word 'substantially' in the abstract is not statistically justified. The authors should either add a significance analysis (e.g., paired tests across folds or bootstrap intervals) or soften the claim to describe the observed differences without causal or generalizing language.","section":"Section 5.3, Table 1"},{"comment":"The MedSAM2 baseline is not a fair comparison: it is fine-tuned for only 100 epochs instead of the 1000 used by the original method (Appendix D.2.1), and its inference requires ground-truth-derived bounding-box prompts. The paper acknowledges this, but still labels MedSAM2 as a 'state-of-the-art competitor' in Section 5.2 and uses it in the Pareto-front comparisons. This baseline choice should be clearly disclosed in the main text, and the comparison should be interpreted as indicative rather than as a head-to-head evaluation.","section":"Section 5.2, Appendix D.2.1, Table 6"},{"comment":"The paper emphasizes that Auto-nnU-Net 'maintain[s] practical resource requirements,' but the total search cost is approximately 60,000 GPU hours (Appendix A). The Pareto-front analyses (e.g., Figure 4) refer to training runtime of individual configurations, not the total optimization budget. The authors should specify whether 'practical resource requirements' refers to the inference/training cost of the final selected model or to the total search cost, and discuss the trade-off accordingly.","section":"Section 5 (Introduction of 'practical resource requirements')"}],"minor_comments":[{"comment":"There is a typo: 'Rule-baesd' should be 'Rule-based'.","section":"Section 3 (Related Work)"},{"comment":"The algorithm is referred to as 'Algoritm 2'; please correct to 'Algorithm 2'.","section":"Appendix C.2"},{"comment":"The phrase 'all test set instance DCSs' should be 'DSC scores' or 'Dice scores' for clarity.","section":"Table 1 caption"},{"comment":"The caption states 'Detailed results for each dataset are stated in Appendix E,' but the figure is already in the main text; consider referring to specific tables for consistency.","section":"Section 6.1, Figure 2 caption"},{"comment":"In the NAS hyperparameter rows, 'Activation' and 'Normalization' are listed as NAS hyperparameters, but they also affect the decoder; clarify the scope of these choices in the text.","section":"Appendix D.2.3, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically interesting and the evaluation is extensive, but the central claim of improving nnU-Net is not supported by the chosen baselines. The authors' own limitation statement admits this. The issue is fixable either by adding a genuine full-nnU-Net comparison or by rewording the claims to refer to the stripped 3D U-Net component. I also note the discrepancy between the abstract's '6 of 10' and the test-set '5 of 10,' which needs reconciliation. Given the paper's scope and the availability of code, I believe a major revision can address these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper builds a genuinely useful AutoML wrapper around nnU-Net, but the headline \"improves nnU-Net on 6 of 10\" only holds against stripped baselines without ensembling or post-processing. The empirical core needs a second pass, but the framework is worth refereeing.\n\nThe genuinely new bits are Regularized PriorBand—PriorBand with non-dominated sorting and crowding distance over accuracy and runtime—and attaching a CFG-based hierarchical NAS space to nnU-Net, with JAHS as the integrated result. That is a sensible engineering contribution, and the code and all-ten-MSD evaluation go beyond what most AutoML-for-MIS papers do. The ablations (HPO, HPO+HNAS), fANOVA, and transfer matrix are useful analysis. Full credit for the public code and the official MSD test set results.\n\nBut the central claim is not backed by the evidence as designed. The nnU-Net baselines are the 3D U-Net without post-processing and ensembling—components that are standard parts of nnU-Net and typically add DSC. The authors admit this in Section 7. Against the best stripped baseline per dataset, test-set gains are tiny on D01 (+0.24) and D02 (+0.10), mixed elsewhere, and the \"6 of 10\" line seems to come from validation, where almost all comparisons are against the weakest baseline. Add a proper full-pipeline nnU-Net (with ensembling, post-processing, and the cascade where applicable) and the list of wins could shrink or vanish. Also, a single random seed for the optimizer, no significance testing, and standard deviations that overlap the gains on several datasets (D01, D06, D10) make the empirical distinctions fragile. The MedSAM2 comparison is also a bit tilted: 100 epochs and a prompt from ground truth at inference, so it is not a like-for-like competitor.\n\nIf the full-pipeline baselines and multiple seeds bring the numbers down, the framework is still a reasonable candidate for \"how to AutoML nnU-Net\", just with a weaker claim. This is a paper for the AutoML and medical imaging crowd, not for someone looking for a settled result. I'd send it to a serious referee—it's an honest, well-organized empirical study with open code, and the flaws are fixable.","headline":"Useful AutoML wrapper around nnU-Net, but the headline claim only holds against stripped baselines without ensembling or post-processing; worth refereeing after a revised comparison.","tokens_in":25617,"tokens_out":3711,"would_cite":true,"duration_ms":28780,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automatically searching over training hyperparameters and network architectures improves nnU-Net's segmentation performance on six of ten Medical Segmentation Decathlon datasets and matches it on the rest, at comparable training cost.","keywords":["automated medical image segmentation","nnU-Net","hyperparameter optimization","neural architecture search","hierarchical NAS","multi-objective optimization","Medical Segmentation Decathlon","PriorBand"],"falsifier":"Run the original nnU-Net pipeline, including its default ensemble selection and post-processing, on the same ten MSD datasets and the same 5-fold splits, then compare mean test DSC against the 77.04% reported for Auto-nnU-Net; if full-pipeline nnU-Net's mean matches or exceeds that value, the claim of improving on nnU-Net on 6 of 10 datasets fails on its own comparison.","tokens_in":24540,"feed_emoji":"🩺","tokens_out":7191,"duration_ms":51844,"temperature":0.7,"pith_summary":"Auto-nnU-Net argues that nnU-Net's fixed hyperparameters and hand-crafted heuristics leave performance on the table, and that treating those choices as searchable variables improves segmentation accuracy. The paper builds an AutoML layer on top of nnU-Net that jointly optimizes training hyperparameters, network architecture, and, in the hierarchical variant, network topology, using Regularized PriorBand to balance accuracy against training runtime. Across all ten Medical Segmentation Decathlon datasets, the resulting configuration beats the convolutional nnU-Net baseline on eight datasets and matches or exceeds all nnU-Net variants on six, with the highest mean test Dice score (77.04%) among the compared methods. A sympathetic reader would care because medical segmentation models are often retrained frequently under tight compute budgets, so a method that finds better models at comparable cost has immediate practical value.","feed_headline":"AutoML search beats nnU-Net on 6 of 10 segmentation tasks","feed_subtitle":"Joint tuning of hyperparameters and architectures lifts mean test Dice to 77.04 percent at comparable compute.","key_machinery":"The load-bearing mechanism is Regularized PriorBand, an extension of the multi-fidelity hyperparameter optimizer PriorBand that turns joint hyperparameter and architecture search into a two-objective problem: minimize $1 - \\text{DSC}$ and training runtime. Regularized PriorBand replaces Successive Halving's single-cost selection with non-dominated sorting and crowding-distance ranking, so promising low-budget configurations are promoted toward higher training budgets only when they are accurate, fast, or a useful trade-off; the final incumbent is chosen by accuracy alone. Architecture search is enabled by encoding architectures as hyperparameters, either directly in a unified configuration space or through CFGUNet, which uses a context-free grammar to generate U-Net topologies with variable depth, block counts, normalization, and nonlinearities.","core_discovery":"The paper's central claim is that a joint hyperparameter and architecture search can replace nnU-Net's fixed and rule-based settings and improve segmentation performance on most tasks without increasing resource requirements. In 5-fold cross-validation on the Medical Segmentation Decathlon, Auto-nnU-Net outperforms nnU-Net's convolutional 3D U-Net baseline on eight of ten datasets and, on the official test platform, reaches a mean DSC of 77.04%, the highest among nnU-Net baselines, the HPO and HPO+HNAS ablations, and the MedSAM2 foundation model. The paper attributes this to searching over a configuration space that includes optimizer, learning rate, loss, foreground oversampling, augmentation strength, encoder type, feature counts, normalization, and activation, with training runtime as a second objective so that larger or slower models are only adopted when they improve accuracy. It also reports that training hyperparameters such as foreground oversampling and initial learning rate matter more than architectural choices, and that optimized configurations transfer partially across datasets.","pith_inferences":["Because training hyperparameters dominate architecture choices in the paper's importance analysis, a cheaper HPO-only search at matched total budget may recover most of the gain; the paper's ablations point in this direction but do not fully isolate it.","The cross-dataset transfer matrix suggests that a configuration repository or meta-learned initializer could warm-start future searches, reducing the substantial per-dataset optimization cost the paper reports; that is a natural extension the paper lists as future work.","The runtime-aware selection rule matters most in continual or federated settings, where the same model is retrained repeatedly; the paper's Pareto analysis measures single-run savings, so cumulative savings over many retrainings are an untested but plausible consequence.","Because MedSAM2 required ground-truth prompts and was excluded from the test leaderboard, the comparison against foundation models is incomplete; a fair test would need a prompt-free variant, which the paper did not have."],"forward_implications":["On six of the ten MSD datasets, Auto-nnU-Net's 5-fold cross-validation DSC exceeds every nnU-Net 3D U-Net baseline the paper compared against; on the remaining datasets it is at least on par.","The highest mean test DSC of 77.04% on the official MSD leaderboard places Auto-nnU-Net ahead of all reported baselines, including the MedSAM2 foundation model, which trails nnU-Net on nine of ten datasets.","Training runtime is an explicit optimization objective, and the Pareto-front analysis shows accuracy-runtime trade-offs on datasets like D03 and D04, with the HPO+HNAS variant cutting training time by a factor of 26 on D04.","Hyperparameter-importance analysis (fANOVA) identifies foreground oversampling, initial learning rate, and SGD momentum as consistently influential, suggesting that fixed nnU-Net defaults for these choices are unlikely to be optimal across tasks.","Optimized configurations transfer across datasets: the D03 (Liver) incumbent achieves the best DSC on four evaluation datasets, although transfer fails for some pairs such as D04 and D05."],"supporting_citations":[{"why":"Supplies the nnU-Net framework and the Conv/ResM/ResL 3D U-Net defaults that Auto-nnU-Net both extends and competes against.","marker":"Isensee et al., 2020a"},{"why":"The PriorBand multi-fidelity optimizer that Regularized PriorBand extends by adding runtime as a second objective.","marker":"Mallik et al., 2023"},{"why":"Introduces the Medical Segmentation Decathlon and its fixed training/test split used for all evaluations.","marker":"Simpson et al., 2019"},{"why":"Describes the MSD benchmark's official evaluation platform where test-set Dice scores were obtained.","marker":"Antonelli et al., 2022"},{"why":"Provides the non-dominated sorting and crowding-distance operators used to select configurations in Regularized PriorBand's Successive Halving.","marker":"Deb et al., 2002"},{"why":"Defines the successive halving budget-allocation subroutine that Regularized PriorBand modifies for two objectives.","marker":"Jamieson et al., 2016"},{"why":"Supplies the context-free grammar machinery that CFGUNet uses to define the hierarchical NAS search space.","marker":"Schrodi et al., 2023"},{"why":"Shows how to encode architectures as hyperparameters in a unified configuration space, enabling joint HPO and NAS.","marker":"Zela et al., 2018"},{"why":"Provides the functional ANOVA method used to estimate hyperparameter importance across the ten datasets.","marker":"Hutter et al., 2014"}],"fun_headline_variants":["Auto-nnU-Net: joint HPO and NAS lift Dice on 6 of 10 datasets","Automated hyperparameter and architecture search improves nnU-Net","Full AutoML for segmentation: better accuracy at comparable compute","Search over configs boosts medical image segmentation on most tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison pits Auto-nnU-Net against stripped nnU-Net 3D U-Net baselines without post-processing or ensembling, and the paper's own limitations section allows that the original nnU-Net's full pipeline might do better than these stand-ins.","fun_headline_variants_meta":{"raw":{"variants":["Auto-nnU-Net: joint HPO and NAS lift Dice on 6 of 10 datasets","Automated hyperparameter and architecture search improves nnU-Net","Full AutoML for segmentation: better accuracy at comparable compute","Search over configs boosts medical image segmentation on most tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000458,"raw_usage":{"total_tokens":2310,"prompt_tokens":970,"completion_tokens":1340,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":1262}},"tokens_in":586,"tokens_out":1340,"duration_ms":11055,"temperature":1.0,"reasoning_tokens":1262,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:58:32.506193+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the original nnU-Net pipeline, including its default ensemble selection and post-processing, on the same ten MSD datasets and the same 5-fold splits, then compare mean test DSC against the 77.04% reported for Auto-nnU-Net; if full-pipeline nnU-Net's mean matches or exceeds that value, the claim of improving on nnU-Net on 6 of 10 datasets fails on its own comparison.","supporting_citations":[],"review_version":1}