REVIEW 4 major objections 4 minor 18 references
An attempt at beating the 3D U-Net
T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Adding residual blocks to a 3D U-Net barely improved cross-validation kidney-tumor segmentation, yet the residual variant won the KiTS 2019 challenge with a composite dice of 91.23.
desk verdict A real KiTS19 win, and an honest but confounded architecture comparison—treat the null result as evidence the baseline is strong, not that residual blocks are useless. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The object that carries the argument is the U-Net itself—an encoder–decoder convolutional network with skip connections that is the standard baseline for medical image segmentation—instantiated in three variants: a plain 3D U-Net with two convolution-instance-normalization-leaky-ReLU blocks per resolution and 30 initial feature maps; a residual 3D U-Net with residual blocks in the encoder (24 initial feature maps); and a pre-activation residual 3D U-Net in which normalization and activation precede the convolution. The comparison is run under a fixed hardware constraint of a 12 GB GPU, with identical preprocessing (resampling to $3.22 \times 1.62 \times 1.62$ mm, intensity clipping to $[-79, 304]$), a fixed patch size of $80 \times 160 \times 160$, and five-fold cross-validation on a slightly modified training set. The metric is composite dice, the geometric mean of kidney and tumor dice, which is also the challenge's ranking criterion.
What would settle it
Run the same five-fold cross-validation on the same data with the activation functions, starting channel counts, and block schedules made identical across all three architectures; if the plain U-Net then reaches or surpasses the residual U-Net's composite dice, the claimed marginal benefit of residual blocks is not supported.
Extended reading notes
Core claim
The central claim is that architectural sophistication—swapping ordinary convolutional blocks for residual or pre-activation residual blocks—yields only marginal, barely measurable gains on the KiTS 2019 kidney tumor segmentation task, so the winning result is best read as evidence for the strength of the U-Net baseline and the robustness of the problem rather than for any particular modification. The residual 3D U-Net, chosen because its cross-validation composite dice (91.54) and tumor dice (85.73) were the highest, scored 91.23 composite dice on the held-out test set and ranked first among 106 teams. The same table that motivates the submission shows the plain U-Net at 91.19 composite dice and the pre-activation variant at 91.25, with an ensemble at 91.50, and the authors state they cannot declare one model better than the others. They also note the comparison was not tested for statistical significance and that a decisive test would require per-architecture hyperparameter optimization.
Load-bearing premise
The comparison assumes that the tiny score differences reflect the residual connections themselves rather than accidental differences among the models, which included a different activation function, a different number of starting channels, and a different block schedule, and the paper admits that the activation difference was an implementation error.
Editorial extensions
If this is right
- A plain 3D U-Net without residual connections is within 0.35 composite dice points of the winning variant on this task, so future U-Net-based submissions should treat it as a serious baseline rather than a starting point that needs architectural upgrades.
- Ensembling the three architectures did not beat the best single model in cross-validation (91.50 versus 91.54), which cautions against assuming ensembles always help in segmentation challenges.
- Because varying the resampling spacing and patch size within reasonable ranges gave basically identical cross-validation results, the KiTS 2019 problem is likely well-behaved and insensitive to typical hyperparameter choices.
- The top of the KiTS 2019 leaderboard is very tight: the winning margin over second place was 0.6 composite dice, meaning small differences in preprocessing, training-data curation, or random seeds can determine challenge rank.
Reading between the lines
- A decisive test of the paper's interpretation would hold all settings fixed except for the residual connection itself; the admitted activation-function mismatch means the current comparison cannot isolate the architectural effect.
- If the plain U-Net is this strong, challenge organizers and practitioners may gain more from better training-data curation, such as correcting mislabeled cases, than from novel block designs.
- The hardware-constrained comparison suggests a norm shift: architecture improvements should be evaluated under a fixed memory and time budget, and under that norm many published gains from adding residual or dense connections may shrink.
- The 0.6-point winning margin is comparable to the cross-validation spread among the paper's own models, so a challenge win at this level should be interpreted as evidence of a strong baseline, not as proof that the submitted architecture is objectively best.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports a 3D U-Net baseline and two residual variants (residual and pre-activation residual) for kidney and kidney tumor segmentation in the KiTS2019 challenge. On a five-fold cross-validation of a slightly modified training set, the three architectures achieve nearly identical composite Dice scores (91.19, 91.54, and 91.25), which the authors interpret as evidence that residual modifications yield only very minor, barely measurable improvements. The authors submit the residual U-Net for test-set prediction and report a composite Dice of 91.23, placing first on the official leaderboard among 106 teams. The paper's central interpretive claim is that architectural modifications to the 3D U-Net do not substantially improve segmentation accuracy, and it explicitly acknowledges that the comparison was not tested for statistical significance.
Significance. If the central negative result were well supported, the paper would provide a useful, externally validated data point against overvaluing residual modifications in medical segmentation, especially because the test-set win is confirmed by the official challenge leaderboard. The authors are transparent about several limitations, including the absence of significance testing and the leaky ReLU implementation error. However, the cross-validation comparison that motivates the main claim does not isolate the effect of residual connections, so the negative result is not yet established. The lasting contribution is the strong and independently verified test-set performance, but the scientific conclusion about residual blocks needs substantially stronger support.
major comments (4)
- [Section 2.2, Table 1] The comparison that supports the paper's main claim is confounded: the plain U-Net uses leaky ReLU, 30 initial feature maps, and two conv-instnorm-LReLU blocks per resolution, while both residual models use ReLU, 24 initial feature maps, and different block and decoder structures. Because the plain and residual models differ simultaneously in nonlinearity, width, block repetition, and decoder depth, the observed differences in Table 1 (Composite Dice 91.19, 91.54, 91.25) cannot be attributed to the residual connections. The leaky ReLU discrepancy is admitted in the Discussion, and the feature-map reduction is stated in Section 2.2. At minimum, the conclusion that architectural modifications yield 'very minor, barely measurable improvements' should be reworded to describe differences between the specific implemented configurations, or the authors should report a controlled comparison with matched nonlinearity and initial feature-map count.
- [Section 2.4, Table 1] The cross-validation scores are computed on a modified training set in which four cases were excluded and the reference labels of cases 15 and 37 were replaced by segmentations generated by the authors' own networks. Evaluating models against labels that were themselves produced by the same family of models introduces a form of circularity, and the effect may differ across the architectures being compared, potentially biasing the tumor Dice component of the ranking in Table 1. The authors should report the per-case contribution of cases 15 and 37 to the mean tumor Dice, repeat the comparison with these cases excluded, or otherwise demonstrate that the replaced labels do not change the conclusions.
- [Section 3, Table 1, Discussion] No statistical significance testing is reported, and the composite Dice differences between architectures (0.06 to 0.35 points) are of a magnitude that could easily arise from fold-to-fold variation. The authors' statement that they are not comfortable declaring a winner is appropriately cautious, but the stronger accompanying claims that the improvements are 'very minor' and 'barely measurable' require support from per-fold paired results, confidence intervals, or a formal significance test. Without such analysis, the central negative claim is not verifiable from Table 1.
- [Section 3, ensemble result] The statement that 'even ensembling ... did not yield an improvement' is not supported by the data as presented. Table 1 reports a single cross-validation mean for the ensemble (Composite Dice 91.50) that is numerically lower than the residual U-Net single-model score (91.54), but no measure of variability or per-fold paired comparison is given. With only one ensemble configuration and no error bars, the claim that ensembling did not help cannot be evaluated; the manuscript should either provide per-fold ensemble and single-model scores with paired comparisons or soften the claim to a statement about the specific mean values observed.
minor comments (4)
- [Section 2.2] In the paragraph describing the plain 3D U-Net, the sentence 'This number is is doubled' contains a duplicated word and should be corrected.
- [Section 2.1] The voxel spacing is printed as '3 .22 × 1.62 × 1.62 mm' with a space after the decimal point; this should be '3.22 × 1.62 × 1.62 mm'.
- [Table 1] The table label appears as 'T able 1' with a stray space; the formatting should be corrected.
- [Section 2.4] It would be helpful to state explicitly whether the 'previous iterations of our networks' used to generate replacement labels for cases 15 and 37 were of the same architectures (plain, residual, or pre-activation residual) that are compared in Table 1, because this affects how much circularity is introduced for those cases.
Circularity Check
Test-set win is externally validated; only a minor circular component in the replacement of two training labels with the authors' own network predictions.
-
self definitional
[Section 2.4, Dataset modifications]
"Case IDs 15 and 37 were confirmed to be faulty by the challenge organizers. Therefore we replaced their reference annotation with the segmentation generated by previous iterations of our networks (visual inspection seemed plausible, no manual fine tuning of the segmentations was performed)."
For cases 15 and 37, the 'reference annotation' used in all cross-validation Dice computations is the output of the authors' own segmentation networks. Evaluating a network against labels that were themselves produced by the same network family measures self-consistency rather than segmentation accuracy against independent truth. Any architecture from this family will trivially score high on its own prior output, so the Tumor Dice and Composite Dice in Table 1 for these two cases are partially forced by construction. This does not affect the external test-set leaderboard, but it subtly biases the cross-validation comparison that motivates the choice of the residual 3D U-Net for submission.
full rationale
The paper's main claim—winning KiTS2019 with Composite Dice 91.23 on the test set—is evaluated on an external test set with ground truth provided by the challenge organizers, so it is not circular. The internal comparison of the plain, residual, and pre-activation residual 3D U-Nets is based on five-fold cross-validation on a slightly modified training set. The only genuinely circular element is the replacement of reference annotations for cases 15 and 37 with segmentations generated by the authors' own networks; for those two cases, the reported Dice scores measure how well a model reproduces its family's own outputs, not agreement with an independent standard. This is disclosed by the authors and permitted by the organizers, and it affects only a small fraction of the training data. The central interpretive conclusion—that architectural modifications yield only 'very minor, barely measurable improvements'—is further hedged by the authors' admission that the results were 'not tested for statistical significance' and by the acknowledged implementation-error confound (leaky ReLU in the plain U-Net vs. ReLU in the residual models). These are correctness risks, not circularity. The self-citation of nnU-Net [7] is used as a baseline implementation and does not carry the argument; the residual-block comparison is an independent experiment. Overall, the circular component is minor and does not affect the externally validated challenge result, so a score of 2 is appropriate.
Assumptions & free parameters
free parameters (5)
- target_spacing =
3.22 x 1.62 x 1.62 mm
- patch_size =
80 x 160 x 160
- intensity_clip_range =
[-79, 304]
- intensity_normalization =
subtract 101, divide by 76.9
- initial_feature_maps =
30 for plain U-Net, 24 for residual U-Nets
assumptions (3)
- domain assumption CT intensity values are quantitative and consistent across scanners, so a fixed level window generalizes
- ad hoc to paper Cases 15 and 37 were indeed faulty and the network-generated replacement labels are acceptable ground truth
- ad hoc to paper Excluding cases 23, 68, 125, 133 does not bias the comparison
Cite this review
Pith. "Pith review of An attempt at beating the 3D U-Net." pith.science (2026). https://pith.science/paper/O4CB6ZS6
@misc{pith2026190802182,
author = {Pith},
title = {Pith review of: An attempt at beating the 3D U-Net},
year = {2026},
howpublished = {\url{https://pith.science/paper/O4CB6ZS6}},
note = {Machine review of arXiv:1908.02182}
}
read the original abstract
The U-Net is arguably the most successful segmentation architecture in the medical domain. Here we apply a 3D U-Net to the 2019 Kidney and Kidney Tumor Segmentation Challenge and attempt to improve upon it by augmenting it with residual and pre-activation residual blocks. Cross-validation results on the training cases suggest only very minor, barely measurable improvements. Due to marginally higher dice scores, the residual 3D U-Net is chosen for test set prediction. With a Composite Dice score of 91.23 on the test set, our method outperformed all 105 competing teams and won the KiTS2019 challenge by a small margin.
Figures
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:1901.04056 (2019)
Bilic, P., Christ, P.F., Vorontsov, E., Chlebus, G., Chen, H., Dou, Q., Fu, C.W., Han, X., Heng, P.A., Hesser, J., et al.: The liver tumor segmentation benchmark (lits). arXiv preprint arXiv:1901.04056 (2019)
arXiv 2019
-
[2]
In: International conference on medical image computing and computer-assisted intervention
C ¸ i¸ cek,¨O., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention. pp. 424–432. Springer (2016)
2016
-
[3]
European urology 56(5), 786–793 (2009)
Ficarra, V., Novara, G., Secco, S., Macchi, V., Porzionato, A., De Caro, R., Art- ibani, W.: Preoperative aspects and dimensions used for an anatomical (padua) classification of renal tumours in patients who are candidates for nephron-sparing surgery. European urology 56(5), 786–793 (2009)
work page 2009
-
[4]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
2016
-
[5]
In: European conference on computer vision
He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: European conference on computer vision. pp. 630–645. Springer (2016)
2016
-
[6]
arXiv preprint arXiv:1904.00445 (2019)
Heller, N., Sathianathen, N., Kalapara, A., Walczak, E., Moore, K., Kaluzniak, H., Rosenberg, J., Blake, P., Rengel, Z., Oestreich, M., et al.: The kits19 challenge data: 300 kidney tumor cases with clinical context, ct semantic segmentations, and surgical outcomes. arXiv preprint arXiv:1904.00445 (2019)
arXiv 2019
-
[7]
arXiv preprint arXiv:1904.08128 (2019) 8 F
Isensee, F., Petersen, J., Kohl, S.A., J¨ ager, P.F., Maier-Hein, K.H.: nnu-net: Breaking the spell on successful medical image segmentation. arXiv preprint arXiv:1904.08128 (2019) 8 F. Isensee and Klaus H. Maier-Hein
arXiv 2019
-
[8]
The Journal of urology 182(3), 844–853 (2009)
Kutikov, A., Uzzo, R.G.: The renal nephrometry score: a comprehensive standard- ized system for quantitating renal tumor size, location and depth. The Journal of urology 182(3), 844–853 (2009)
work page 2009
Show all 18 references
-
[9]
Landman, B., Xu, Z., Eugenio Igelsias, J., Styner, M., Langerak, T., Klein, A.: Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge (2015)
2015
-
[10]
IEEE TMI 37(12), 2663–2674 (2018)
Li, X., Chen, H., Qi, X., Dou, Q., Fu, C.W., Heng, P.A.: H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from ct volumes. IEEE TMI 37(12), 2663–2674 (2018)
2018
-
[11]
In: International Conference on 3D Vision (3DV)
Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: International Conference on 3D Vision (3DV). pp. 565–571. IEEE (2016)
2016
-
[12]
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch (2017)
2017
-
[13]
In: MICCAI
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: MICCAI. pp. 234–241. Springer (2015)
2015
-
[14]
arXiv preprint arXiv:1902.09063 (2019)
Simpson, A.L., Antonelli, M., Bakas, S., Bilello, M., Farahani, K., van Ginneken, B., Kopp-Schneider, A., Landman, B.A., Litjens, G., Menze, B., et al.: A large an- notated medical image dataset for the development and evaluation of segmentation algorithms. arXiv preprint arXi...
2019 arXiv
-
[15]
In: 2016 IEEE International Conference on Imaging Systems and Techniques (IST)
Skalski, A., Jakubowski, J., Drewniak, T.: Kidney tumor segmentation and detec- tion on computed tomography data. In: 2016 IEEE International Conference on Imaging Systems and Techniques (IST). pp. 238–242. IEEE (2016)
2016
-
[16]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Taha, A., Lo, P., Li, J., Zhao, T.: Kid-net: convolution networks for kidney ves- sels segmentation from ct-volumes. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 463–471. Springer (2018)
2018
-
[17]
In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society
Yang, G., Gu, J., Chen, Y., Liu, W., Tang, L., Shu, H., Toumoulin, C.: Automatic kidney segmentation in ct images based on multi-atlas image registration. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. pp. 5538–5541. IEEE (2014)
2014
-
[18]
arXiv preprint arXiv:1804.10484 (2018)
Yu, Q., Shi, Y., Sun, J., Gao, Y., Dai, Y., Zhu, J.: Crossbar-net: A novel con- volutional network for kidney tumor segmentation in ct images. arXiv preprint arXiv:1804.10484 (2018)
2018 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.