REVIEW 4 major objections 5 minor 31 references
Kidney tumor segmentation using an ensembling multi-stage deep learning approach. A contribution to the KiTS19 challenge
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A three-stage deep-learning ensemble segments CT kidneys at 0.96 Dice and tumors at 0.74 on 90 unseen cases.
desk verdict A clean KiTS19 challenge write-up: externally measured Dice (0.96 kidney, 0.74 tumor) with a clear ablation, but the class-agnostic 5000-pixel post-processing filter could be silently removing small tumors and deserves sensitivity analysis. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the multi-stage 2.5D cascade. Stage 1 is a Residual UNet with four encoding levels and pre-activated residual blocks; it receives five adjacent axial slices stacked as channels, subsampled to $256\times 256$, and predicts a single mask that merges kidney and tumor into one meta-class. That mask defines bounding boxes for each kidney. Stage 2 crops each box to $256\times 256$ at full resolution and runs two networks—the same Res-UNet and a Res-Net adapted from the authors' cited organ-at-risk segmentation work—trained with weighted categorical cross-entropy to separate kidney, tumor, and background. Stage 3 combines the two predictions, and a post-processing step keeps only connected components larger than 5000 pixels. The cascade concentrates the expensive full-resolution computation on relevant regions, while the ensembling reduces single-model variance, which the validation table supports.
What would settle it
Run the two stage-2 networks without stage-1 cropping, using full-resolution sliding windows over each CT volume, and compare tumor Dice on the same 90 test cases; if the uncropped version beats 0.74, the cascade is cutting off tumor tissue, whereas a similar score would place the deficit inside the high-resolution segmentation itself.
Extended reading notes
Core claim
On its own terms, the paper's central discovery is that a three-stage cascade built on Residual UNets—coarse kidney/tumor detection at half resolution, then two full-resolution networks working inside bounding boxes from the first stage, then an ensembling step—produces near-complete kidney segmentation and markedly weaker but useful tumor segmentation. The paper reports mean Dice scores of 0.96 for kidneys and 0.74 for tumors on the 90-case test set, and on its 20-case validation set the ensemble reaches 0.98 kidney Dice and 0.73 tumor Dice, improving over each individual stage-2 network. It also identifies the main failure mode: benign renal cysts are often labeled as tumors, which is consistent with the tumor score lagging the kidney score.
Load-bearing premise
The load-bearing premise is that the first-stage detector always finds and fully encloses every tumor region; if it misses or clips a tumor, the later stages never see that tissue, and the paper does not measure how often that happens.
Editorial extensions
If this is right
- If the reported Dice scores hold, surgeons planning partial nephrectomy would get kidney contours that are nearly complete automatically, while tumor contours at 0.74 Dice would still require manual correction.
- Because the ensemble outperforms each stage-2 network on the validation set, the accuracy gain is tied to combining models rather than to a single architecture.
- The 2.5D input format makes volumetric context available at 2D memory cost, which is what allows the large batch size and residual networks used here.
- The paper's own conclusion is that adding cyst-aware training data is the direct route to raising tumor Dice, since cysts are the dominant source of false positives.
Reading between the lines
- The stage-1 bounding box is an unrecoverable bottleneck: any tumor tissue the first stage fails to enclose is invisible to both stage-2 networks, so the reported 0.74 tumor Dice may partly reflect cascade truncation rather than the limits of high-resolution segmentation.
- The preprocessing choices (3 mm reslicing and the -30 to 300 HU window) are not ablated, so their contribution to the tumor deficit is untested; a reader should not attribute all of the 0.26 Dice gap to cysts alone.
- A direct experiment would re-run the pipeline with a third class for benign cysts; if tumor Dice rises substantially, the paper's cyst diagnosis is confirmed, and if not, the bottleneck is likely elsewhere, such as small or low-contrast lesions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a multi-stage 2.5D convolutional neural network pipeline for segmenting kidneys and kidney tumors in contrast-enhanced CT, developed for the KiTS19 challenge. A stage-1 Residual-UNet localizes kidney/tumor regions, two stage-2 networks (a Res-UNet and a Res-Net) segment the extracted ROIs, and a final ensembling stage combines their predictions. Validation on 20 cases shows tumor Dice improving from 0.52 (stage-1 only) to 0.73 (ensemble), and on the 90 unseen test cases the authors report mean Dice scores of 0.96 for kidneys and 0.74 for tumors. The discussion attributes the modest tumor performance to false positives on renal cysts and false negatives on lesions that are difficult to identify on CT.
Significance. If the reported test-set performance holds, this is a solid and reproducible challenge contribution: the results are measured on an external benchmark that was not used for training or validation, the stage-by-stage ablation in Table 2 supports the value of the multi-stage design, validation scores are reported with standard deviations, and the training configurations are disclosed in reasonable detail. The main limitations are the absence of uncertainty quantification on the test-set Dice, an underspecified ensembling rule, and an unexamined post-processing step that could materially affect the interpretation of the tumor segmentation results.
major comments (4)
- [Section 2.4 (Post-Processing)] The connected-component filter that removes all objects with fewer than 5000 pixels is class-agnostic and could delete genuine small tumors, yet the paper does not state whether the filter is applied in 2D or 3D, does not provide a sensitivity analysis of the threshold, and does not report the ground-truth tumor size distribution or Dice stratified by tumor size. Since the method predicts per-slice 2.5D masks, a 2D filter could split a moderately sized 3D tumor into per-slice components and discard each one; the ambiguity between 'pixels' and 'voxels' is therefore load-bearing. This issue should be resolved by clarifying the exact filtering operation, reporting threshold sensitivity, and providing size-stratified tumor Dice to substantiate the attribution of errors to cysts and difficult lesions.
- [Section 2.3 (Stage 2) and Figure 1] The stage-2 networks only see ROIs derived from stage-1 kidney/tumor predictions, so any tumor tissue missed by stage-1 or truncated by the symmetric 256x256 bounding-box expansion cannot be recovered by the downstream stages. The paper does not quantify how often this cascade error occurs or how much it contributes to the tumor Dice deficit. An analysis of stage-1 tumor-region recall and the impact of ROI truncation on final predictions would make the central claim more robust.
- [Section 3.2 (Evaluation results)] The test-set Dice scores (0.96 and 0.74) are reported as point estimates with no confidence intervals, despite validation tumor Dice standard deviations above 0.25. The ensembling improvement over the best single network on validation (0.73 vs. 0.72) is also not statistically tested. Bootstrapped confidence intervals for the 90 test cases and a paired statistical test for the ensemble versus individual models would support the stated conclusions.
- [Section 2.3 (Stage 3)] The ensembling operation is described only as 'combines prediction masks' without specifying whether predictions are averaged as probabilities, combined by majority vote, or weighted in some other way, or whether all stage-2 networks contribute equally. This underspecification hinders reproducibility of the central claim and should be corrected with an exact description of the ensembling rule.
minor comments (5)
- [Abstract] The word 'ensambling' should be 'ensembling'.
- [Introduction] 'outmost importance' should read 'utmost importance'.
- [Discussion] 'east to identify' should be 'easy to identify', and the sentence 'Furthermore, the we would like' should be 'Furthermore, we would like'.
- [Section 2.2 (Preprocessing)] The choice of the HU window [-30, 300] is justified qualitatively; a brief sensitivity analysis or a reference to its use in the KiTS19 challenge would strengthen the presentation.
- [Table 1] Please clarify that the data augmentation operations apply only to group KT images, and specify the central-crop and zoom parameters used in that augmentation.
Circularity Check
No significant circularity: the central claim is an empirical benchmark score measured against an external, unseen test set.
full rationale
The paper's central claim is the reported mean Dice score of 0.96 for kidneys and 0.74 for kidney tumors on 90 unseen KiTS19 test cases. This is an empirical evaluation against ground-truth masks that were not available to the authors during development, not a derivation from assumptions that already contain the result. The preprocessing, network architectures, training configuration, and post-processing (including the 5000-pixel connected-component filter) are method choices; they may affect the reported scores, but they are not inputs from which the Dice values are algebraically forced. No parameter is fitted to the test-set Dice and then renamed as a prediction. The paper contains no load-bearing self-citation: references to challenge data, network architectures, and ensemble methods are external and do not smuggle in the target conclusion. The only notable weaknesses are methodological (e.g., a class-agnostic size filter that could remove true small tumors, and cascade error from stage-1 bounding boxes), but these are correctness or generalization concerns, not circularity. Accordingly, the circularity score is 0.
Assumptions & free parameters
free parameters (7)
- HU window lower and upper bounds =
-30 to 300 HU
- Resampled slice thickness =
3 mm
- Stage-1 class loss weights =
[0.3, 1.0, 3.0] for background, kidney, tumor
- Stage-2 Res-Net class loss weights =
[0.2, 0.25, 0.55]
- Post-processing minimum object size =
5000 pixels
- Data augmentation probabilities =
rotation p=1.0, horizontal flip p=0.5, central crop p=0.66
- Stage-1 input size and stage-2 ROI size =
256x256 pixels
assumptions (4)
- domain assumption KiTS19 ground truth labels are a valid reference standard for evaluating segmentation.
- domain assumption The 90-case test set is representative of the same distribution as the 210 training and validation cases.
- domain assumption Dice score on kidney and tumor is the appropriate metric for the clinical use case.
- ad hoc to paper Connected-component filtering with a 5000-pixel threshold does not remove genuine kidney or tumor tissue.
Cite this review
Pith. "Pith review of Kidney tumor segmentation using an ensembling multi-stage deep learning approach. A contribution to the KiTS19 challenge." pith.science (2026). https://pith.science/paper/EC2HTXA6
@misc{pith2026190900735,
author = {Pith},
title = {Pith review of: Kidney tumor segmentation using an ensembling multi-stage deep learning approach. A contribution to the KiTS19 challenge},
year = {2026},
howpublished = {\url{https://pith.science/paper/EC2HTXA6}},
note = {Machine review of arXiv:1909.00735}
}
read the original abstract
Precise characterization of the kidney and kidney tumor characteristics is of outmost importance in the context of kidney cancer treatment, especially for nephron sparing surgery which requires a precise localization of the tissues to be removed. The need for accurate and automatic delineation tools is at the origin of the KiTS19 challenge. It aims at accelerating the research and development in this field to aid prognosis and treatment planning by providing a characterized dataset of 300 CT scans to be segmented. To address the challenge, we proposed an automatic, multi-stage, 2.5D deep learning-based segmentation approach based on Residual UNet framework. An ensembling operation is added at the end to combine prediction results from previous stages reducing the variance between single models. Our neural network segmentation algorithm reaches a mean Dice score of 0.96 and 0.74 for kidney and kidney tumors, respectively on 90 unseen test cases. The results obtained are promising and could be improved by incorporating prior knowledge about the benign cysts that regularly lower the tumor segmentation results.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
CA: a cancer journal for clinicians 68(6) (2018) 394–424
Bray, F., Ferlay, J., Soerjomataram, I., Siegel, R.L., Torre, L.A., Jemal, A.: Global cancer statistics 2018: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a cancer journal for clinicians 68(6) (2018) 394–424
work page 2018
-
[2]
Journal of Clinical Oncology 36(36) (2018) 3574–3581 10 G.Santini et al
Scelo, G., Larose, T.L.: Epidemiology and risk factors for kidney cancer. Journal of Clinical Oncology 36(36) (2018) 3574–3581 10 G.Santini et al
work page 2018
-
[3]
The Journal of urology 176(6) (2006) 2397–2400
Nguyen, M.M., Gill, I.S., Ellison, L.M.: The evolving presentation of renal car- cinoma in the united states: trends from the surveillance, epidemiology, and end results program. The Journal of urology 176(6) (2006) 2397–2400
work page 2006
-
[4]
Annals of surgical oncology 19(7) (2012) 2380–2387
Sun, M., Abdollah, F., Bianchi, M., Trinh, Q.D., Jeldres, C., Thuret, R., Tian, Z., Shariat, S.F., Montorsi, F., Perrotte, P., et al.: Treatment management of small renal masses in the 21st century: a paradigm shift. Annals of surgical oncology 19(7) (2012) 2380–2387
work page 2012
-
[5]
Journal of minimal access surgery 7(4) (2011) 205
Dominguez-Escrig, J.L., Vasdev, N., O’Riordon, A., Soomro, N.: Laparoscopic partial nephrectomy: Technical considerations and an update. Journal of minimal access surgery 7(4) (2011) 205
work page 2011
-
[6]
European urology 56(5) (2009) 786–793
Ficarra, V., Novara, G., Secco, S., Macchi, V., Porzionato, A., De Caro, R., Art- ibani, W.: Preoperative aspects and dimensions used for an anatomical (padua) classification of renal tumours in patients who are candidates for nephron-sparing surgery. European urology 56(5) (2009) 786–793
work page 2009
-
[7]
The Journal of urology 182(3) (2009) 844–853
Kutikov, A., Uzzo, R.G.: The renal nephrometry score: a comprehensive standard- ized system for quantitating renal tumor size, location and depth. The Journal of urology 182(3) (2009) 844–853
work page 2009
-
[8]
The Journal of urology 183(5) (2010) 1708–1713
Simmons, M.N., Ching, C.B., Samplaski, M.K., Park, C.H., Gill, I.S.: Kidney tumor location measurement using the c index method. The Journal of urology 183(5) (2010) 1708–1713
work page 2010
Show all 31 references
-
[9]
Computer methods and programs in biomedicine 157 (2018) 49–67
Torres, H.R., Queiros, S., Morais, P., Oliveira, B., Fonseca, J.C., Vilaca, J.L.: Kid- ney segmentation in ultrasound, magnetic resonance and computed tomography images: A systematic review. Computer methods and programs in biomedicine 157 (2018) 49–67
2018
-
[10]
Medical image analysis 42 (2017) 60–88
Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., Van Der Laak, J.A., Van Ginneken, B., S´ anchez, C.I.: A survey on deep learning in medical image analysis. Medical image analysis 42 (2017) 60–88
2017
-
[11]
Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization 6(3) (2018) 277–282
Thong, W., Kadoury, S., Pich´ e, N., Pal, C.J.: Convolutional networks for kidney segmentation in contrast-enhanced ct scans. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization 6(3) (2018) 277–282
2018
-
[12]
In: Deep Learning and Convolutional Neural Networks for Medical Image Computing
Zheng, Y., Liu, D., Georgescu, B., Xu, D., Comaniciu, D.: Deep learning based automatic segmentation of pathological kidney in ct: local versus global image context. In: Deep Learning and Convolutional Neural Networks for Medical Image Computing. Springer (2017) 241–255
2017
-
[13]
Frontiers in oncology 8 (2018) 215
Jackson, P., Hardcastle, N., Dawe, N., Kron, T., Hofman, M., Hicks, R.J.: Deep learning renal segmentation for fully automated radiation dose estimation in un- sealed source therapy. Frontiers in oncology 8 (2018) 215
2018
-
[14]
Scientific reports 7(1) (2017) 2049
Sharma, K., Rupprecht, C., Caroli, A., Aparicio, M.C., Remuzzi, A., Baust, M., Navab, N.: Automatic segmentation of kidneys using deep learning for total kidney volume quantification in autosomal dominant polycystic kidney disease. Scientific reports 7(1) (2017) 2049
2017
-
[15]
Radiology 158(1) (1986) 1–10
Bosniak, M.A.: The current radiological approach to renal cysts. Radiology 158(1) (1986) 1–10
1986
-
[16]
Acta radiologica 45(7) (2004) 791–795
Kim, D.Y., Park, J.W.: Computer-aided detection of kidney tumor on abdominal computed tomography scans. Acta radiologica 45(7) (2004) 791–795
2004
-
[17]
Zhou, B., Chen, L.: Atlas-based semi-automatic kidney tumor detection and seg- mentation in ct images. In: 2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), IEEE (2016) 1397–1401 Deep Learning for kidney tumor s...
2016
-
[18]
arXiv preprint arXiv:1904.00445 (2019)
Heller, N., Sathianathen, N., Kalapara, A., Walczak, E., Moore, K., Kaluzniak, H., Rosenberg, J., Blake, P., Rengel, Z., Oestreich, M., et al.: The kits19 challenge data: 300 kidney tumor cases with clinical context, ct semantic segmentations, and surgical outcomes. arXiv prep...
2019 arXiv
-
[19]
In: International Conference on Medical image computing and computer-assisted intervention, Springer (2015) 234–241
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedi- cal image segmentation. In: International Conference on Medical image computing and computer-assisted intervention, Springer (2015) 234–241
2015
-
[20]
In: European conference on computer vision, Springer (2016) 630–645
He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: European conference on computer vision, Springer (2016) 630–645
2016
-
[21]
In: Interna- tional conference on medical image computing and computer-assisted intervention, Springer (2014) 520–527
Roth, H.R., Lu, L., Seff, A., Cherry, K.M., Hoffman, J., Wang, S., Liu, J., Turk- bey, E., Summers, R.M.: A new 2.5 d representation for lymph node detection using random sets of deep convolutional neural network observations. In: Interna- tional conference on medical image comp...
2014
-
[22]
Medical image analysis 34 (2016) 123–136
Wolterink, J.M., Leiner, T., de Vos, B.D., van Hamersvelt, R.W., Viergever, M.A., Iˇ sgum, I.: Automatic coronary artery calcium scoring in cardiac ct angiography using paired convolutional neural networks. Medical image analysis 34 (2016) 123–136
2016
-
[23]
IEEE transactions on medical imaging 35(5) (2016) 1160–1169
Setio, A.A.A., Ciompi, F., Litjens, G., Gerke, P., Jacobs, C., Van Riel, S.J., Wille, M.M.W., Naqibullah, M., S´ anchez, C.I., van Ginneken, B.: Pulmonary nodule detection in ct images: false positive reduction using multi-view convolutional net- works. IEEE transactions on me...
2016
-
[24]
In: Medical Imaging 2016: Computer-Aided Diagnosis
Roth, H.R., Wang, Y., Yao, J., Lu, L., Burns, J.E., Summers, R.M.: Deep convolu- tional networks for automated detection of posterior-element fractures on spine ct. In: Medical Imaging 2016: Computer-Aided Diagnosis. Volume 9785., International Society for Optics and Photonics...
2016
-
[25]
In: European conference on computer vision, Springer (2016) 694–711
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision, Springer (2016) 694–711
2016
-
[26]
In: SegTHOR@ ISBI
van Harten, L., Noothout, J.M., Verhoeff, J., Wolterink, J.M., Isgum, I.: Auto- matic segmentation of organs at risk in thoracic ct scans by combining 2d and 3d convolutional neural networks. In: SegTHOR@ ISBI. (2019)
2019
-
[27]
In: Scandinavian Conference on Image Analysis, Springer (2015) 201–211
Lyksborg, M., Puonti, O., Agn, M., Larsen, R.: An ensemble of 2d convolutional neural networks for tumor segmentation. In: Scandinavian Conference on Image Analysis, Springer (2015) 201–211
2015
-
[28]
In: International MICCAI Brainlesion Workshop, Springer (2017) 450–462
Kamnitsas, K., Bai, W., Ferrante, E., McDonagh, S., Sinclair, M., Pawlowski, N., Rajchl, M., Lee, M., Kainz, B., Rueckert, D., et al.: Ensembles of multiple models and architectures for robust brain tumour segmentation. In: International MICCAI Brainlesion Workshop, Springer (...
2017
-
[29]
In: 12th{USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16)
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghe- mawat, S., Irving, G., Isard, M., et al.: Tensorflow: A system for large-scale ma- chine learning. In: 12th{USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16). (2016) 265–283
2016
-
[30]
In: Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops
J´ egou, S., Drozdzal, M., Vazquez, D., Romero, A., Bengio, Y.: The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In: Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. (2017) 11–19
2017
-
[31]
IEEE transactions on pattern analysis and machine intelligence 40(4) (2017) 834–848
Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence 40(4) (2017) 834–848
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.