Pith. sign in

REVIEW 3 major objections 4 minor 22 references

Lung segmentation on chest x-ray images in patients with severe abnormal findings using deep learning

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A tuned U-net raises lung-segmentation overlap on severely abnormal chest x-rays from 0.889 to 0.932 Dice, the paper reports.

desk verdict New annotated severe-CXR dataset, but the central robustness claim is undermined by tuning on the test set from the same database and a tiny final sample. read the letter →

arxiv 1908.07704 v1 pith:6PJXT3P6 submitted 2019-08-21 eess.IV cs.CV

Mizuho Nishio , Koji Fujimoto , Kaori Togashi This is my paper · ORCID
classification eess.IVcs.CV
keywords lungsegmentationchestx-raydeeplearningU-netBayesianoptimizationsevereabnormalfindingsDicesimilaritycoefficienthyperparameter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether U-net, a standard deep-learning network for biomedical image segmentation, can reliably outline the lungs on chest x-rays that contain large disease shadows, where earlier deep-learning studies mostly used images with mild or moderate findings. The authors built a test database of 65 severely abnormal chest x-rays with radiologist-drawn lung boundaries, then tuned the U-net's architecture and training hyperparameters automatically with Bayesian optimization. They report that the tuned network raises the Dice overlap score on these hard images from 0.889 to 0.932, while keeping performance essentially unchanged on the mild-finding comparison set. A sympathetic reader would take the paper to establish that task-specific hyperparameter optimization, especially adding batch normalization, is what makes deep-learning lung segmentation robust to severe abnormalities.

What carries the argument

The central object is the U-net, an encoder-decoder convolutional network whose shortcut connections let the synthesis pathway use high-resolution features from the analysis pathway; the paper starts from a publicly available U-net implementation and modifies it. The mechanism that carries the argument is Bayesian optimization over eight hyperparameters, including network depth, feature-map counts, batch normalization, dropout, batch size, learning rate, and optimizer type, using Dice loss on the test data as the optimization target. What it does is turn a generic segmentation network into one whose architecture and training schedule are fitted specifically to the difficult severe-abnormality images.

What would settle it

Split the 65 severe-abnormality images into disjoint tuning and evaluation sets, run Bayesian optimization only on the tuning half, and measure Dice overlap on the held-out half; if the optimized model no longer beats the baseline by the reported margin, the robustness claim collapses. A pre-registered independent collection of severe chest x-rays with expert lung boundaries would settle the question definitively.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that an optimized U-net is more robust than the baseline U-net for lung segmentation in chest x-rays with severe abnormalities. In the authors' 65-image database, the optimized model achieved a Dice similarity coefficient (a standard overlap score between predicted and expert-drawn lung regions) of 0.932, compared with 0.889 for the baseline; on a 138-image public comparison set it rose from 0.941 to 0.973, while on a 247-image public set it was essentially unchanged (0.976 versus 0.979). The optimization searched over the number of pooling layers, initial feature maps, feature-doubling count, batch normalization, dropout probability, batch size, learning rate, and optimizer type, selecting a configuration with batch normalization and 40 initial feature maps. The paper concludes that Bayesian optimization of hyperparameters yields a robust and accurate lung segmentation system for severe abnormal findings.

Load-bearing premise

The reported robustness gain assumes that the 65 severe images used to judge the best hyperparameters are not the same images used to compute the final 0.932 Dice score.

Editorial extensions

If this is right

  • The optimized network lifts Dice overlap on severe-abnormality chest x-rays from 0.889 to 0.932, bringing the hard cases close to the level the baseline already reached on mild cases.
  • Batch normalization, selected in low-loss trials throughout the optimization, appears to be the main driver of the robustness gain and of faster convergence.
  • Task-specific hyperparameter tuning can improve an existing segmentation network without designing a new architecture from scratch.
  • The same recipe—start from a public U-net, define a small hyperparameter space, and search with Bayesian optimization on Dice loss—can be reused for other medical segmentation tasks.
  • The optimized model's performance on mild-finding images stays essentially at baseline level, so the improvement is concentrated where the baseline fails rather than being a uniform gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 0.932 Dice score is likely an upper bound, because the Bayesian optimization target was Dice loss on test data drawn from the same 65-image database used for the final evaluation; an independent set of severe images could show a smaller improvement.
  • With only 65 severe images, the 0.043 Dice gap between baseline and optimized models has no reported uncertainty, so the practical size of the robustness gain is not yet bounded.
  • The tuning recipe shown here could plausibly transfer to segmentation of other organs whose shape is distorted by pathology, but the paper does not test that transfer and it remains an open question.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper addresses lung segmentation in chest X-ray (CXR) images with severe abnormalities, for which publicly available segmentation benchmarks are scarce. The authors constructed a database of 65 CXR images with severe abnormal findings selected from the NIH CXR database, each annotated by a board-certified radiologist. Using a baseline U-net and a version modified by Bayesian optimization (search over architecture and training hyperparameters), they report Dice similarity coefficients on three databases: JSRT (247 images), Montgomery (138 images), and their own severe-abnormality database. The abstract reports baseline DSC values of 0.979, 0.941, and 0.889, and optimized DSC values of 0.976, 0.973, and 0.932, respectively. The central claim is that the optimized U-net is more robust than the baseline for lung segmentation on CXRs with large-sized abnormalities, based mainly on the improvement from 0.889 to 0.932 on the authors' database.

Significance. If the central claim were properly validated, the contribution would be meaningful: a new annotated database of severe-abnormality CXRs, a demonstration of Bayesian optimization for medical image segmentation, and a potentially more robust lung segmentation model. The study uses public data and references publicly available implementations, which supports reproducibility. The paper is also careful to report several metrics (DSC, JI, SE, SP) and to provide the full set of Bayesian-optimization trials in supplementary material. However, as explained in the major comments, the experimental design does not currently establish the central robustness claim because the evaluation on the severe-abnormality database is not independent of the hyperparameter search and is based on a very small test set.

major comments (3)
  1. [Materials and Methods, 'Model modification and hyperparameter optimization' and 'Evaluation of lung segmentation'] The optimization target of Optuna was explicitly 'the Dice loss of the test data,' and 'all the CXR images in the test data were selected from our own database.' With an 80/10/10 split of the 450 combined images, the Bayesian-optimization test set consisted of 45 images drawn from the 65-image severe-abnormality database. The subsequent quantitative evaluation of each database uses the same 80/10/10 split but does not state that the final test set for the authors' database is disjoint from those 45 optimization test images. The reported DSC of 0.932 on the authors' database is therefore likely to be a selected value from the hyperparameter search rather than an unbiased estimate of performance on severe CXRs, and the comparison with the baseline DSC of 0.889 does not provide independent evidence of robustness.
  2. [Evaluation of lung segmentation and Table 2] Even if the final test set were disjoint from the optimization test set, the per-database 80/10/10 split leaves only about six or seven test images for the authors' 65-image database. No confidence intervals, per-image results, or statistical tests are reported for any of the DSC comparisons. With such a small test set, the 0.043 DSC gap between optimized and baseline models on the severe database could easily arise from sampling variability. The paper's limitations section acknowledges that the database is small and calls for validation on a larger database, but it does not address the independence problem or provide uncertainty quantification.
  3. [Table 2 and Conclusion] The claim that the optimized U-net is 'more robust than baseline U-net in lung segmentation from CXR images with large-sized abnormalities' is not supported by the full set of results. On the JSRT database, the optimized model is slightly worse than the baseline (DSC 0.976 vs. 0.979), so the improvement is not consistent across all databases. The Montgomery improvement (0.973 vs. 0.941) is not specific to severe abnormalities, and the only severe-abnormality evidence is the compromised comparison on the authors' own database. The conclusion overstates what the experiment can establish.
minor comments (4)
  1. [Abstract / Table 2] There is a typo: 'Montogomery' should be 'Montgomery' in the abstract and in Table 2.
  2. [Table 1] The header 'Dices loss in test data' should be 'Dice loss in test data.' Also, the 'Hyperparameters' column lists entries with spaces and equals signs; formatting could be made more readable.
  3. [Materials and Methods, 'Evaluation of lung segmentation'] The text says 'the ratio of splitting that was used was same as what was used for the combined dataset,' but it does not specify the exact numbers of training, validation, and test images per database. Please report the actual counts, especially for the 65-image database.
  4. [Figure 2 and supplementary material] Figure 2 shows the change in test loss during Bayesian optimization; it would be helpful to mark the best trial and to indicate whether the test set used for optimization is the same as the final test set for the authors' database.

Circularity Check

1 steps flagged · score 6.0 of 10

Optimized-model DSC on the severe-abnormality database is the same quantity Optuna was fit to minimize, and the paper never shows the final test images are disjoint from the optimization test images.

  1. fitted input called prediction [Materials and Methods, 'Model modification and hyperparameter optimization' and 'Evaluation of lung segmentation'; Results, Table 2]
    "The optimization target of Optuna was the Dice loss of the test data that was obtained using the modified U-net model. ... all the CXR images in the test data were selected from our own database. ... After determining the optimal hyperparameters, a quantitative evaluation of lung segmentation was performed for each test dataset from the JSRT and Montogomery databases and our own database."

    Bayesian optimization selected the model by minimizing Dice loss on a test set drawn entirely from the 65-image severe-abnormality database (10% of the 450 combined images, so about 45 of those 65 images). The final evaluation then reports DSC on 'our own database' using the same 80/10/10 splitting ratio, yielding about 6-7 test images, and the paper never states that these final test images are disjoint from the 45 images used to choose the hyperparameters. If any overlap exists, the reported DSC of 0.932 for the optimized model is the optimization target itself (or a selection-biased version of it), so the claimed improvement over the baseline's 0.889 is not an independent prediction but a fitted result.

full rationale

The paper's central claim is that the optimized U-net is more robust than the baseline on CXRs with large-sized abnormalities, supported mainly by DSC 0.932 vs 0.889 on the authors' 65-image severe database. That database is exactly the source of the test images used as the Optuna optimization target: the paper states the target was 'the Dice loss of the test data' and that 'all the CXR images in the test data were selected from our own database.' The subsequent per-database evaluation uses the same 80/10/10 split and never states that the final severe-database test images are disjoint from those used during optimization. Consequently, the headline comparison is potentially a resubstitution of the fitting objective. The JSRT and Montgomery results are more independent, but JSRT shows no improvement and Montgomery does not specifically address severe abnormalities; the severe-database result is the load-bearing evidence for the robustness claim. This is partial circularity rather than full collapse, because a genuinely disjoint final split would restore independence, and the paper's external benchmarks provide some context. No load-bearing self-citation or imported uniqueness result is present, so no other circular step is identified.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the new 65-image annotation and on the Bayesian optimization procedure; all axioms are domain assumptions about data quality and preprocessing. No new theoretical entities are introduced.

free parameters (9)
  • Batch size B = 4
    Selected by Bayesian optimization to minimize test Dice loss.
  • Learning rate R = 0.001
    Selected by Bayesian optimization.
  • Optimizer OP = 2 (Nadam)
    Selected by Bayesian optimization.
  • Number of max-pooling layers N = 4
    Architecture hyperparameter selected by Bayesian optimization.
  • Number of feature maps F = 40
    Architecture hyperparameter selected by Bayesian optimization.
  • Feature doubling times T = 4
    Architecture hyperparameter selected by Bayesian optimization.
  • Dropout probability D = 0.016
    Selected by Bayesian optimization.
  • Batch normalization BN = 1 (used)
    Selected by Bayesian optimization.
  • Probability threshold for segmentation mask = 0.5
    Chosen threshold for computing DSC; standard but arbitrary.
assumptions (5)
  • domain assumption Single-radiologist manual segmentation of the 65 severe CXR images is an accurate ground truth.
    No inter-observer variability or independent verification is reported; the annotations are treated as reference standard.
  • domain assumption Resizing to 256x256 and histogram equalization preserve sufficient information for lung segmentation.
    The paper does not test alternative preprocessing; the central results depend on this choice.
  • domain assumption The baseline U-net implementation from ref [21] is a suitable baseline.
    No justification beyond citing the public repository.
  • domain assumption Dice loss is an appropriate objective for this segmentation task.
    Standard practice, but not justified in the paper.
  • domain assumption Bayesian optimization over the specified 8-dimensional search space can find a robust configuration.
    The search space is finite and manually defined; no guarantees of global optimality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lung segmentation on chest x-ray images in patients with severe abnormal findings using deep learning." pith.science (2026). https://pith.science/paper/6PJXT3P6

@misc{pith2026190807704,
  author       = {Pith},
  title        = {Pith review of: Lung segmentation on chest x-ray images in patients with severe abnormal findings using deep learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6PJXT3P6}},
  note         = {Machine review of arXiv:1908.07704}
}
read the original abstract

Rationale and objectives: Several studies have evaluated the usefulness of deep learning for lung segmentation using chest x-ray (CXR) images with small- or medium-sized abnormal findings. Here, we built a database including both CXR images with severe abnormalities and experts' lung segmentation results, and aimed to evaluate our network's efficacy in lung segmentation from these images. Materials and Methods: For lung segmentation, CXR images from the Japanese Society of Radiological Technology (JSRT, N = 247) and Montgomery databases (N = 138), were included, and 65 additional images depicting severe abnormalities from a public database were evaluated and annotated by a radiologist, thereby adding lung segmentation results to these images. Baseline U-net was used to segment the lungs in images from the three databases. Subsequently, the U-net network architecture was automatically optimized for lung segmentation from CXR images using Bayesian optimization. Dice similarity coefficient (DSC) was calculated to confirm segmentation. Results: Our results demonstrated that using baseline U-net yielded poorer lung segmentation results in our database than those in the JSRT and Montgomery databases, implying that robust segmentation of lungs may be difficult because of severe abnormalities. The DSC values with baseline U-net for the JSRT, Montgomery and our databases were 0.979, 0.941, and 0.889, respectively, and with optimized U-net, 0.976, 0.973, and 0.932, respectively. Conclusion: For robust lung segmentation, the U-net architecture was optimized via Bayesian optimization, and our results demonstrate that the optimized U-net was more robust than baseline U-net in lung segmentation from CXR images with large-sized abnormalities.

Figures

Figures reproduced from arXiv: 1908.07704 by the authors.

Figure 1
Figure 1. Illustration of the baseline model. Abbreviations: CXR, chest x-ray; conv, convolution layer; maxpool, max-pooling layer; up conv, up-sampling and convolution layer [PITH_FULL_IMAGE:figures/full_fig_p017_1.png] view at source ↗
Figure 2
Figure 2. Change in the Dice loss in the test data during Bayesian optimization [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the optimized model. Abbreviations: CXR, chest x-ray; conv, convolution layer; BN, batch normalization layer; maxpool, max-pooling layer; up conv, up-sampling and convolution layer [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Chest x-ray image and its segmentation results. (A) Chest x-ray image, (B) Ground truth of lung segmentation, (C) Segmentation result of the optimized model, and (D) Segmentation result of the baseline model [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 20 canonical work pages

  1. [1]

    Dermatologist- level classification of skin cancer with deep neural networks

    Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S. Dermatologist- level classification of skin cancer with deep neural networks. Nature. 2017 Feb 2;542(7639):115-118

    work page 2017

  2. [2]

    Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs

    Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, Venugopalan S, Widner K, Madams T, Cuadros J, Kim R, Raman R, Nelson PC, Mega JL, Webster DR. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA. 2016 Dec 13;316(22):2402-2410

    work page 2016

  3. [3]

    Nishio M, Sugiyama O, Yakami M, Ueno S, Kubo T, Kuroda T, Togashi K. Computer- aided diagnosis of lung nodule classification between benign nodule, primary lung cancer, and metastatic lung cancer at different image size using deep convolutional neural network with transfer learning. PLoS One. 2018 Jul 27;13(7):e0200721

    work page 2018

  4. [4]

    Computer-aided diagnosis of liver tumors on computed tomography images

    Chang CC, Chen HH, Chang YC, Yang MY, Lo CM, Ko WC, Lee YF, Liu KL, Chang RF. Computer-aided diagnosis of liver tumors on computed tomography images. Comput Methods Programs Biomed. 2017 Jul;145:45-51

    work page 2017

  5. [5]

    Pulmonary nodule detection in CT images with quantized convergence index filter

    Matsumoto S, Kundel HL, Gee JC, Gefter WB, Hatabu H. Pulmonary nodule detection in CT images with quantized convergence index filter. Med Image Anal. 2006 Jun;10(3):343-52

    work page 2006

  6. [6]

    Computer-aided detection of exophytic renal lesions on non-contrast CT images

    Liu J, Wang S, Linguraru MG, Yao J, Summers RM. Computer-aided detection of exophytic renal lesions on non-contrast CT images. Med Image Anal. 2015 Jan;19(1):15- 29

    work page 2015

  7. [7]

    Improved detection of lung nodules on chest radiographs using a commercial computer-aided diagnosis system

    Kakeda S, Moriya J, Sato H, Aoki T, Watanabe H, Nakata H, Oda N, Katsuragawa S, Yamamoto K, Doi K. Improved detection of lung nodules on chest radiographs using a commercial computer-aided diagnosis system. AJR Am J Roentgenol. 2004 Feb;182(2):505-10. 14

    work page 2004

  8. [8]

    Deep Learning with Lung Segmentation and Bone Shadow Exclusion Techniques for Chest X-Ray Analysis of Lung Cancer

    Gordienko Y, Gang P, Hui J, Zeng W, Kochura Y, Alienin O, Rokovyi O, Stirenko S. Deep Learning with Lung Segmentation and Bone Shadow Exclusion Techniques for Chest X-Ray Analysis of Lung Cancer. In: Hu Z., Petoukhov S., Dychka I., He M. (eds) Advances in Computer Science for Engineering and Education. ICCSEEA 2018. Advances in Intelligent Systems and Com...

    doi:10.1007/978-3- 2018

Show all 22 references
  1. [9]

    Pulmonary nodules at chest CT: effect of computer-aided diagnosis on radiologists' detection performance

    Awai K, Murao K, Ozawa A, Komi M, Hayakawa H, Hori S, Nishimura Y. Pulmonary nodules at chest CT: effect of computer-aided diagnosis on radiologists' detection performance. Radiology. 2004 Feb;230(2):347-52

    2004

  2. [10]

    Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification

    Baltruschat IM, Nickisch H, Grass M, Knopp T, Saalbach A. Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification. Scientific Reports. 2019;9:6381. doi:10.1038/s41598-019-42294-8

    2019 doi

  3. [11]

    Deep Learning Techniques for Medical Image Segmentation: Achievements and Challenges

    Hesamian MH, Jia W, He X, Kennedy P. Deep Learning Techniques for Medical Image Segmentation: Achievements and Challenges. J Digit Imaging. 2019 May 29. doi:10.1007/s10278-019-00227-x. [Epub ahead of print]

    2019 doi

  4. [12]

    A fully automated algorithm for the segmentation of lung fields on digital chest radiographic images

    Duryea J, Boone JM. A fully automated algorithm for the segmentation of lung fields on digital chest radiographic images. Med Phys. 1995 Feb;22(2):183-91

    1995

  5. [13]

    Segmentation of anatomical structures in chest radiographs using supervised methods: a comparative study on a public database

    van Ginneken B, Stegmann MB, Loog M. Segmentation of anatomical structures in chest radiographs using supervised methods: a comparative study on a public database. Med Image Anal. 2006 Feb;10(1):19-40

    2006

  6. [14]

    Lung segmentation in chest radiographs using anatomical atlases with nonrigid registration

    Candemir S, Jaeger S, Palaniappan K, Musco JP, Singh RK, Zhiyun Xue, Karargyris A, Antani S, Thoma G, McDonald CJ. Lung segmentation in chest radiographs using anatomical atlases with nonrigid registration. IEEE Trans Med Imaging. 2014 Feb;33(2):577-90. 15

    2014

  7. [15]

    U-net: convolutional networks for biomedical image segmentation

    Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer- Assisted Intervention. 2015;9351:234–241

    2015

  8. [16]

    Shiraishi J, Katsuragawa S, Ikezoe J, Matsumoto T, Kobayashi T, Komatsu K, Matsui M, Fujita H, Kodera Y, Doi K. Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists' detection o...

    2000

  9. [17]

    Scalable Bayesian Optimization Using Deep Neural Networks

    Snoek J, Rippel O, Swersky K, Kiros R, Satish N, Sundaram N, Patwary M, Prabhat M, Adams R. Scalable Bayesian Optimization Using Deep Neural Networks. In Proceedings of the 32nd International Conference on Machine Learning, in PMLR 2015;37:2171-2180

    2015

  10. [18]

    Computer-aided diagnosis of lung nodule using gradient tree boosting and Bayesian optimization

    Nishio M, Nishizawa M, Sugiyama O, Kojima R, Yakami M, Kuroda T, Togashi K. Computer-aided diagnosis of lung nodule using gradient tree boosting and Bayesian optimization. PLoS One. 2018 Apr 19;13(4):e0195875

    2018

  11. [19]

    Two public chest X-ray datasets for computer-aided screening of pulmonary diseases

    Jaeger S, Candemir S, Antani S, Wáng YX, Lu PX, Thoma G. Two public chest X-ray datasets for computer-aided screening of pulmonary diseases. Quant Imaging Med Surg. 2014 Dec;4(6):475-7

    2014

  12. [20]

    ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases

    Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. IEEE CVPR 2017

    2017

  13. [21]

    https://github.com/imlab-uiip/lung-segmentation-2d (Last visited on 2019/06/29)

    Lung fields segmentation on CXR images using convolutional neural networks. https://github.com/imlab-uiip/lung-segmentation-2d (Last visited on 2019/06/29)

    2019

  14. [22]

    Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

    Ioffe S, Szegedy C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML'15 Proceedings of the 32nd International Conference on International Conference on Machine Learning. 2015;37:448-456. 16 Acknowledgment The present study was ...

    2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.