{"id":"7694ec6b-1d1f-45d9-ab9c-56f1671c4c0b","arxiv_id":"1908.02924","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A chest X-ray segmentation model, a Feature Pyramid Network with instance normalization and Monte-Carlo dropout, estimates cardiothoracic ratio with uncertainty and is tested on four public datasets.","lead":"This paper trains a neural network to outline the heart and lungs in chest X-rays and to estimate the cardiothoracic ratio, a heart-size measure, together with uncertainty bounds. The work matters because it adds confidence information to an automated screening measurement, which could help clinicians judge when to trust the result.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equations (2) and (3) apply a softmax-style entropy/MI to a multi-label sigmoid task, and no calibration of the resulting CTR uncertainty bounds is reported; the uncertainty claim is therefore not yet substantiated.","rationale":"The reader's weakest assumption was inter-rater agreement between radiologist A and B on the external test sets. That is a real concern, but it primarily affects the quantitative generalization numbers. The more directly load-bearing issue for the paper's stated novelty is the uncertainty quantification itself: the central claim is 'for the first time, estimate CTR with uncertainty bounds.' The paper gives no evidence that those bounds are meaningful, and the entropy/MI formulas appear incorrect for a multi-label, two-sigmoid-output network. This is an internal correctness risk, not a disagreement with field consensus. The concern is concrete and checkable: recomputing the formulas for two independent Bernoulli outputs shows Equation (2) is not the predictive entropy. This does not necessarily invalidate the segmentation results or the CTR point estimates, so the verdict remains CONDITIONAL rather than REJECT: the authors should correct the uncertainty computation and provide calibration evidence. Because the reader's weakest assumption was different, I mark disagreement with the reader's identified concern, though both point to the overall need for stronger validation.","tokens_in":11168,"tokens_out":7927,"duration_ms":95290,"concrete_test":"Analytically re-derive Equations (2) and (3) under the actual multi-label setup: for two independent sigmoid outputs, compute the per-pixel joint entropy as H[p_h] + H[p_l] and the corresponding mutual information. Then test on a simple pixel with p_h = p_l = 0.9: if Equation (2) yields approximately 0.19 instead of 0.94, the published uncertainty maps are not valid predictive entropies, and the central uncertainty claim requires recomputation and calibration before it can be accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is estimating CTR with uncertainty bounds, but the uncertainty quantification has two load-bearing gaps. First, Equation (2) and Equation (3) are the standard MC-dropout formulas for a softmax over mutually exclusive classes. The network is explicitly multi-label with two independent sigmoid outputs (heart and lungs), so for a pixel that can be both heart and lung, the quantities p(y=heart|x,w_t) and p(y=lungs|x,w_t) are not a normalized distribution over c. The correct joint predictive entropy for two independent Bernoulli outputs is H[p_h] + H[p_l] = -p_h log p_h - (1-p_h) log(1-p_h) - p_l log p_l - (1-p_l) log(1-p_l). Equation (2) instead sums only over the label dimension and omits the (1-p) terms, so the displayed aleatoric uncertainty maps are not proper predictive entropies. For example, a boundary pixel with p_h = p_l = 0.9 gives about 0.19 by Equation (2) but about 0.94 for the true joint binary entropy. Second, no quantitative validation of the CTR uncertainty bounds is provided: the experiments report IoU and Pearson correlation of point estimates, but never interval coverage, calibration, or correlation between MC spread and |CTR_pred - CTR_true|. Figure 5 also selects the number of MC samples on the test sets, so the reported performance is not a clean hold-out evaluation. Even if the segmentation results are valid, the 'uncertainty bounds' part of the central claim is currently unsupported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Bayesian Feature Pyramid Network (FPN) with a ResNet-50 backbone, instance normalization in the decoder, and Monte-Carlo dropout for multi-label segmentation of heart and lung fields in chest X-rays. The segmentation outputs are used to compute the cardiothoracic ratio (CTR), and the paper claims, for the first time, to estimate CTR with uncertainty bounds derived from MC dropout. The method is trained on a newly annotated subset of ChestXray14 and evaluated on ChestXray14, JSRT, Montgomery, and Shenzhen datasets, reporting IoU/Dice for segmentation and Pearson correlation for CTR. The authors also release annotations and report an extensive ablation study over encoders, decoders, and normalization layers.","tokens_in":11511,"tokens_out":3557,"duration_ms":37963,"significance":"If the uncertainty-quantification claim holds, the paper would make a useful clinical contribution by providing confidence bounds for an automated CTR measurement, and the released annotations and multi-dataset evaluation are valuable assets to the community. The segmentation results appear solid, and the ablation study is thorough. However, the central uncertainty contribution is compromised by the use of categorical entropy formulas on multi-label sigmoid outputs and by the absence of any quantitative validation of the uncertainty bounds, so the significance of the paper in its current form is limited to the segmentation and point-estimate CTR results.","major_comments":[{"comment":"Equations (2) and (3) are the standard MC-dropout predictive entropy and mutual information formulas for a categorical softmax distribution over mutually exclusive classes. The network has two independent sigmoid outputs for heart and lungs, so at any pixel the joint predictive distribution is a product of two Bernoulli distributions. The correct predictive entropy is H[p_h] + H[p_l] = -p_h log p_h - (1-p_h) log(1-p_h) - p_l log p_l - (1-p_l) log(1-p_l). As written, Eq. (2) sums only over the label dimension and omits the (1-p) terms, systematically underestimating the aleatoric uncertainty; Eq. (3) inherits this error. The authors should either correct the formulas to the joint multi-label entropy or explicitly justify why the categorical approximation is appropriate for two independent sigmoid outputs.","section":"Bayesian Segmentation Framework: Aleotoric and Epistemic Uncertainties (Eqs. 2-3)"},{"comment":"The number of MC dropout samples T=20 is selected by inspecting IoU and Pearson correlation computed on the test sets (Fig. 5 states 'optimal number of iterations on all datasets' from test curves). Selecting hyperparameters on the test data makes the reported numbers in Table 3 a selected optimum rather than a clean hold-out evaluation. The choice of T should be made on the validation set, or the reported results should be accompanied by a sensitivity analysis that explicitly accounts for the selection.","section":"Section 4.4, Fig. 5"},{"comment":"The central claim of the paper is estimating CTR with uncertainty bounds, yet no quantitative validation of these bounds is provided. The experiments report IoU, Dice, and Pearson correlation of point estimates only; there is no assessment of interval coverage, calibration curves, or correlation between the MC spread and the absolute error |CTR_pred - CTR_true|. Without such validation, the uncertainty bounds are unsupported. Please add a calibration or coverage analysis to substantiate the uncertainty claim.","section":"Table 3 and Section 4.4"},{"comment":"The manuscript states that radiologist A annotated the training and ChestXray14 test data while radiologist B annotated JSRT, Montgomery, and Shenzhen, and that inter-rater agreement was not computed. The authors assert that this limitation has insignificant impact on the results, but provide no supporting evidence. Since cross-dataset generalization is a main contribution, the lack of inter-rater agreement assessment is load-bearing. A small double-annotated subset with IoU/Dice between the two radiologists would make the evaluation more compelling.","section":"Section 5 (Limitations)"}],"minor_comments":[{"comment":"The title and abstract contain the typo 'Cardio-Thoratic'; the correct spelling is 'Cardiothoracic' or 'Cardio-Thoracic'.","section":"Title and Abstract"},{"comment":"The loss in Eq. (5) is written as BCE - J, but J is not defined explicitly. If J is the soft Jaccard index (higher is better), the minimization should be BCE + (1 - J) or an equivalent form; please clarify the definition of J.","section":"Equation (5)"},{"comment":"The citation to Mukhoti and Gal [25] is misspelled as 'Mukohti' in the text.","section":"Related Work / References"},{"comment":"The phrase 'detection of plural effusion' should read 'pleural effusion'.","section":"Section 2"},{"comment":"The URL given for the released dataset and code is the placeholder 'http://will.be.placed.after.review.' and should be replaced with an actual repository location.","section":"Data Availability"},{"comment":"Table 3 reports IoU/Dice and Pearson correlations without any confidence intervals, although Fig. 5 shows bootstrapped 95% intervals for similar metrics; adding such intervals to Table 3 would improve the comparability and statistical transparency of the results.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The segmentation contribution is potentially sound and the dataset release is valuable, but the uncertainty-quantification claim, which is a stated central novelty, rests on incorrect entropy formulas and lacks validation. The test-set selection of T and the unquantified annotator inconsistency also weaken the headline claims. These issues are fixable within the scope of a major revision, but they require substantive re-analysis rather than mere text changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a reasonable engineering contribution: it applies MC dropout to chest X-ray segmentation, adds the first uncertainty estimates for CTR, and runs a fairly thorough ablation of encoders, decoders, and normalizations. The segmentation numbers on four test sets look plausible, the external datasets (JSRT, Montgomery, Shenzhen) are a good stress test, and the authors are admirably upfront about their limitations. The promised public release of annotations is a nice contribution, though the URL is still a placeholder.\n\nThe strongest soft spot is the uncertainty claim. The paper says it estimates CTR with uncertainty bounds, but nowhere does it validate those bounds. There is no calibration curve, no coverage check, no correlation between MC spread and actual CTR error. The displayed aleatoric/epistemic maps are illustrative only. The stress-test note argues the entropy formula in Eqs. (2)/(3) is wrong for multi-label sigmoid outputs because it omits the (1-p) terms. I'm not fully convinced: if c ranges over {organ, background} for each output, the formula is the usual binary entropy, and the authors do compute per-organ entropies and sum them. But the paper never states this explicitly, so a referee should ask. The deeper point stands: the uncertainty quantification is unvalidated.\n\nA second real issue is the test-set selection of T=20. The authors plot IoU and CTR correlation versus T across their test sets and pick 20. That is tuning on the test data, and the final Table 3 follows without error bars. The bootstrapped intervals in Fig. 5 suggest they know how to compute them; they should have applied the same to the final numbers.\n\nThe inter-rater concern is legitimate but minor: radiologist A annotated train and ChestXray14 test, radiologist B annotated the other three, and no agreement metric is given. The authors acknowledge this and assert the impact is insignificant without evidence. That is a bit hand-wavy, but not fatal. The training set is small (294 images), which limits confidence in the generalization claim, though external testing helps.\n\nOverall, this is a solid applied paper with a modest but real novelty. It deserves peer review, but the uncertainty story needs proper validation, and the test-set tuning must be addressed or at least justified. The right recommendation is: major revision, with the uncertainty evaluation as the focus.\n\nI'd bring it to a reading group as an example of applied Bayesian deep learning, and I'd send it to referees if I were the editor.","headline":"A competent, honest applied paper whose central novelty—CTR uncertainty—is not yet substantiated because the MC-dropout spread is never calibrated or validated.","tokens_in":12030,"tokens_out":3262,"would_cite":false,"duration_ms":33595,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a Bayesian Feature Pyramid Network with Monte-Carlo dropout can segment heart and lungs on chest X-rays and estimate the cardiothoracic ratio with uncertainty bounds, generalizing across four test sets from…","keywords":["chest X-ray segmentation","cardiothoracic ratio","cardiomegaly","Monte Carlo dropout","Bayesian deep learning","Feature Pyramid Network","instance normalization","uncertainty estimation"],"falsifier":"Segment a shared set of, say, 50 images with both radiologist A and radiologist B masks and compute their agreement; if their mutual IoU is no better than the model's IoU against radiologist B, the reported segmentation results mostly reflect annotation style rather than anatomical accuracy. Separately, collect many X-rays with CT-confirmed heart widths and check whether the model's 95% CTR intervals contain the true ratio at the claimed rate; coverage far below 95% would falsify the uncertainty claim.","tokens_in":10989,"feed_emoji":"🫀","tokens_out":6680,"duration_ms":58621,"temperature":0.7,"pith_summary":"This paper tries to show that one deep network can do two clinical jobs at once: outline the heart and lungs in a chest X-ray and compute the cardiothoracic ratio (CTR), the standard marker of an enlarged heart. The new piece is that the output is not a single CTR number but a distribution, so each measurement comes with an uncertainty range derived from Monte-Carlo dropout at test time. If the claim works in practice, an automated system could flag low-confidence measurements for human review and could track cardiomegaly in settings where a radiologist is not immediately available. The authors support the claim by training on a radiologist-annotated subset of ChestXray14 and testing on three independently collected public X-ray datasets re-annotated in the same style.","feed_headline":"Heart-lung ratio from X-rays now comes with uncertainty bounds","feed_subtitle":"A Bayesian FPN segments both organs and reports a cardiothoracic ratio spread, not just a single number.","key_machinery":"The load-bearing mechanism is the Feature Pyramid Network with a ResNet-50 backbone, modified in three ways: dropout layers placed before the second, third, and fourth residual blocks and inside the decoder; instance normalization replacing batch normalization in every decoder block; and spatial dropout before the final two-channel output. At test time the network is run multiple times with dropout enabled, producing Monte-Carlo samples of the heart and lung masks; the mean of these samples is the segmentation and their spread gives uncertainty. CTR is then the quotient of the widest heart chord to the widest lung chord, computed directly from the sampled masks. The variational interpretation of dropout, where training with dropout and cross-entropy minimizes the KL divergence to the posterior, is what licenses treating the sample spread as a Bayesian uncertainty estimate.","core_discovery":"On the paper's own terms, the central discovery is that inserting dropout into both a ResNet-50 encoder and a Feature Pyramid Network decoder, and replacing batch normalization with instance normalization, turns a standard segmentation network into a Bayesian one that produces multi-label heart and lung masks together with per-pixel aleatoric (data noise) and epistemic (model uncertainty) uncertainty maps. Averaging twenty stochastic forward passes gives the segmentation, and the ratio of the widest heart diameter to the widest lung diameter computed from these masks reproduces the ground-truth CTR with Pearson correlations between 0.87 and 0.97 across four test sets. The paper claims this is the first CTR-assessment method that reports uncertainty bounds rather than point estimates.","pith_inferences":["The reported IoU numbers likely mix segmentation accuracy with annotation-style agreement, because a different radiologist labeled the external test sets; computing inter-rater agreement on an overlapping subset would separate the two effects.","If the uncertainty estimates are well calibrated, the epistemic (mutual information) maps could serve as an automatic flag for images that need human review, a use the paper mentions but does not validate.","The same dropout-and-instance-normalization recipe could be applied to other ratio-based measurements in radiography, such as vertebral or cardiomediastinal ratios, where a confidence interval matters clinically.","A direct calibration test, checking that the 95% predictive interval contains the true CTR roughly 95% of the time, would turn the proposed uncertainty bounds into a clinically usable quantity; the paper does not report this."],"forward_implications":["CTR measurements for a single X-ray can be reported as a range, not a point estimate, so a clinician sees when the model is unsure.","A model trained on one annotation style with diverse radiological findings transfers to external scanners without domain adaptation, provided the test masks are annotated the same way.","Twenty Monte-Carlo samples are enough for stable segmentation and CTR correlation; more samples add little.","Instance normalization in the decoder improves heart and lung segmentation over batch and group normalization across all tested decoders.","The released multi-label annotations with true organ boundaries create a more challenging benchmark than existing tuberculosis-oriented datasets."],"supporting_citations":[{"why":"Supplies the FPN-with-ResNet encoder segmentation architecture that the paper modifies with instance normalization and dropout.","marker":"[32]"},{"why":"Establishes MC dropout as the uncertainty estimation method for segmentation, including the averaging procedure over stochastic passes.","marker":"[21]"},{"why":"Provides the CTR definition and estimation protocol (widest heart and lung diameters) that the paper adopts.","marker":"[9]"},{"why":"Justifies dropout as variational Bayesian inference via KL divergence minimization, grounding the Bayesian claim.","marker":"[12]"},{"why":"Supplies the ResNet-50 encoder used as the backbone.","marker":"[15]"},{"why":"Motivates replacing batch normalization with instance normalization in the decoder.","marker":"[38]"},{"why":"Provides the ChestXray14 source images and labels from which the training set is sampled and annotated.","marker":"[41]"},{"why":"Sources the Montgomery and Shenzhen test sets; the paper re-annotates them for evaluation.","marker":"[20]"},{"why":"Sources the JSRT test set; the paper re-annotates a subset for evaluation.","marker":"[33]"}],"fun_headline_variants":["X-ray heart-lung ratio now with uncertainty bounds","Bayesian X-ray segmentation gives heart-lung ratio with error bars","First CTR estimates with uncertainty bounds from chest X-rays","Uncertainty-aware heart-lung ratio from X-ray segmentation","Dropout turns X-ray segmentation into Bayesian CTR with error bars"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The external test sets were annotated by a different radiologist than the training data, and the paper assumes the two annotation styles are similar enough that IoU against the second radiologist's masks measures segmentation quality; this is stated but not verified by any inter-rater agreement score.","fun_headline_variants_meta":{"raw":{"variants":["X-ray heart-lung ratio now with uncertainty bounds","Bayesian X-ray segmentation gives heart-lung ratio with error bars","First CTR estimates with uncertainty bounds from chest X-rays","Uncertainty-aware heart-lung ratio from X-ray segmentation","Dropout turns X-ray segmentation into Bayesian CTR with error bars"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000681,"raw_usage":{"total_tokens":3066,"prompt_tokens":892,"completion_tokens":2174,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":2091}},"tokens_in":508,"tokens_out":2174,"duration_ms":15438,"temperature":1.0,"reasoning_tokens":2091,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:29:27.435034+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Segment a shared set of, say, 50 images with both radiologist A and radiologist B masks and compute their agreement; if their mutual IoU is no better than the model's IoU against radiologist B, the reported segmentation results mostly reflect annotation style rather than anatomical accuracy. Separately, collect many X-rays with CT-confirmed heart widths and check whether the model's 95% CTR intervals contain the true ratio at the claimed rate; coverage far below 95% would falsify the uncertainty claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the FPN-with-ResNet encoder segmentation architecture that the paper modifies with instance normalization and dropout."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CTR definition and estimation protocol (widest heart and lung diameters) that the paper adopts."},{"cited_title":"Gal and Z","cited_arxiv_id":null,"evidence_quote":"Justifies dropout as variational Bayesian inference via KL divergence minimization, grounding the Bayesian claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ResNet-50 encoder used as the backbone."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ChestXray14 source images and labels from which the training set is sampled and annotated."},{"cited_title":"Jaeger, S","cited_arxiv_id":null,"evidence_quote":"Sources the Montgomery and Shenzhen test sets; the paper re-annotates them for evaluation."},{"cited_title":"Shiraishi, S","cited_arxiv_id":null,"evidence_quote":"Sources the JSRT test set; the paper re-annotates a subset for evaluation."}],"review_version":1}