{"id":"daed5307-7b32-4f1f-bdee-84a15afff7a2","arxiv_id":"2507.08952","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An explainable XGBoost model using CT-derived heart and lung measurements detects radiological signs of acute heart failure with AUROC 0.87 on an internal test set.","lead":"This study trained a tree-based AI model to spot signs of acute heart failure on emergency chest CT scans, using measurements of heart and lung structures. The model reached an AUC of 0.87 on an internal test set, and most of its mistakes were traced to errors in the original radiology reports.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on report-derived labels; the paper's own error review shows 38% of model errors are label errors, so the AUROC of 0.87 may not reflect true diagnostic accuracy on CT signs.","rationale":"The reader's weakest-assumption analysis correctly identifies the validity of report-derived labels as the most load-bearing assumption. The paper's own error review provides direct evidence that this assumption is violated in a non-negligible fraction of cases (38% of errors). Because the central claim explicitly states an AUROC of 0.87 with comparability to radiologists, and both components depend on the quality of the reference standard, this is the right point to stress. The proposed concrete test—expert re-reading of a random sample—would settle whether label noise materially changes the measured performance. The reader's verdict of CONDITIONAL is appropriate; my analysis does not change that verdict, so verdict_should_be is UNCHANGED. The paper otherwise has reasonable internal methodology: subject-level splitting, separate tuning and test sets, and transparent feature selection. The lack of external validation and no direct radiologist comparison are additional limitations, but they are secondary to the label validity concern.","tokens_in":9067,"tokens_out":4018,"duration_ms":50516,"concrete_test":"Select a random stratified sample of approximately 300 test CT studies, enriched for near-threshold model predictions, and have two to three expert thoracic radiologists independently score predefined radiological signs of AHF using a standardized protocol while blinded to the original report and model output. Compute the AUROC of the model against the expert consensus labels and compare it to 0.87. If the 95% confidence interval excludes 0.87 or the AUROC drops below 0.80, the report-derived labels are not a reliable reference and the claim should be revised. On the same sample, compute the radiologists' AUROC against the same expert consensus to directly test the 'comparable to radiologists' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The model's performance is measured against ground truth extracted from radiology reports (Section 2.3, 3.2). Reports without any AHF-related keyword are assumed negative, and negative findings are captured by regular expressions (Table S1). The authors acknowledge underreporting and label variability (Section 4.3), and their error review (Section 3.5) found that 24 of 64 false positives and 24 of 61 false negatives were actually correct predictions, i.e., 38% of errors were label errors. This indicates that the reference standard itself is substantially noisy. If the label noise is nondifferential, the true AUROC may be higher than 0.87; but if it is systematic—e.g., certain patient subgroups or report styles are more likely to omit AHF signs—the model may be learning associations with report content rather than independent imaging features. The error review only examined cases where the model disagreed with the report, so it cannot quantify overall label accuracy or correct the AUROC. This label uncertainty also undermines the separate claim of comparability to thoracic radiologists, since no direct radiologist benchmark is provided. The central claim therefore rests on an unvalidated reference standard.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops an explainable machine-learning model (XGBoost with TotalSegmentator-derived cardiac and pulmonary measurements) to detect radiological signs of acute heart failure on acute chest CT scans. Using report-derived labels from a single-center retrospective cohort (4,672 subjects), the final twelve-feature model achieves an internal test AUROC of 0.87. The authors also report an expert radiologist error review of 126 misclassifications, finding that roughly 38% of errors were label errors. The central claim is that the model's performance is 'comparable to thoracic radiologists' and that its stepwise, SHAP-based explanations make it useful for triage before the formal radiology report is available.","tokens_in":9251,"tokens_out":4175,"duration_ms":50445,"significance":"If the result holds, the work has practical value: it demonstrates that fully automated extraction of a small set of physiologically motivated CT measurements can flag AHF with reasonably good discrimination in an undifferentiated emergency population, and the use of off-the-shelf segmentation plus tree ensembles lowers the barrier to clinical deployment. The authors are transparent about many limitations, including single-center data, report-derived labels, and the absence of external validation. The strongest assets are the subject-level split, threshold fixed on training data, manual review of positive/negative keyword labels, and the inclusion of an error-review effort. However, the headline claim of comparability to thoracic radiologists is not directly tested, and the reference standard is noisy; these issues are load-bearing for the stated conclusions.","major_comments":[{"comment":"The ground-truth labels are derived from radiology reports via regular expressions, and reports without any AHF-related keyword are assumed negative. The authors themselves find in the error review that 24/64 false positives and 24/61 false negatives were actually correct model predictions, i.e., 38% of errors are label errors. This makes the reported AUROC of 0.87 an estimate against a noisy reference standard. If the label noise is nondifferential, the true AUROC may be higher, but if it is differential (e.g., certain report styles or patient subgroups underreport AHF signs), the model may be learning associations with report content rather than imaging signs. Because the error review only inspects cases where the model disagrees with the report, it cannot correct the AUROC or quantify overall label accuracy. The authors should re-adjudicate a random sample (ideally all) of the test set with a radiologist blinded to the original report and model prediction, and report the model's AUROC, calibration, and sensitivity/specificity against that adjudicated reference. This would also allow a sensitivity analysis comparing performance on adjudicated versus report-derived labels.","section":"3.2, 3.5, 4.3"},{"comment":"The claim that the model is 'comparable to thoracic radiologists' is not supported by any direct comparison. No radiologist AUC, sensitivity, specificity, or inter-observer variability on the same cohort is provided. The only radiologist involvement is the error review of misclassified cases, which is not a reader study and cannot establish equivalence. To support the central claim, the authors should either (a) add a reader study in which multiple radiologists independently score a random subset (or all) of the test CTs for AHF likelihood, then compute and statistically compare their ROC curves with the model's, or (b) substantially temper the wording to 'performance in a range consistent with reported radiologist accuracy' and provide explicit literature benchmarks. As written, the abstract and conclusion overstate the evidence.","section":"1, 4.1, Abstract"},{"comment":"The radiologist error review appears to be unblinded: the text does not state whether the reviewing radiologist was blinded to the model prediction and to the original report label. Without blinding, the estimate that 38% of errors are label errors is vulnerable to confirmation bias, which would directly undermine the authors' interpretation that the model's true accuracy is better than the report-based metrics suggest. The authors should state whether blinding was used and, if not, re-run the review with a blinded protocol or acknowledge this as a serious limitation of the error-analysis result.","section":"3.5"},{"comment":"The data-partition description is ambiguous regarding multiple studies per subject. The text says 'For test subjects with multiple studies at different time points, only the latest study was included,' but it does not say whether the same restriction applies to training subjects. Since the total dataset includes 5,111 studies for 4,672 subjects, some subjects contribute more than one study. If training includes all studies per subject while test includes only the latest, the training and test distributions may differ (e.g., in disease severity or scan context), and the threshold calibrated on training data may be mismatched. Please report the number of studies per subject in train and test, and justify the asymmetry or apply the same policy to both partitions.","section":"2.5, 3.1"}],"minor_comments":[{"comment":"The text reports 62 false negatives in the test set, but the error-review paragraph says '24 of 61 false negatives' and Table 3 shows 61 false negatives. Clarify that one case was not reviewed due to a PACS issue, and reconcile the counts.","section":"3.5, Table 2"},{"comment":"Section numbering jumps from 2.3 to 2.5; a subsection 2.4 appears to be missing or the numbering is misaligned. Please renumber.","section":"2.3-2.5"},{"comment":"The caption says 'blue errors' where it should say 'blue arrows' when describing features contributing to a negative prediction.","section":"Figure 5"},{"comment":"The citation sequence '(8–10)(9)' appears to be a formatting error; it should likely be '(8–10)' with appropriate citation placement.","section":"4.2"},{"comment":"The term 'density' is used for measurements that are later described in Hounsfield units (HU). Define the unit at first use and use HU consistently throughout.","section":"2.7"},{"comment":"The 'No decompensation' regular expression is split across two lines in the table, making it difficult to read as a single pattern; format it as one code block.","section":"Supplementary Table S1"},{"comment":"The trained model is not made publicly available, which limits reproducibility of the exact reported results. Please state explicitly that model weights are available upon reasonable request, or provide a minimal reproducible example with features and code (even if the data cannot be shared).","section":"6"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely within scope for a radiology/imaging informatics journal. The main concern is that the headline claim of radiologist-level performance is not supported by the experimental design; this is fixable by adding a reader study or softening the claim. The label-noise issue is acknowledged but needs a more rigorous treatment (blinded re-adjudication of a random sample) before the AUROC can be interpreted as a diagnostic accuracy estimate. I would not recommend rejection, because the underlying methodology is sound and the limitations are mostly addressed in the discussion, but the manuscript needs substantial additional analysis or restructured claims before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick read on arXiv:2507.08952. The genuinely new bit is the application: training an AI model to flag radiological signs of acute heart failure on acute chest CT in an undifferentiated emergency population. That has not been done before, and the 0.87 test AUROC with twelve interpretable features is a reasonable first baseline. The paper is well-structured, the methods are reproducible in principle, and the authors are refreshingly honest about their label problem. Their error review found that 38% of the model's apparent errors were actually errors in the radiology reports. That is a real strength—they report it directly instead of burying it.\n\nThe central claim, however, is not supported. The abstract and conclusion say the model is 'comparable to thoracic radiologists,' but there is no radiologist AUC on the same test set. The error review gives a five-point likelihood score, but that is not a reader study. So that claim should be removed or substantially softened.\n\nThe bigger soft spot is the reference standard. Labels come from text-mined radiology reports; reports without AHF-related keywords are assumed negative. The positive labels were manually corrected, but the negatives are mostly assumed. If a radiologist omitted a sign, the scan is labeled negative even when the sign is present. The stress-test note worries the model might learn report content rather than imaging features—that's not quite right, since the model never sees text, but the model may internalize the bias in the labels, such as only obvious cases being reported. Either way, the true AUROC on imaging signs is unknown, and the 38% mislabel rate means the reported number is a moving target.\n\nOther gaps: no external validation, no confidence interval on the AUROC, and neither data nor trained model are released. For a clinical AI paper, that is increasingly the expected standard. The observed sex differences in error rates are mentioned but not explored.\n\nBottom line: this is a modest but useful contribution. It deserves a serious referee—the methodology is sound and the error analysis is genuinely informative—but the central claim needs rework and external validation is essential before it is ready for clinical use. I'd bring it to a reading group as a case study in label noise and transparent reporting, but I wouldn't cite it in my own work yet.","headline":"A solid, honest internal validation of an explainable CT-based AHF detector, but the radiologist-comparability claim is not tested and the report-derived labels leave the AUROC uncertain.","tokens_in":9876,"tokens_out":2315,"would_cite":false,"duration_ms":27688,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper develops a Boosted Trees model that predicts radiological signs of acute heart failure from segmented cardiac and pulmonary structures on acute chest CT, reaching AUROC 0.87 on an independent test set.","keywords":["acute heart failure","chest CT","explainable AI","XGBoost","TotalSegmentator","SHAP values","radiology reports","emergency triage"],"falsifier":"Take the independent test set and have a panel of thoracic radiologists re-score each CT for AHF signs using a structured protocol, masking the original reports; if the model's AUROC against these standardized labels falls to near chance (for example around 0.7 or below), the central claim of radiologist-comparable accuracy would be refuted.","tokens_in":8835,"feed_emoji":"❤️","tokens_out":5169,"duration_ms":55424,"temperature":0.7,"pith_summary":"The paper sets out to show that an explainable machine-learning model can detect radiological signs of acute heart failure (AHF) on acute chest CT scans with accuracy comparable to thoracic radiologists. Using measurements of segmented cardiac and pulmonary structures from 4,672 patients, a Boosted Trees model achieves an AUROC of 0.87 on an independent test set. The authors' own review of errors found that 38% of the model's apparent mistakes were actually correct predictions whose labels were wrong, suggesting the reported performance may understate the model's accuracy. If correct, the model could flag likely AHF during emergency triage, before the formal radiology report is available.","feed_headline":"AI flags heart failure on chest CT at AUROC 0.87","feed_subtitle":"Twelve explainable heart and lung measurements match radiologists in flagging AHF on CT.","key_machinery":"The machinery is a Boosted Trees classifier (XGBoost) fed with measurements extracted by TotalSegmentator segmentation of cardiac and pulmonary structures. The final model incorporates twelve features that survive forward feature selection and manual pruning, all directly tied to the pathophysiology of heart failure. Model transparency comes from SHAP values, which assign each feature a contribution to the predicted log odds of AHF. Missing segmentation-derived values are handled as possible splits inside the trees rather than imputed, and predictions from multiple CT acquisitions and reconstructions are averaged at the study level.","core_discovery":"The central discovery is that a small set of twelve pathophysiologically relevant features—such as pleural effusion volume and ratio, left and right atrial volumes, right ventricle ratio, mean lung boundary density, vena cava inferior diameter and density, and a heart-size $Z$-score—carries enough signal to predict AHF signs on acute chest CT with AUROC 0.87, comparable to clinical radiologists. The model is not a black box: because it is a tree ensemble trained on segmented anatomy, each prediction can be traced through the feature values and visualized with SHAP, and the segmentation overlays can be reviewed directly on the scan.","pith_inferences":["A natural extension would be to use the model as a second reader that flags scans whose reports may have underreported AHF signs, turning label error into a quality-control signal for radiology reporting.","The sex difference in error patterns hinted by the data suggests separate thresholds or sex-specific features may be needed in deployment; the paper only notes the trend, so testing this is a next step.","The model's design—off-the-shelf segmentation plus a small tree ensemble—means it could be re-trained on other centers' scans with modest computational cost, provided the report-labeling caveat is addressed."],"forward_implications":["An emergency physician could see an AHF probability before the formal radiology report, potentially triggering earlier cardiology assessment and treatment.","Because each prediction is explained by SHAP feature contributions and visible segmentations, clinicians can audit which anatomical measurement drove the result instead of accepting a black-box output.","At the chosen threshold (a 5% false-positive rate on training data), the model detects about half of the test cases labeled AHF; lowering the threshold increases sensitivity at the cost of more false alarms.","Since 38% of the model's errors were traced to incorrect report labels, the measured specificity and sensitivity against report-derived labels likely underestimate performance against a standardized radiological review."],"supporting_citations":[{"why":"Supplies the Boosted Trees (XGBoost) method used for prediction.","marker":"(4)"},{"why":"Provides the TotalSegmentator software that segments cardiac and pulmonary structures.","marker":"(6)"},{"why":"Supplies Shapley additive explanation values used for interpreting predictions.","marker":"(7)"},{"why":"Prior study defining chest CT features of heart failure that the feature set builds on.","marker":"(5)"},{"why":"Clinical guidelines motivating acute heart failure as a key diagnostic target.","marker":"(1)"},{"why":"Evidence that AI assistance reduces chest CT interpretation time, motivating the triage use.","marker":"(3)"}],"fun_headline_variants":["AI matches radiologists in spotting heart failure on CT","Twelve CT features let AI flag heart failure at AUROC 0.87","Explainable AI detects heart failure on chest CT at 0.87 AUROC","Tree-based AI explains heart failure signs on CT with 0.87 AUC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model's accuracy is measured against labels mined from radiology reports, where scans without any AHF keyword are assumed negative; if those labels are biased or underreport AHF signs, the AUROC of 0.87 may not reflect the model's true diagnostic performance on the scans.","fun_headline_variants_meta":{"raw":{"variants":["AI matches radiologists in spotting heart failure on CT","Twelve CT features let AI flag heart failure at AUROC 0.87","Explainable AI detects heart failure on chest CT at 0.87 AUROC","Tree-based AI explains heart failure signs on CT with 0.87 AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1384,"prompt_tokens":984,"completion_tokens":400,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":320}},"tokens_in":600,"tokens_out":400,"duration_ms":4831,"temperature":1.0,"reasoning_tokens":320,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:07:57.578810+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the independent test set and have a panel of thoracic radiologists re-score each CT for AHF signs using a structured protocol, masking the original reports; if the model's AUROC against these standardized labels falls to near chance (for example around 0.7 or below), the central claim of radiologist-comparable accuracy would be refuted.","supporting_citations":[],"review_version":1}