{"id":"9b022772-9e8a-4f2d-869f-d43dfa90a6ae","arxiv_id":"2411.18063","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"On 193 single-center CT scans, a 3DResNet-plus-XGBoost pipeline reports 94% accuracy for 30-day PE mortality, but the evaluation has likely leakage and internal inconsistencies.","lead":"PEP-Net combines a 3D ResNet feature extractor with XGBoost to predict 30-day mortality from CT scans of 193 pulmonary embolism patients, reporting 94% accuracy. A generalist reader might care because a reliable imaging-only mortality prediction could change how PE patients are triaged, but the result is not yet externally validated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported metrics are internally inconsistent: 94.5% accuracy, 97.7% sensitivity, and 87.4% specificity cannot hold for 193 patients with 38 deaths; the unresolved B-SMOTE/PCA leakage question compounds this, so the central 94.5% claim is unsupported.","rationale":"The reader already rejects the paper, and I agree. My single most load-bearing concern is the internal inconsistency of the headline metrics, which is stronger than the leakage concern because it does not depend on interpreting an ambiguous pipeline description: the numbers in Section 3 cannot all describe the same evaluation. This is not a disagreement with consensus; it is a direct arithmetic conflict. I also agree with the reader's weaker concern about B-SMOTE/PCA leakage; the manuscript never states that oversampling and PCA are confined to training folds, and the choice of 100 PCA components on the same data is a feature-selection leak risk. A confusion-matrix release would settle the inconsistency; if the authors can produce a coherent matrix, the leakage question still needs a fold-restricted rerun. Given the tiny positive class (38 deaths), the missing PESI comparison, and the absence of code, data, and external validation, the result cannot be accepted as a reliable benchmark; the verdict should remain REJECT. That is why verdict_should_be is UNCHANGED rather than a new category.","tokens_in":7855,"tokens_out":8222,"duration_ms":73261,"concrete_test":"Obtain or reconstruct the per-fold confusion matrices and the exact test predictions for all 193 patients, then check the identities Accuracy=(TP+TN)/193, Sensitivity=TP/38, and Specificity=TN/155. The reported Table 2 values require TP about 37 and TN about 136, giving about 173 correct, not about 182; the '7 misclassified patients' claim requires TP=34 and TN=152, giving 186 correct. If the released counts match neither the table nor the prose, the reported results are internally inconsistent; if they match one, the other is wrong. This direct arithmetic check requires no reimplementation and would settle whether the central 94.5% accuracy number can be taken at face value.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3's headline metrics are mutually incompatible, which is more damaging than any missing procedural detail. For Table 2 (Lung-ROI): N=193, with 38 deaths and 155 survivors. Sensitivity 0.977 implies about 37 true positives and 1 false negative; specificity 0.874 implies about 135 true negatives and 20 false positives; total correct is about 173, which is an accuracy of about 89.4%, not 94.5%. The prose then says 'our algorithm misclassified only 7 patients'; 7 errors would give accuracy 96.4%, sensitivity 34/38=89.5%, and specificity 152/155=98.1%, again not matching. Unless the table and the prose describe different runs or subsets, which the paper does not state, no per-fold averaging can reconcile these because the pooled TP/TN counts are just the sums of the fold counts that define the averages. The central 94.5% accuracy claim therefore cannot be true as stated. Separately, Section 2 describes B-SMOTE and PCA before the final XGBoost step without stating that they are fitted independently inside each training fold, and the PCA dimension (100) was chosen by experimentation over 50 to 150 on the same data; if any of these steps see the full dataset, validation accuracy can be inflated by leakage. Either problem alone is reject-worthy; together they leave the abstract's benchmark unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents PEP-Net, a pipeline that combines a 3D ResNet feature extractor with PCA dimensionality reduction, BorderlineSMOTE oversampling, and an XGBoost classifier to predict 30-day in-hospital mortality from CT scans of 193 patients with acute pulmonary embolism. The authors report accuracy of 94.5% (±0.3) for Lung-ROI and 94.0% (±0.7) for Cardiac-ROI inputs, with AUCs of 0.92 and 0.90, and claim that this significantly outperforms baseline CNN models whose accuracies are in the 71–80% range. The paper describes five-fold cross-validation and includes a brief limitations discussion acknowledging the single-center dataset and absence of clinical parameters.","tokens_in":8094,"tokens_out":2852,"duration_ms":26756,"significance":"If the reported performance were valid, the result would be clinically notable: a purely imaging-based model achieving roughly 94% accuracy for 30-day mortality after acute PE would substantially exceed commonly cited prognostic scores such as PESI. The paper also addresses a relatively understudied task, mortality prognosis rather than PE diagnosis, which is a legitimate and important problem. However, the central claim is not supported as stated. The headline metrics in Table 2 are mutually inconsistent with the cohort size and class distribution, and the manuscript does not establish that oversampling and PCA-based dimensionality selection are confined to the training folds. Both issues are load-bearing for the abstract's benchmark. The paper deserves credit for using publicly available segmentation checkpoints, describing the imaging protocol, and framing the task clearly, but the reported quantitative claims cannot be accepted without correction and a precisely specified evaluation protocol.","major_comments":[{"comment":"The reported metrics for PEP-Net with Lung-ROI are internally inconsistent. For N=193 with 38 deaths and 155 survivors, sensitivity 0.977 implies approximately 37 true positives and 1 false negative, while specificity 0.874 implies approximately 135 true negatives and 20 false positives. The resulting accuracy is (37+135)/193 ≈ 0.89, not 0.945. Even after rounding to integer counts, 37 true positives, 1 false negative, 135 true negatives, and 20 false positives give 172 correct out of 193, or 89.1%. No per-fold averaging can reconcile these pooled counts, because the pooled true-positive and true-negative totals are simply the sums of the fold-level counts that define the averaged sensitivity and specificity. The central accuracy claim in the abstract and Table 2 is therefore not supported by the numbers reported.","section":"Section 3, Table 2"},{"comment":"The manuscript never states that B-SMOTE is applied only inside each training fold. The pipeline description in Figure 2 and the text present B-SMOTE and PCA as sequential steps before the XGBoost classification step, and the PCA component count of 100 is described as selected empirically by testing a range of 50 to 150 components. If B-SMOTE is applied to the full dataset before cross-validation, synthetic minority-class samples will appear in the validation folds, and the classifier can achieve artificially high accuracy by recognizing memorized synthetic examples rather than by generalizing to real patients. Similarly, selecting the PCA dimension using the entire dataset without nested validation makes the reported accuracy partly a fitted quantity. The manuscript must explicitly state that all preprocessing, including oversampling, PCA fitting, and component-count selection, is performed independently within each training fold, or the reported performance cannot be interpreted as a predictive claim.","section":"Section 2, 'Oversampling and Feature Selection' and 'Further Details on Training'"},{"comment":"The prose states 'our algorithm misclassified only 7 patients' and then specifies three false positives (alive predicted as dead) and four false negatives (dead predicted as alive). For the Lung-ROI results in Table 2, seven errors would correspond to an accuracy of 186/193 ≈ 96.4%, not 94.5%. For the Cardiac-ROI results in Table 3, seven errors would correspond to 96.4% as well, not 94.0%. Moreover, with three false positives and four false negatives, sensitivity would be 34/38 ≈ 89.5% and specificity would be 152/155 ≈ 98.1%, neither of which matches the sensitivity and specificity in Table 2 or Table 3. Unless the tables and the prose refer to different runs, datasets, or evaluation subsets, which the manuscript does not state, the results are mutually incompatible and cannot be reproduced from the reported confusion counts.","section":"Section 3, text after Figure 3"},{"comment":"The standard deviations reported for PEP-Net are implausibly small relative to the baselines: ±0.003 in accuracy and ±0.002 in sensitivity for Lung-ROI, and ±0.007 in accuracy for Cardiac-ROI. With five-fold cross-validation on only 193 patients, such tiny variance would require nearly identical performance across folds. Given the internal inconsistency of the aggregate metrics, the reported standard deviations do not provide evidence of stability. The authors should report per-fold results or a confusion matrix for the pooled predictions so that the reader can verify the reported mean and variance.","section":"Section 3, Tables 2 and 3"}],"minor_comments":[{"comment":"The citation for 3DResNet appears to be [18], but the reference list entry [18] is Zhou et al.'s meta-analysis of the PESI score, not a ResNet architecture paper. The manuscript should cite the original ResNet publications or a correct medical-imaging adaptation of ResNet.","section":"Introduction and References"},{"comment":"There is a typo in 'Sparse Categorical Cross Entropyas the loss function'; it should read 'Sparse Categorical Cross Entropy as the loss function'.","section":"Section 2, 'Further Details on Training'"},{"comment":"The affiliation for the third author contains a typo: 'Univesity' should be 'University'.","section":"Title page / Author affiliations"},{"comment":"Several numbers in the tables contain an errant space after the decimal point, such as '0 .581' and '0 .782'; these should be formatted consistently as '0.581' and '0.782'.","section":"Tables 2 and 3"},{"comment":"The caption describes left, middle, and right panels, but the text only references the right and middle panels. Please ensure that the panel descriptions are complete and that the ROC curve (left) is explicitly discussed.","section":"Section 3, Figure 3"}],"recommendation":"reject","confidential_remarks":"The internal inconsistency of Table 2 and the prose in Section 3 is the strongest reason for rejection: it is not a stylistic issue but a contradiction in the central quantitative claim. The B-SMOTE/PCA leakage concern is also serious and should have been addressed in the methods even for a conference-length paper. If the authors can supply corrected numbers and a fully specified fold-restricted preprocessing protocol, the underlying idea may be salvageable as a new submission, but the current manuscript cannot be published as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core result here is the claim that a CT-only pipeline (ROI segmentation, 3DResNet features, PCA, BorderlineSMOTE, XGBoost) predicts 30-day PE mortality at 94.5% accuracy and 0.92 AUC on 193 scans. That would be a strong result for a common emergency condition, and the underlying question—whether outcome can be prognosticated from initial imaging without emboli annotations or clinical variables—is legitimately interesting and understudied. The pipeline is a sensible composition of off-the-shelf components, and the paper is transparent about the single-center design and the absence of clinical parameters.\n\nThe problem is that the headline numbers do not add up. With 38 deaths and 155 survivors, sensitivity 0.977 and specificity 0.874 imply roughly 172 correct predictions, i.e., about 89% accuracy, not 94.5%. The text also says only 7 patients were misclassified, which would give 96.4% accuracy and sensitivity around 89.5%. These three statements cannot all be true for the same test set, and the paper doesn't explain that they come from different runs or subsets. The AUC might be consistent with some of these, but the mismatch is fatal to the abstract's benchmark.\n\nOn top of that, the evaluation protocol is under-specified. B-SMOTE and PCA are described as pipeline steps before XGBoost, with no explicit statement that they are fitted inside each training fold. The PCA component count (100) was selected by experimentation over 50–150 on the full dataset. If oversampling or dimensionality selection sees the validation fold, the measured accuracy is inflated by leakage. The tiny positive class (38 events) makes fold-level estimates fragile even without leakage.\n\nI want to give credit where it's due: the motivation is clear, the use of TotalSegmentator for rough ROIs is sensible, and the limitations section is honest about needing multi-center data and clinical variables. But the central empirical claim is not supported as stated. The arithmetic inconsistency alone means the abstract cannot be trusted, and the leakage ambiguity means even a corrected accuracy would need re-analysis.\n\nWho gets value from this? Someone building imaging-based PE prognosis might use the pipeline as a starting point, but not the reported numbers. It is not a paper I'd cite for its results. If it were submitted to a journal, I'd desk reject it: the flaws are not fixable by minor clarification, and the core claim needs re-running with a properly nested evaluation and consistent reporting.\n\nRecommendation: don't send this to peer review in its current form. The authors should be invited to re-analyze with a strict per-fold protocol and to reconcile the reported metrics.","headline":"Imaging-only PE mortality prediction is a real clinical aim and the pipeline is coherent, but the reported metrics contradict each other and the evaluation may leak through SMOTE/PCA, so the headline result is unsupported.","tokens_in":8767,"tokens_out":4224,"would_cite":false,"duration_ms":37472,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that a hybrid CT pipeline predicts 30-day pulmonary embolism mortality with 94% accuracy, outperforming clinical scores and CNN baselines.","keywords":["pulmonary embolism","30-day mortality prediction","CT imaging","deep learning","XGBoost","3D residual network","class imbalance","prognosis"],"falsifier":"Re-run the five-fold evaluation twice on the same 193 scans: once with Borderline-SMOTE and PCA applied only inside each training fold, and once with the pipeline order exactly as described (oversampling and PCA before cross-validation). If the two accuracies are both near 94%, the claim survives; if the leakage-free version drops toward the 71–80% baseline range, the reported gain is an artifact of synthetic validation samples.","tokens_in":7577,"feed_emoji":"🫁","tokens_out":7702,"duration_ms":61213,"temperature":0.7,"pith_summary":"This paper claims that the initial chest CT scan, without any annotation of where the emboli are, carries enough information to predict whether an acute pulmonary embolism patient will die within 30 days. The authors built PEP-Net, a five-stage pipeline that extracts volumetric features with a 3D residual network, reduces them with PCA, balances the classes with Borderline-SMOTE, and classifies with XGBoost. On 193 CT scans, it reports 94.5% accuracy (AUC 0.917) when the input is the lung region and 94.0% accuracy (AUC 0.901) when the input is the cardiac region, clearly above the 71–80% range of the deep-learning baselines. If reliable, this would let clinicians use the scan already taken at admission as a prognostic tool, supplementing clinical scores whose specificity is limited.","feed_headline":"CT scans alone predict pulmonary embolism deaths with 94% accuracy","feed_subtitle":"Imaging-only prognosis could flag high-risk PE patients without waiting on clinical scores.","key_machinery":"The central object is PEP-Net, a five-step pipeline. Step one uses a conventional 3D U-Net, initialized with publicly available pretrained weights, to segment rough lung and cardiac regions from the CT and crop bounding boxes around them. Step two uses a pretrained 3D ResNet18 adapted for single-channel grayscale input, with the first and last layers replaced, to extract hierarchical volumetric features. Step three applies Borderline-SMOTE, which creates synthetic minority-class samples by interpolating between a minority sample and one of its neighbors, to correct the 20/80 death/survival imbalance. Step four applies PCA and keeps 100 components, chosen by experimenting over 50–150. Step five trains XGBoost with a logistic objective, a learning rate of 0.1, tree depth 3, and 100 boosting rounds, using scale_pos_weight to emphasize the minority class. The machinery's job is to shift the burden from end-to-end deep classification to a feature-extraction-plus-boosting design that the authors argue is better suited to small, imbalanced medical cohorts.","core_discovery":"On its own terms, the paper's core claim is that 30-day mortality in acute PE can be read from routine CT imaging by a model that never sees the embolus location. PEP-Net takes a rough lung or cardiac region from a pretrained segmentation model, extracts 3D features with a ResNet18 backbone adapted for single-channel CT, compresses those features to 100 principal components, generates synthetic minority-class samples with Borderline-SMOTE, and feeds the result to XGBoost. In five-fold cross-validation on 193 patients (38 deaths, 155 survivors), the Lung-ROI version scored accuracy 0.945±0.003, AUC 0.917±0.007, sensitivity 0.977, specificity 0.874; the Cardiac-ROI version scored accuracy 0.940±0.007, AUC 0.901±0.008. The paper reports that only seven patients were misclassified overall, and that the near-parity of lung and cardiac results suggests the model detects cardiac or vascular changes that are not obvious to the human eye. The authors position the result as a new benchmark for imaging-based PE prognostics and argue that the ResNet-plus-XGBoost combination beats end-to-end deep classifiers on this small dataset.","pith_inferences":["Editorial inference: the reported 94% accuracy is probably optimistic for a single-center 193-scan cohort with synthetic oversampling; a multi-center replication would likely yield a lower figure, and even a drop to the 85–90% range would still be clinically meaningful.","Editorial inference: the cardiac-ROI result suggests a testable hypothesis that CT-detectable right ventricular strain, measurable as RV/LV diameter ratio, mediates part of the mortality signal; comparing PEP-Net's cardiac features to this established measurement would separate a genuine biomarker from classifier artifacts.","Editorial inference: the same pipeline could be applied to other admission-imaging prognosis tasks, such as predicting sepsis or stroke outcomes, but only if future work strictly confines oversampling and PCA fitting to training folds; otherwise the accuracy gain could be an artifact of validation leakage rather than real signal."],"forward_implications":["If the reported accuracy holds, an admission CT alone could flag high-risk PE patients for early intensive management without waiting for clinical scoring or biomarker results.","The model needs only patient-level outcome labels during training, not radiologist annotations of emboli location or extent, so it could be retrained on existing hospital outcome data.","Cardiac-ROI performance nearly matching Lung-ROI performance implies the mortality signal may be partly cardiac (for example, right-heart strain) rather than solely the visible clot burden, which could motivate targeted imaging-biomarker studies.","The ResNet-plus-XGBoost architecture, with class balancing and dimensionality reduction, gives a recipe for other small-data medical imaging prognosis tasks where pure deep networks overfit."],"supporting_citations":[{"why":"Defines the clinical PESI baseline (sensitivity 85–90%, specificity 35–45%) that PEP-Net's accuracy is compared against; the same marker is also attached to the 3DResNet architecture in the Methods.","marker":"[18]"},{"why":"Supplies the pretrained lung and heart segmentation models used to localize the Lung-ROI and Cardiac-ROI inputs.","marker":"[14]"},{"why":"Provides the 3D ResNet architecture the feature extractor is adapted from and one of the deep-learning baselines.","marker":"[16]"},{"why":"Provides the modified 3D EfficientNet used as an additional baseline model.","marker":"[17]"},{"why":"Documents the high early mortality of acute PE (over 20% at 14 days in the ICOPER registry), motivating the prognostic task.","marker":"[10]"},{"why":"States the current ESC guideline approach to PE diagnosis and prognosis that the paper positions itself against.","marker":"[11]"}],"fun_headline_variants":["CT alone predicts PE death risk with 94% accuracy","AI reads CT to predict 30-day PE death risk","ResNet+XGBoost predicts PE deaths from CT alone","No emboli annotation needed: CT predicts PE deaths","Imaging-only model hits 94% accuracy for PE mortality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the synthetic samples created by Borderline-SMOTE and the choice of 100 PCA components are kept inside each cross-validation fold, so the validation fold never sees patients synthesized from training data; the paper describes both as preprocessing steps without stating this safeguard, and if they leak, the 94% accuracy could be a memorization artifact rather than a true prognostic signal.","fun_headline_variants_meta":{"raw":{"variants":["CT alone predicts PE death risk with 94% accuracy","AI reads CT to predict 30-day PE death risk","ResNet+XGBoost predicts PE deaths from CT alone","No emboli annotation needed: CT predicts PE deaths","Imaging-only model hits 94% accuracy for PE mortality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001112,"raw_usage":{"total_tokens":4702,"prompt_tokens":1087,"completion_tokens":3615,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":703,"completion_tokens_details":{"reasoning_tokens":3533}},"tokens_in":703,"tokens_out":3615,"duration_ms":21375,"temperature":1.0,"reasoning_tokens":3533,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:32:32.731694+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the five-fold evaluation twice on the same 193 scans: once with Borderline-SMOTE and PCA applied only inside each training fold, and once with the pipeline order exactly as described (oversampling and PCA before cross-validation). If the two accuracies are both near 94%, the claim survives; if the leakage-free version drops toward the 71–80% baseline range, the reported gain is an artifact of synthetic validation samples.","supporting_citations":[{"cited_title":"Zhou, S.-Q","cited_arxiv_id":null,"evidence_quote":"Defines the clinical PESI baseline (sensitivity 85–90%, specificity 35–45%) that PEP-Net's accuracy is compared against; the same marker is also attached to the 3DResNet architecture in the Methods."},{"cited_title":"Wasserthal, H.-C","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained lung and heart segmentation models used to localize the Lung-ROI and Cardiac-ROI inputs."},{"cited_title":"Xue and C","cited_arxiv_id":null,"evidence_quote":"Provides the 3D ResNet architecture the feature extractor is adapted from and one of the deep-learning baselines."},{"cited_title":"Zheng, A","cited_arxiv_id":null,"evidence_quote":"Provides the modified 3D EfficientNet used as an additional baseline model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the high early mortality of acute PE (over 20% at 14 days in the ICOPER registry), motivating the prognostic task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"States the current ESC guideline approach to PE diagnosis and prognosis that the paper positions itself against."}],"review_version":1}