{"id":"6189d3b4-2cb1-4926-ab7c-6e706d0cd92b","arxiv_id":"1908.02170","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A three-network ensemble on MURA X-rays reaches Cohen Kappa 0.66 overall and 0.653 on finger studies, below the original DenseNet overall (0.705) but above it on fingers.","lead":"This paper tests ensembles of three convolutional neural networks on a public dataset of musculoskeletal X-rays. It reports that one ensemble improves detection on finger studies but admits lower overall agreement than the existing DenseNet baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Same validation set used for model selection and final evaluation makes reported Kappa an optimistic order statistic, not an unbiased generalization estimate; a held-out test is required before claiming ensemble superiority.","rationale":"The reader's weakest assumption is the load-bearing one: final performance was measured on the same validation set used to pick ensemble200. I agree. The central claim is about generalization—lower variability and better finger performance than DenseNet—and generalization cannot be assessed from data that influenced model selection. The manuscript even labels Table 5 with 'based on the validation set data,' and the methodology describes selecting the best ensemble by Kappa, so the bias is concrete rather than hypothetical. Secondary issues (single DenseNet201 Kappa of -0.44 while its ensemble is best; inconsistent AUROC values 0.929 vs 0.815; absence of code or training details) lower confidence further, but the validation-selection problem alone is sufficient to keep the verdict at REJECT. The proposed held-out split is a single check that would settle whether the property survives; if it does, the result would become credible despite the reporting issues.","tokens_in":11696,"tokens_out":5664,"duration_ms":65333,"concrete_test":"Partition the official MURA validation set into two patient-disjoint subsets, V_select and V_test (e.g., 55%/45%). Retrain the four base networks and all candidate ensemble combinations described in Tables 3-4 using only the official training set plus V_select; pick the best ensemble by V_select Kappa; then compute overall and finger Kappa on V_test with 1,000 bootstrap resamples. If the V_test overall Kappa falls below 0.705 or the finger Kappa falls below 0.389 by more than the bootstrap uncertainty, the reported generalization improvement is not supported. This isolates selection bias because V_test is untouched by model choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central generalization claim rests on Table 4/Table 5 Kappa values, and Table 5's own note says these results were 'based on the validation set data.' Section 4.7 describes evaluating different ensemble combinations and selecting the best using the Kappa statistic. So ensemble200 was chosen because it had the highest validation Kappa, and the same validation predictions were then reported as final performance. With 11 candidate models, the reported 0.66 overall Kappa and 0.653 finger Kappa are order statistics of a validation search; even under the null, the maximum validation Kappa will tend to exceed any single fixed model's value, such as the 0.705 DenseNet baseline taken from MURA v4. No confidence intervals or significance tests are given, so the comparison cannot separate a genuine ensemble effect from selection noise. This is acknowledged in the Table 5 note rather than hidden, but it means the manuscript does not establish the claimed improvement in generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of convolutional neural network ensembles for abnormality detection in musculoskeletal radiographs using the public MURA dataset. The authors train four single models (DenseNet201, Xception, MobileNet, NASNetMobile), combine them into eight ensembles via averaging of sigmoid outputs, and evaluate all candidates on the MURA validation set using accuracy, F1 score, precision, recall, AUROC, and Cohen's kappa. The best ensemble, Ensemble200 (DenseNet201, MobileNet, NASNetMobile), reportedly achieves an overall kappa of 0.66 and a finger-study kappa of 0.653, which the authors compare with the MURA v4 model's finger score of 0.389. The paper also includes Grad-CAM localization examples and a web-based demonstration.","tokens_in":11981,"tokens_out":5396,"duration_ms":58012,"significance":"If the reported result held on a properly held-out test set, the paper would provide a useful data point on whether ensembling standard CNN architectures improves abnormality detection on MURA, particularly for finger studies and for reducing performance variability across anatomical regions. The paper addresses a practically important problem and honestly discloses in the Table 5 note that its final results are based on validation-set data. The qualitative Grad-CAM examples and the web interface are welcome additions. However, the central generalization claim is not supported by the current evaluation design, and the absence of uncertainty quantification and reproducible code further limits the paper's present contribution.","major_comments":[{"comment":"","section":"§4.7 and §5, Table 5 note"},{"comment":"","section":"§5, Tables 3 and 4"},{"comment":"","section":"§5, Tables 3 and 4, and §4.6"},{"comment":"","section":"§4.6 and §5"}],"minor_comments":[{"comment":"","section":"Abstract and throughout"},{"comment":"","section":"§2.2 vs. §1"},{"comment":"","section":"§4.4 and Figure 4"},{"comment":"","section":"§5, Tables 3 and 4"},{"comment":"","section":"§5, Table 5, Shoulder row"},{"comment":"","section":"References"}],"recommendation":"reject","confidential_remarks":"The central empirical claim rests on validation-set reuse, which is disclosed in the Table 5 note but nevertheless invalidates the headline comparison. The paper has no code release and many internal inconsistencies, so I do not see this as salvageable by minor revision. If the authors can provide a genuinely held-out evaluation, a resubmission may be worth considering, but the current manuscript does not establish its main claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the paper's only genuinely new empirical claim is that a specific ensemble (DenseNet201+MobileNet+NASNetMobile) gets a Cohen Kappa of 0.653 on the MURA finger subset, well above the 0.389 published baseline. Second, that number is not trustworthy as a generalization estimate because the authors chose the ensemble by looking at validation-set Kappa across 11 combinations and then reported performance on that same validation set. Table 5 says so outright.\n\nThe work isn't worthless. It uses the public MURA dataset, compares against published radiologist baselines, tries a reasonable set of ensembles, and includes Grad-CAM examples and a web demo. The empirical sweep across architectures is the kind of thing that could be a useful benchmark data point. The authors also deserve credit for not hiding the validation-set origin of the results.\n\nThe soft spots are mostly about inference. Selecting the best of 11 candidates on the validation set makes the reported Kappa an order statistic, so it will look better than any fixed model, and without confidence intervals or significance tests you can't tell whether ensemble200 is genuinely better or just lucky. The paper's own table shows the overall Kappa (0.66) is below the MURA v4 baseline (0.705), so the 'outperforms' claim rests entirely on the finger subgroup and F1, which is cherry-picking without a multiple-comparison correction. There is also a numerical slip: the introduction reports the MURA DenseNet as AUROC 0.929, sensitivity 0.815, specificity 0.887, while Section 2.2 says AUROC 0.815. The original MURA paper lists 0.929 as AUROC, so Section 2.2 is likely a typo, but it shows the reporting isn't carefully checked. No code or training details are provided, so reproducibility is limited to the described architecture and hyperparameters.\n\nMy bottom line: the central claim, that ensembling improves generalization on MURA, is not established by the evidence as presented. The work is a reasonable engineering exploration that could be made sound with a held-out test set, proper error bars, and corrected numbers. For someone working on MURA or on ensembles in medical imaging, this is worth a skim for the architecture combination results, but I wouldn't cite the current numbers. If this is submitted somewhere, it deserves a real review rather than a desk reject; the fix is straightforward and the topic is legitimate. I'd send it back with a request for a true test-set evaluation.","headline":"A modest ensemble benchmark on MURA whose headline result is undermined by using the validation set for both model selection and final evaluation.","tokens_in":12400,"tokens_out":2747,"would_cite":false,"duration_ms":27966,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An ensemble of three CNNs (DenseNet201, MobileNet, NASNetMobile) scores a Cohen's kappa of 0.66 overall and 0.653 on finger studies, beating the MURA baseline and all three radiologists on fingers.","keywords":["MURA dataset","musculoskeletal radiographs","abnormality detection","convolutional neural networks","ensemble learning","Cohen's kappa","class activation maps","transfer learning"],"falsifier":"Run the trained ensemble200 on a held-out test cohort that was never used for model selection, for example the official MURA test set, and compare per-study Cohen's kappa with the MURA v4 baseline. If the finger-study kappa no longer exceeds the radiologists' finger scores, or if the wrist-to-hand spread widens, the claimed consistency advantage is an artifact of validation-set selection.","tokens_in":11465,"feed_emoji":"🦴","tokens_out":9471,"duration_ms":86403,"temperature":0.7,"pith_summary":"Musculoskeletal radiograph interpretation is strained by a shortage of imaging experts relative to the volume of examinations, so an automated abnormality detector could support triage. The authors train four CNN architectures on the MURA dataset and stack them into prediction-averaging ensembles, selecting the best by Cohen's kappa. Their ensemble200 (DenseNet201, MobileNet, NASNetMobile) reaches an overall kappa of 0.66, F1 0.86, precision 0.81, and recall 0.92, with a finger-study kappa of 0.653 that exceeds the MURA baseline (0.389) and the best radiologist on that study type. The paper's claim is that ensembling one deep network with two compact networks, together with class balancing and augmentation, lowers performance variability across anatomical sites even though overall kappa stays below the baseline model.","feed_headline":"Ensemble CNN tops radiologists and MURA baseline on finger X-rays","feed_subtitle":"Three-network averaging cuts performance variability across upper-extremity studies, with the biggest gain on fingers.","key_machinery":"The load-bearing mechanism is a prediction-averaging ensemble: the sigmoid outputs of DenseNet201, MobileNet, and NASNetMobile feed an added averaging layer whose output is the abnormality probability. Cohen's kappa is the selection statistic because it measures agreement with radiologist labels beyond chance. Training uses ImageNet-initialized weights, binary cross-entropy, the Adam optimizer, class-weight balancing for the imbalanced normal/abnormal split, and augmentation via random flips and rotations. The paper argues that architectural variety matters: the best ensemble pairs one deep CNN with two compact CNNs. Gradient-weighted Class Activation Maps are then applied to highlight the image regions behind each prediction, which the paper presents as the interpretability layer for clinical use.","core_discovery":"The central claim is that a deliberately heterogeneous ensemble—one deep architecture, DenseNet201, paired with two compact architectures, MobileNet and NASNetMobile—detects abnormality in upper-extremity radiographs more consistently than the single-model DenseNet baseline. On the MURA validation set, this ensemble200 model scores Cohen's kappa 0.66 with F1 0.86, precision 0.81, and recall 0.92; across study types its kappa ranges from 0.7408 (wrist) to 0.5844 (hand), and on finger studies it reaches 0.653. That finger result is above the MURA v4 model's 0.389 and every radiologist's finger score (best 0.410). The overall kappa remains below the MURA v4 model (0.705) and the best radiologist (0.778), so the contribution is framed as improved consistency and balanced recall rather than higher overall agreement.","pith_inferences":["Because the ensemble was selected on the same validation set used for the reported numbers, the finger-study kappa of 0.653 is likely an optimistic estimate; a held-out cohort would probably show a smaller gap (my inference from the paper's table note).","The paper attributes the gain to architectural diversity, which suggests a testable comparison: averaging the same architecture over many seeds should yield less cross-study stability than mixing DenseNet201 with compact models.","The recipe may transfer to other small-bone radiographs (toes, ankles) or to modalities with similar class imbalance and expert scarcity, but that is an extrapolation beyond the evidence presented.","A cleaner test of the ensemble's value would compare ensemble200 against a single DenseNet201 trained with the same augmentation, class balancing, and training budget, isolating ensemble averaging from the other optimization techniques."],"forward_implications":["Finger studies, where the baseline model and every radiologist score lowest, become the clearest target for automated pre-screening if the ensemble result holds.","A model with per-study kappa spanning 0.5844 to 0.7408 would still need per-site reporting and calibration before use as a general upper-extremity screener.","The reported gains depend jointly on architecture choice, class balancing, and augmentation, so removing any one component should erode the consistency advantage.","Grad-CAM localization gives clinicians a concrete region to check, which is the intended path from a prediction to a verifiable clinical decision."],"supporting_citations":[{"why":"Supplies the MURA dataset, the study-type split, radiologist labels, and baseline DenseNet scores that define the task.","marker":"[3]"},{"why":"Provides the ImageNet classification setting whose pretrained weights initialize the CNNs.","marker":"[4]"},{"why":"Supplies the DenseNet-plus-ImageNet transfer-learning recipe that the single models build on.","marker":"[8]"},{"why":"Provides the gradient-based localization method used for model interpretation.","marker":"[13]"},{"why":"Defines the densely connected architecture used for DenseNet201, the deep component of the ensemble.","marker":"[14]"},{"why":"Defines the MobileNet architecture, one of the compact ensemble components.","marker":"[15]"},{"why":"Defines the NASNetMobile architecture, the other compact ensemble component.","marker":"[16]"},{"why":"Defines the Xception architecture used in alternative ensemble combinations tested against the winner.","marker":"[17]"},{"why":"Provides the Adam optimizer used to train all models end to end.","marker":"[18]"}],"fun_headline_variants":["Ensemble CNN tops radiologists and baseline on finger X-rays","Three-network ensemble boosts finger X-ray detection consistency","Compact ensemble cuts variability in upper-extremity X-ray screening","Ensemble model bests finger studies, reduces performance swings","Heterogeneous CNN ensemble beats radiologists on finger radiographs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported kappa values were computed on the same validation set used to choose the best ensemble, so the headline scores assume that this validation performance carries over to patients never seen during development.","fun_headline_variants_meta":{"raw":{"variants":["Ensemble CNN tops radiologists and baseline on finger X-rays","Three-network ensemble boosts finger X-ray detection consistency","Compact ensemble cuts variability in upper-extremity X-ray screening","Ensemble model bests finger studies, reduces performance swings","Heterogeneous CNN ensemble beats radiologists on finger radiographs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1472,"prompt_tokens":1057,"completion_tokens":415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":333}},"tokens_in":673,"tokens_out":415,"duration_ms":5028,"temperature":1.0,"reasoning_tokens":333,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:51:02.297586+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained ensemble200 on a held-out test cohort that was never used for model selection, for example the official MURA test set, and compare per-study Cohen's kappa with the MURA v4 baseline. If the finger-study kappa no longer exceeds the radiologists' finger scores, or if the wrist-to-hand spread widens, the claimed consistency advantage is an artifact of validation-set selection.","supporting_citations":[],"review_version":1}