Abnormality Detection in Musculoskeletal Radiographs with Convolutional Neural Networks(Ensembles) and Performance Optimization

Dennis Banga; Peter Waiganjo

REVIEW 4 major objections 6 minor 17 references

Abnormality Detection in Musculoskeletal Radiographs with Convolutional Neural Networks(Ensembles) and Performance Optimization

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An ensemble of three CNNs (DenseNet201, MobileNet, NASNetMobile) scores a Cohen's kappa of 0.66 overall and 0.653 on finger studies, beating the MURA baseline and all three radiologists on fingers.

desk verdict A modest ensemble benchmark on MURA whose headline result is undermined by using the validation set for both model selection and final evaluation. read the letter →

arxiv 1908.02170 v1 pith:X7IMGFLU submitted 2019-08-06 cs.CY cs.LGeess.IV

Dennis Banga , Peter Waiganjo This is my paper · ORCID

classification cs.CYcs.LGeess.IV

keywords MURAdatasetmusculoskeletalradiographsabnormalitydetectionconvolutionalneuralnetworksensemblelearningCohen'skappaclassactivationmapstransfer

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Musculoskeletal radiograph interpretation is strained by a shortage of imaging experts relative to the volume of examinations, so an automated abnormality detector could support triage. The authors train four CNN architectures on the MURA dataset and stack them into prediction-averaging ensembles, selecting the best by Cohen's kappa. Their ensemble200 (DenseNet201, MobileNet, NASNetMobile) reaches an overall kappa of 0.66, F1 0.86, precision 0.81, and recall 0.92, with a finger-study kappa of 0.653 that exceeds the MURA baseline (0.389) and the best radiologist on that study type. The paper's claim is that ensembling one deep network with two compact networks, together with class balancing and augmentation, lowers performance variability across anatomical sites even though overall kappa stays below the baseline model.

What carries the argument

The load-bearing mechanism is a prediction-averaging ensemble: the sigmoid outputs of DenseNet201, MobileNet, and NASNetMobile feed an added averaging layer whose output is the abnormality probability. Cohen's kappa is the selection statistic because it measures agreement with radiologist labels beyond chance. Training uses ImageNet-initialized weights, binary cross-entropy, the Adam optimizer, class-weight balancing for the imbalanced normal/abnormal split, and augmentation via random flips and rotations. The paper argues that architectural variety matters: the best ensemble pairs one deep CNN with two compact CNNs. Gradient-weighted Class Activation Maps are then applied to highlight the image regions behind each prediction, which the paper presents as the interpretability layer for clinical use.

What would settle it

Run the trained ensemble200 on a held-out test cohort that was never used for model selection, for example the official MURA test set, and compare per-study Cohen's kappa with the MURA v4 baseline. If the finger-study kappa no longer exceeds the radiologists' finger scores, or if the wrist-to-hand spread widens, the claimed consistency advantage is an artifact of validation-set selection.

Watch

Extended reading notes

Core claim

The central claim is that a deliberately heterogeneous ensemble—one deep architecture, DenseNet201, paired with two compact architectures, MobileNet and NASNetMobile—detects abnormality in upper-extremity radiographs more consistently than the single-model DenseNet baseline. On the MURA validation set, this ensemble200 model scores Cohen's kappa 0.66 with F1 0.86, precision 0.81, and recall 0.92; across study types its kappa ranges from 0.7408 (wrist) to 0.5844 (hand), and on finger studies it reaches 0.653. That finger result is above the MURA v4 model's 0.389 and every radiologist's finger score (best 0.410). The overall kappa remains below the MURA v4 model (0.705) and the best radiologist (0.778), so the contribution is framed as improved consistency and balanced recall rather than higher overall agreement.

Load-bearing premise

The reported kappa values were computed on the same validation set used to choose the best ensemble, so the headline scores assume that this validation performance carries over to patients never seen during development.

Editorial extensions

If this is right

Finger studies, where the baseline model and every radiologist score lowest, become the clearest target for automated pre-screening if the ensemble result holds.
A model with per-study kappa spanning 0.5844 to 0.7408 would still need per-site reporting and calibration before use as a general upper-extremity screener.
The reported gains depend jointly on architecture choice, class balancing, and augmentation, so removing any one component should erode the consistency advantage.
Grad-CAM localization gives clinicians a concrete region to check, which is the intended path from a prediction to a verifiable clinical decision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the ensemble was selected on the same validation set used for the reported numbers, the finger-study kappa of 0.653 is likely an optimistic estimate; a held-out cohort would probably show a smaller gap (my inference from the paper's table note).
The paper attributes the gain to architectural diversity, which suggests a testable comparison: averaging the same architecture over many seeds should yield less cross-study stability than mixing DenseNet201 with compact models.
The recipe may transfer to other small-bone radiographs (toes, ankles) or to modalities with similar class imbalance and expert scarcity, but that is an extrapolation beyond the evidence presented.
A cleaner test of the ensemble's value would compare ensemble200 against a single DenseNet201 trained with the same augmentation, class balancing, and training budget, isolating ensemble averaging from the other optimization techniques.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know. First, the paper's only genuinely new empirical claim is that a specific ensemble (DenseNet201+MobileNet+NASNetMobile) gets a Cohen Kappa of 0.653 on the MURA finger subset, well above the 0.389 published baseline. Second, that number is not trustworthy as a generalization estimate because the authors chose the ensemble by looking at validation-set Kappa across 11 combinations and then reported performance on that same validation set. Table 5 says so outright.

The work isn't worthless. It uses the public MURA dataset, compares against published radiologist baselines, tries a reasonable set of ensembles, and includes Grad-CAM examples and a web demo. The empirical sweep across architectures is the kind of thing that could be a useful benchmark data point. The authors also deserve credit for not hiding the validation-set origin of the results.

The soft spots are mostly about inference. Selecting the best of 11 candidates on the validation set makes the reported Kappa an order statistic, so it will look better than any fixed model, and without confidence intervals or significance tests you can't tell whether ensemble200 is genuinely better or just lucky. The paper's own table shows the overall Kappa (0.66) is below the MURA v4 baseline (0.705), so the 'outperforms' claim rests entirely on the finger subgroup and F1, which is cherry-picking without a multiple-comparison correction. There is also a numerical slip: the introduction reports the MURA DenseNet as AUROC 0.929, sensitivity 0.815, specificity 0.887, while Section 2.2 says AUROC 0.815. The original MURA paper lists 0.929 as AUROC, so Section 2.2 is likely a typo, but it shows the reporting isn't carefully checked. No code or training details are provided, so reproducibility is limited to the described architecture and hyperparameters.

My bottom line: the central claim, that ensembling improves generalization on MURA, is not established by the evidence as presented. The work is a reasonable engineering exploration that could be made sound with a held-out test set, proper error bars, and corrected numbers. For someone working on MURA or on ensembles in medical imaging, this is worth a skim for the architecture combination results, but I wouldn't cite the current numbers. If this is submitted somewhere, it deserves a real review rather than a desk reject; the fix is straightforward and the topic is legitimate. I'd send it back with a request for a true test-set evaluation.

Referee Report

4 major / 6 minor

Summary. The paper presents an empirical study of convolutional neural network ensembles for abnormality detection in musculoskeletal radiographs using the public MURA dataset. The authors train four single models (DenseNet201, Xception, MobileNet, NASNetMobile), combine them into eight ensembles via averaging of sigmoid outputs, and evaluate all candidates on the MURA validation set using accuracy, F1 score, precision, recall, AUROC, and Cohen's kappa. The best ensemble, Ensemble200 (DenseNet201, MobileNet, NASNetMobile), reportedly achieves an overall kappa of 0.66 and a finger-study kappa of 0.653, which the authors compare with the MURA v4 model's finger score of 0.389. The paper also includes Grad-CAM localization examples and a web-based demonstration.

Significance. If the reported result held on a properly held-out test set, the paper would provide a useful data point on whether ensembling standard CNN architectures improves abnormality detection on MURA, particularly for finger studies and for reducing performance variability across anatomical regions. The paper addresses a practically important problem and honestly discloses in the Table 5 note that its final results are based on validation-set data. The qualitative Grad-CAM examples and the web interface are welcome additions. However, the central generalization claim is not supported by the current evaluation design, and the absence of uncertainty quantification and reproducible code further limits the paper's present contribution.

major comments (4)

[§4.7 and §5, Table 5 note]
[§5, Tables 3 and 4]
[§5, Tables 3 and 4, and §4.6]
[§4.6 and §5]

minor comments (6)

[Abstract and throughout]
[§2.2 vs. §1]
[§4.4 and Figure 4]
[§5, Tables 3 and 4]
[§5, Table 5, Shoulder row]
[References]

Circularity Check

1 steps flagged · score 6.0 of 10

Validation-set Kappa is used both to select ensemble200 and to report its final performance, so the claimed improvement over DenseNet is an order statistic, not independent evidence.

fitted input called prediction [Section 1 (Introduction) and Section 5 / Table 5 note]
"To improve on the performance consistency and generalization of the convolutional neural network model, an ensemble model was built ... which involved evaluating different ensemble combination performance and selecting the best using the copen kappa statistic. ... (This evaluation results were based on the validation set data.)"

The ensemble model is chosen by maximizing Cohen Kappa on the validation set (Section 1: 'selecting the best using the copen kappa statistic'), and the same validation-set Cohen Kappa is then reported as the final evidence of performance (Tables 4-5, note: 'This evaluation results were based on the validation set data.'). The reported Kappa of 0.66 overall and 0.653 for finger studies is therefore the maximum of the candidate ensembles' validation Kappas, not an independent estimate of generalization. By construction, selecting a model on a metric and then evaluating that same metric on the same data turns the reported value into an order statistic; even with no true ensemble advantage, the selected model's Kappa will tend to exceed a fixed baseline such as MURA v4's 0.705.

full rationale

This paper is an empirical benchmark rather than a formal derivation, so most circularity patterns do not apply: there is no self-citation chain, no imported uniqueness theorem, and no ansatz disguised as a derived rescaling. The central generalization claim, however, rests on a validation-set selection loop. Section 1 states that ensemble combinations were evaluated and the best was selected using the Cohen Kappa statistic; Section 5/Table 5 then reports the selected ensemble200's Kappa on the same validation set as its final performance, explicitly noting 'This evaluation results were based on the validation set data.' The reported Kappa is therefore an order statistic of the candidate ensembles on the validation set, not an unbiased estimate of performance on unseen data. This makes the claimed superiority over the fixed MURA(v4) baseline of 0.705 statistically forced by the selection procedure rather than demonstrated on a held-out test. The paper's own note is a limitation statement that confirms the issue rather than resolving it. This is a partial circularity in the evaluation, not a circular derivation of equations; hence the score is 6 rather than higher. If a held-out test had been reported separately, this paper would merit a score of 0-2 as a self-contained external benchmark.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

This is an empirical ML paper, so it postulates no new physical or mathematical entities. The central result depends on the public MURA dataset's labels and split, on standard but hand-set hyperparameters, and on the validation-based selection of the ensemble. No independent evidence for these assumptions is provided beyond the MURA dataset itself.

free parameters (5)

Ensemble composition (model subset) = DenseNet201, MobileNet, NASNetMobile
Chosen as the combination with the highest validation Cohen Kappa among 8 evaluated combinations (Table 3/4, rank 1). This is a model-selection parameter fitted to the validation set.
Learning rate (Adam) = 1e-4
Section 4.6: learning rate=1e-4. Set by hand, affects all trained models.
Weight decay = 1e-4
Section 4.6: decay=1e-4. Set by hand.
Batch size = 32
Section 4.6: mini-batches of 32. Set by hand.
Class weight balancing heuristic = Not specified
Section 4.6: 'A balanced heuristic was used to weight the two classes' with no numerical details. This weighting affects the loss and hence the trained models.

assumptions (4)

domain assumption MURA dataset labels are ground truth.
The paper relies on radiologist-provided labels in the MURA dataset as correct labels for training and evaluation (Section 4.1).
domain assumption The MURA validation set is an i.i.d. sample representative of the clinical population.
The paper reports final results on the validation set and assumes it represents generalization (Section 4.1, Table 5).
domain assumption Cohen Kappa is the appropriate metric for comparing model and radiologist performance.
The paper adopts Cohen Kappa as the primary comparison metric follows Rajpurkar et al. (Sections 4.4, 5).
domain assumption Pretrained ImageNet weights transfer positively to radiographs.
The models are initialized with pretrained ImageNet weights (Section 4.6), assuming useful feature transfer to X-ray images.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Abnormality Detection in Musculoskeletal Radiographs with Convolutional Neural Networks(Ensembles) and Performance Optimization." pith.science (2026). https://pith.science/paper/X7IMGFLU

@misc{pith2026190802170,
  author       = {Pith},
  title        = {Pith review of: Abnormality Detection in Musculoskeletal Radiographs with Convolutional Neural Networks(Ensembles) and Performance Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X7IMGFLU}},
  note         = {Machine review of arXiv:1908.02170}
}

read the original abstract

Musculoskeletal conditions affect more than 1.7 billion people worldwide based on a study by Global Burden Disease, and they are the second greatest cause of disability[1,2]. The diagnosis of these conditions vary but mostly physical exams carried out and image tests. There are few imaging and diagnostic experts while there is a huge workload of radiograph examinations which might affect diagnostic accuracy. We built machine learning models to perform abnormality detection using the available musculoskeletal public dataset [3]. Convolutional Neural Networks (CNN) were used as are the most successful models in performing various tasks such as classification and object detection [4]. The development of the models involved theoretical study, iterative prototyping, and empirical evaluation of the results. The current model, 169 layer DenseNet, by Pranav et al.(2018) on the abnormality detection task, the performance was lower than the worst radiologist in 5 out of the 7 studies, and the overall model performance was lower than the best radiologist. We developed the ensemble200 model which scored 0.66 Cohen Kappa which was lower than the DenseNet model (Pranav et al, 2018) but the model performance with the F1 score outperforms the DenseNet model and its Cohen Kappa score variability with the different studies is lower as the best cohen kappa score on the upper extremity studies is 0.7408 (Wrist) and the lowest is (0.5844) hand. The ensemble200 model outperformed DenseNet model on the finger studies with a Cohen Kappa score of 0.653 showing reduced performance variability on the model performance.

Figures

Figures reproduced from arXiv: 1908.02170 by the authors.

**Figure 1.** Ensembled Model 4.4 Data Analysis The MURA dataset was utilized for the evaluation of the models. The training and the validation dataset was utilized and the quantitative performance metrics measured were the accuracy, specificity, sensitivity, precision, recall, F1-Score, AUROC (Area Under Receiver Operating Characteristics), and Cohen's Kappa statistic to compare model and radiologist score as mapped out by Rajpu… view at source ↗

**Figure 2.** Ensemble Combinations and Single Model. 4.6 Network Training [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Ensembled Architectural Design. Four different single models were trained on the MURA dataset, namely: DenseNet201, Xception, MobileNet, NASNetMobile models. The models were initialized with pretrained imagenet weights, then an additional layer of global average pooling was added to the network that gets an input of a 4 dimensional tensor and outputs a 2 dimensional tensor. Then a sigmoid function was applied on the… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Sigmoid function Equation. It converts the input value to a value between 0 and 1. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 6.** Figure 6: Binary cross entropy loss function. The label y is 1 for normal radiographic images and 0 for abnormal radiographic images, while p(y) is the predicted probability for the normal radiographic image. The Adam optimizer is a 1st order gradient optimizer for stochastic ob…

**Figure 7.** Figure 7: Ensemble Function and the Ensemble Varying Combinations [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 8.** Figure 8: Ensemble Model Implementation. (Average predictions layer) [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 8 canonical work pages

[1]

Thelancet.com. (2016). On the road to universal health care in Indonesia, 1990–2016: a systematic analysis for the Global Burden of Disease Study

work page 2016
[5]

World Health Organization. (n.d.). Musculoskeletal conditions. [online] Available at: http://www.who.int/mediacentre/factsheets/musculoskeletal/en/ [Accessed 27 Jun. 2018]

work page 2018
[6]

[online] Available at: https://hiskenya.org/ [Accessed 24 Jun

Kenya Health Information System. [online] Available at: https://hiskenya.org/ [Accessed 24 Jun. 2018]

work page 2018
[7]

[online] Ihris.or.ke

iHRIS Manage Kenya. [online] Ihris.or.ke. Available at: http://ihris.or.ke/moh/index.php/login [Accessed 24 Jun. 2018]

work page 2018
[8]

Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning

Rajpurkar, Pranav, Irvin, Jeremy, Zhu, Kaylie, Yang, Brandon, Mehta, Hershel, Duan, Tony, Ding,Daisy, Bagul, Aarti, Langlotz, Curtis, Shpanskaya, Katie, et al. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225, 2017b

arXiv
[9]

RADNET: Radiologist Level Accuracy using Deep Learning for HEMORRHAGE detection in CT Scans

Grewal, M., Srivastava, M., Kumar, P. and Varadarajan, S. (2017). RADNET: Radiologist Level Accuracy using Deep Learning for HEMORRHAGE detection in CT Scans. [online] arXiv.org. Available at: https://arxiv.org/abs/1710.04934 [Accessed 24 Jun. 2018]

work page Pith review arXiv 2017
[10]

Dermatologist-level classification of skin cancer with deep neural networks

Esteva A, et al., (2017). Dermatologist-level classification of skin cancer with deep neural networks. - PubMed - NCBI . [online] Ncbi.nlm.nih.gov. Available at: https://www.ncbi.nlm.nih.gov/pubmed/28117445 [Accessed 27 Jun. 2018]

work page arXiv 2017
[11]

and Goubran, R

Xi, P., Shu, C. and Goubran, R. (2018). Abnormality Detection in Mammography using Deep Convolutional Neural Networks - IEEE Conference Publication. [online] Ieeexplore.ieee.org. Available at: https://ieeexplore.ieee.org/document/8438639 [Accessed 27 Jun. 2018]

work page arXiv 2018

Show all 17 references

[13]

and Batra, D

Selvaraju, R., Cogswell, M., Das, A., Vedantam, R., Parikh, D. and Batra, D. (2019). Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. [online] arXiv.org. Available at: https://arxiv.org/abs/1610.02391 [Accessed 28 Jan. 2019]

2019 arXiv
[14]

and Weinberger, K

Huang, G., Liu, Z., van der Maaten, L. and Weinberger, K. (2019). Densely Connected Convolutional Networks. [online] arXiv.org. Available at: 15 https://arxiv.org/abs/1608.06993 [Accessed 20 Jan. 2019]

2019 arXiv
[15]

and Adam, H

Howard, A., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M. and Adam, H. (2019). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. [online] arXiv.org. Available at: https://arxiv.org/abs/1704.04861 [Accessed 20 Feb. 2019]

2019 arXiv
[16]

and Le, Q

Zoph, B., Vasudevan, V., Shlens, J. and Le, Q. (2019). Learning Transferable Architectures for Scalable Image Recognition. [online] arXiv.org. Available at: https://arxiv.org/abs/1707.07012 [Accessed 20 Apr. 2019]

2019 arXiv
[17]

Chollet, F. (2019). Xception: Deep Learning with Depthwise Separable Convolutions. [online] arXiv.org. Available at: https://arxiv.org/abs/1610.02357 [Accessed 20 Feb.. 2019]

2019 arXiv
[18]

and Ba, J

Kingma, D. and Ba, J. (2014). Adam: A Method for Stochastic Optimization. [online] arXiv.org. Available at: https://arxiv.org/abs/1412.6980 16

2014 arXiv
[2015]

Available: http://arxiv.org/abs/1409.4842

[Online]. Available: http://arxiv.org/abs/1409.4842

arXiv
[2016]

[online] Available at: https://www.thelancet.com/pdfs/journals/lancet/PIIS0140-6736(18)30595-6.pdf [Accessed 24 Jun. 2018]

2018
[2018]

Available: https://arxiv.org/abs/1712.06957v4

[Online]. Available: https://arxiv.org/abs/1712.06957v4

arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Thelancet.com. (2016). On the road to universal health care in Indonesia, 1990–2016: a systematic analysis for the Global Burden of Disease Study

work page 2016

[2] [5]

World Health Organization. (n.d.). Musculoskeletal conditions. [online] Available at: http://www.who.int/mediacentre/factsheets/musculoskeletal/en/ [Accessed 27 Jun. 2018]

work page 2018

[3] [6]

[online] Available at: https://hiskenya.org/ [Accessed 24 Jun

Kenya Health Information System. [online] Available at: https://hiskenya.org/ [Accessed 24 Jun. 2018]

work page 2018

[4] [7]

[online] Ihris.or.ke

iHRIS Manage Kenya. [online] Ihris.or.ke. Available at: http://ihris.or.ke/moh/index.php/login [Accessed 24 Jun. 2018]

work page 2018

[5] [8]

Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning

Rajpurkar, Pranav, Irvin, Jeremy, Zhu, Kaylie, Yang, Brandon, Mehta, Hershel, Duan, Tony, Ding,Daisy, Bagul, Aarti, Langlotz, Curtis, Shpanskaya, Katie, et al. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225, 2017b

arXiv

[6] [9]

RADNET: Radiologist Level Accuracy using Deep Learning for HEMORRHAGE detection in CT Scans

Grewal, M., Srivastava, M., Kumar, P. and Varadarajan, S. (2017). RADNET: Radiologist Level Accuracy using Deep Learning for HEMORRHAGE detection in CT Scans. [online] arXiv.org. Available at: https://arxiv.org/abs/1710.04934 [Accessed 24 Jun. 2018]

work page Pith review arXiv 2017

[7] [10]

Dermatologist-level classification of skin cancer with deep neural networks

Esteva A, et al., (2017). Dermatologist-level classification of skin cancer with deep neural networks. - PubMed - NCBI . [online] Ncbi.nlm.nih.gov. Available at: https://www.ncbi.nlm.nih.gov/pubmed/28117445 [Accessed 27 Jun. 2018]

work page arXiv 2017

[8] [11]

and Goubran, R

Xi, P., Shu, C. and Goubran, R. (2018). Abnormality Detection in Mammography using Deep Convolutional Neural Networks - IEEE Conference Publication. [online] Ieeexplore.ieee.org. Available at: https://ieeexplore.ieee.org/document/8438639 [Accessed 27 Jun. 2018]

work page arXiv 2018