REVIEW 3 major objections 4 minor 33 references
Assessing Knee OA Severity with CNN attention-based end-to-end architectures
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A trainable attention CNN can grade knee osteoarthritis without the separate joint-localization stage that earlier pipelines require.
desk verdict A modest, clearly reported knee-OA grading paper whose headline claim is under-tested: without a no-attention VGG-16 baseline on the same split, the reported gain cannot be attributed to the attention masks. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the trainable attention branch. Given a convolutional volume $D_l$ with $N$ feature maps, stacked $1\times1$ convolutions extract spatial features, and a $1\times1$ locally connected layer with sigmoid activation produces an attention mask $A_l$. The original feature maps are element-wise multiplied by the mask, global-average-pooled, and normalized by the average value of the mask to form a feature vector, with a softmax on top yielding class probabilities. Placing such branches after different pooling blocks gives attention at different resolutions, and multi-loss training, a weighted sum of per-branch cross-entropies with best weights $w_0=1$ and $w_1=0.8$, balances branches that converge at different speeds. At test time the single best branch (att0) is used as the classifier.
What would settle it
Take the same VGG-16 backbone, the same multi-loss schedule, and the same data split, and replace every attention mask with an all-ones map (or remove the attention branch entirely); if test accuracy and kappa stay near 64.3% and 0.63, the attention masks are not carrying the result, whereas a clear drop would show the masks are doing the localization work.
Extended reading notes
Core claim
The central claim is that attaching trainable attention modules to a standard CNN at several depths lets the network learn, from image-level class labels alone, where the knee joint is and how severe its osteoarthritis is. On the OAI and MOST public datasets, the best configuration, the att0 branch of VGG-16 trained with multi-loss weighting $w_0=1$, $w_1=0.8$, achieves 64.3% test accuracy and a Cohen's kappa of 0.63, described as substantial agreement reaching the margins of human accuracy. This matches or improves on earlier pipelines that required a separate FCN-based knee-joint localization stage, while avoiding the need for manual joint annotations. The authors also state that the method does not surpass the state of the art; its value is reducing training complexity through an end-to-end design.
Load-bearing premise
The approach stands on the assumption that the attention masks reliably point at the knee-joint region and add signal rather than noise; the paper itself concedes that unsupervised localization can inject noise into the masks, and it never compares the same base network trained without attention.
Editorial extensions
If this is right
- Knee OA severity can be graded from raw X-ray halves with no separate knee-joint detector and no manual joint annotations, because the attention branches learn to localize while classifying.
- Multi-loss training lets shallower and deeper attention branches converge together; weighting the deeper branch lower ($w_1=0.8$) reduces its overfitting and improves the shallower branch's masks.
- The attention mechanism transfers across base architectures, but the best branch location shifts with network depth: shallower branches (att0, att1) work best in deeper models like VGG-16 and ResNet-50.
- The resulting agreement ($\kappa=0.63$) sits within the moderate-to-substantial range reported for human readers, so the approach is clinically relevant even though it does not beat state-of-the-art accuracy.
Reading between the lines
- If the attention masks are as reliable as the accuracy figures suggest, the same masks could be exported as interpretable heatmaps for clinicians, turning localization into a free byproduct of grading; the paper does not explore this visualization use.
- The reported comparison does not include a VGG-16 baseline trained with the same multi-loss recipe but without attention, so the contribution of the masks themselves is not fully isolated; part of the gain may come from multi-loss regularization rather than the attention mechanism.
- A natural testable extension is to run the same architecture on full, unsplit X-rays and let attention find both knees simultaneously, checking whether bilateral context improves grading over the split-half approach.
- Because the authors observe that unsupervised localization can add noise, a sparsity or smoothness penalty on the attention masks could reduce that noise and potentially push accuracy above the reported 64.3%.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an end-to-end CNN architecture for knee osteoarthritis severity grading that inserts trainable attention modules at multiple convolutional blocks, acting as unsupervised ROI detectors. The modules generate attention masks multiplied with feature maps, followed by global average pooling and softmax classification branches; branches are combined by early fusion or multi-loss training. On OAI+MOST images (split into left/right knees, 70/30 train-test split), a VGG-16 attention branch att0 trained with multi-loss (w0=1, w1=0.8) achieves 64.3% accuracy and Cohen's kappa 0.63, compared to retrained Antony et al. baselines of 59% and 62.29%. The authors conclude that the end-to-end attention design avoids a separate localization step while reaching the margins of human accuracy.
Significance. If the central result holds, the contribution is practically useful: a single network trained only on KL grade labels can localize and grade the knee joint, removing the FCN localization stage and its manual annotation requirement. The paper is clearly written, reports hyperparameters in detail, and promises public code. However, the significance is currently limited by the absence of a no-attention control on the same split, the use of a non-matching human-kappa reference, and the lack of uncertainty estimates, which means the quantitative superiority of the attention mechanism is not yet established.
major comments (3)
- [Section 4, Table 1] The claim that attention branches act as unsupervised fine-grained ROI detectors and yield the reported 64.3% accuracy requires a control: train the same base architectures (at least VGG-16) on the full images with the attention modules removed, under the identical 70/30 split, preprocessing, optimizer, batch size, and multi-loss weight selection. A uniformly-ones attention mask reduces the proposed module to a GAP+FC classifier on conv features, so the no-attention model is a special case of the proposed architecture and is exactly the comparison needed to support the central claim. Section 5 concedes that unsupervised localization 'can reduce performance by adding noise in the attention masks,' so the authors themselves anticipate a regime where the mask hurts. Without this control, the reported gain over Antony et al. could be due to the deeper VGG-16 backbone or other training details rather than to the attention module. Please add the no-attention baseline and, ideally, repeat it across multiple seeds.
- [Section 4, Table 2] The comparison with human performance uses the radiologic reliability readings of Klara et al., which, as the paper states, 'does not match our test set.' The human kappa range of 0.5-0.8 is computed on different images and readers, so the statement that the model 'reaches the margins of human accuracy' is not directly supported. To support this claim, compute Cohen's kappa between the model and human grades on the same test images, or soften the claim to note that the kappa value lies within the range reported for human readers in the literature on a different dataset.
- [Section 4, Table 1] The paper reports results from a single 70/30 train-test split with no confidence intervals or repeated runs. Differences such as VGG-16 att0 multi-loss (64.3%) versus att1 multi-loss (63.2%) and early fusion (63%) may be within run-to-run noise, especially given the class imbalance in KL grades. Please provide bootstrap confidence intervals for accuracy and kappa, or report results over multiple random splits, to assess the stability of the headline numbers.
minor comments (4)
- [Section 3.3] There is a typo in the sentence 'Each attention branch makes separate predictions via a softmax to to generate their class probabilities': the word 'to' is duplicated.
- [Section 3.4] The preprocessing step says 'all the X-ray images are manually split in the middle, generating two vertical sections'; please clarify whether this manual split is considered a fixed preprocessing step rather than a localization step, to avoid confusion about the meaning of 'end-to-end'.
- [Table 2] The test loss values are reported for different loss formulations (e.g., joint classification+regression for Antony Joint Clsf & Reg versus pure classification for the proposed model); a brief note on the comparability of these loss values would help.
- [Figure 3 and Section 4] The text states that multi-loss training improves the shallower attention masks, but no quantitative mask-quality metric is provided; consider reporting a localization proxy or mask-overlap measure if available.
Circularity Check
No significant circularity: the paper reports an empirical architecture comparison with externally sourced baselines; the missing no-attention control is an experimental gap, not a circular reduction.
full rationale
The paper is an empirical evaluation of supervised CNN architectures; there is no derivation chain in which an output is constructed from its own inputs. The only fitted quantities are multi-loss branch weights and branch locations, selected by a 2D grid search on the validation loss ('validating the att0 and att1 loss weights between a range of 0.5 to 1 with a step size of 0.1, and using the validation loss as monitor'), which is standard hyperparameter selection rather than a prediction forced by the data that defines the target. The headline 64.3% test accuracy and kappa 0.63 are measured on a held-out portion of the 70/30 split, with Table 2 comparing against retrained external baselines (Antony et al.) and human reliability values explicitly qualified as not from the same test set. The attention module itself is taken from an external source (Mader, 2018), and no uniqueness or first-principles claim is imported from the authors' own prior work. The absence of a no-attention VGG-16 control on the same split is a real experimental limitation that weakens the attribution of the gain to attention, but this is a missing-control issue rather than a circular reduction: no equation, fitted parameter, or self-citation makes the reported prediction equivalent to its input by construction. Therefore no significant circularity is present.
Assumptions & free parameters
free parameters (2)
- Multi-loss branch weights (w0, w1) =
w0 = 1, w1 = 0.8
- Attention branch locations =
att0/att1 for VGG-16 and ResNet-50; att1/att2 for Antony models
assumptions (4)
- standard math Backpropagation and Adam optimization train all models as assumed.
- domain assumption OAI BU and MOST KL grades are reliable ground truth.
- domain assumption Global average pooling over attention-masked features preserves discriminative information.
- domain assumption Attention masks learned from class labels alone will focus on informative knee regions without explicit localization.
Cite this review
Pith. "Pith review of Assessing Knee OA Severity with CNN attention-based end-to-end architectures." pith.science (2026). https://pith.science/paper/7Z4HNI24
@misc{pith2026190808856,
author = {Pith},
title = {Pith review of: Assessing Knee OA Severity with CNN attention-based end-to-end architectures},
year = {2026},
howpublished = {\url{https://pith.science/paper/7Z4HNI24}},
note = {Machine review of arXiv:1908.08856}
}
read the original abstract
This work proposes a novel end-to-end convolutional neural network (CNN) architecture to automatically quantify the severity of knee osteoarthritis (OA) using X-Ray images, which incorporates trainable attention modules acting as unsupervised fine-grained detectors of the region of interest (ROI). The proposed attention modules can be applied at different levels and scales across any CNN pipeline helping the network to learn relevant attention patterns over the most informative parts of the image at different resolutions. We test the proposed attention mechanism on existing state-of-the-art CNN architectures as our base models, achieving promising results on the benchmark knee OA datasets from the osteoarthritis initiative (OAI) and multicenter osteoarthritis study (MOST). All code from our experiments will be publicly available on the github repository: https://github.com/marc-gorriz/KneeOA-CNNAttention
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Quantifying radiographic knee osteoarthritis severity using deep convolutional neural networks
Joseph Antony, Kevin McGuinness, Noel E O'Connor, and Kieran Moran. Quantifying radiographic knee osteoarthritis severity using deep convolutional neural networks. In Pattern Recognition (ICPR), 2016 23rd International Conference on, pages 1195--1200. IEEE, 2016
work page 2016
-
[2]
Joseph Antony, Kevin McGuinness, Kieran Moran, and Noel E O’Connor. Automatic detection of knee joints and quantification of knee osteoarthritis severity using convolutional neural networks. In International Conference on Machine Learning and Data Mining in Pattern Recognition, pages 376--390. Springer, 2017
work page 2017
-
[3]
Neural codes for image retrieval
Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. Neural codes for image retrieval. In European conference on computer vision, pages 584--599. Springer, 2014
work page 2014
-
[4]
Return of the devil in the details: Delving deep into convolutional nets
Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets. arXiv preprint arXiv:1405.3531, 2014
arXiv 2014
-
[5]
Locally-connected and convolutional neural networks for small footprint speaker recognition
Yu-hsin Chen, Ignacio Lopez-Moreno, Tara N Sainath, Mirk \'o Visontai, Raziel Alvarez, and Carolina Parada. Locally-connected and convolutional neural networks for small footprint speaker recognition. In Sixteenth Annual Conference of the International Speech Communication Association, 2015
work page 2015
-
[6]
A coefficient of agreement for nominal scales
Jacob Cohen. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20 0 (1): 0 37--46, 1960
1960
-
[7]
A unified architecture for natural language processing: Deep neural networks with multitask learning
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160--167. ACM, 2008
work page 2008
-
[8]
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In Proceedings of the IEEE International Conference on Computer Vision, pages 2650--2658, 2015
work page 2015
Show all 33 references
-
[9]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770--778, 2016
2016
-
[10]
Knee osteoarthritis prevalence, risk factors, pathogenesis and features: Part i
Behzad Heidari. Knee osteoarthritis prevalence, risk factors, pathogenesis and features: Part i. Caspian journal of internal medicine, 2 0 (2): 0 205, 2011
2011
-
[11]
Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers
Jui-Ting Huang, Jinyu Li, Dong Yu, Li Deng, and Yifan Gong. Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 7304--7308. IEEE, 2013
2013
-
[12]
Learn to pay attention
Saumya Jetley, Nicholas A Lord, Namhoon Lee, and Philip HS Torr. Learn to pay attention. arXiv preprint arXiv:1804.02391, 2018
2018 arXiv
-
[13]
Caffe: Convolutional architecture for fast feature embedding
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia, pages 675--678....
2014
-
[14]
Recognizing image style
Sergey Karayev, Matthew Trentacoste, Helen Han, Aseem Agarwala, Trevor Darrell, Aaron Hertzmann, and Holger Winnemoeller. Recognizing image style. arXiv preprint arXiv:1311.3715, 2013
2013 arXiv
-
[15]
Radiological assessment of osteo-arthrosis
JH Kellgren and JS Lawrence. Radiological assessment of osteo-arthrosis. Annals of the rheumatic diseases, 16 0 (4): 0 494, 1957
1957
-
[16]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[17]
Reliability and accuracy of cross-sectional radiographic assessment of severe knee osteoarthritis: role of training and experience
Kristina Klara, Jamie E Collins, Ellen Gurary, Scott A Elman, Derek S Stenquist, Elena Losina, and Jeffrey N Katz. Reliability and accuracy of cross-sectional radiographic assessment of severe knee osteoarthritis: role of training and experience. The Journal of rheumatology, p...
2016
-
[18]
Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory
Iasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In CVPR, volume 2, page 8, 2017
2017
-
[19]
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097--1105, 2012
2012
-
[20]
Face recognition: A convolutional neural-network approach
Steve Lawrence, C Lee Giles, Ah Chung Tsoi, and Andrew D Back. Face recognition: A convolutional neural-network approach. IEEE transactions on neural networks, 8 0 (1): 0 98--113, 1997
1997
-
[21]
The age-related changes in cartilage and osteoarthritis
YongPing Li, XiaoChun Wei, JingMing Zhou, and Lei Wei. The age-related changes in cartilage and osteoarthritis. BioMed research international, 2013, 2013
2013
-
[22]
Fully convolutional attention networks for fine-grained recognition
Xiao Liu, Tian Xia, Jiang Wang, Yi Yang, Feng Zhou, and Yuanqing Lin. Fully convolutional attention networks for fine-grained recognition. arXiv preprint arXiv:1603.06765, 2016
2016 arXiv
-
[23]
Attention on pretrained-vgg16 for bone age
Kevin Mader. Attention on pretrained-vgg16 for bone age. https://www.kaggle.com/kmader/attention-on-pretrained-vgg16-for-bone-age, 2018
2018
-
[24]
Fully automatic quantification of knee osteoarthritis severity on plain radiographs
H Oka, S Muraki, T Akune, A Mabuchi, T Suzuki, H Yoshida, S Yamamoto, K Nakamura, N Yoshimura, and H Kawaguchi. Fully automatic quantification of knee osteoarthritis severity on plain radiographs. Osteoarthritis and Cartilage, 16 0 (11): 0 1300--1306, 2008
2008
-
[25]
Imagenet large scale visual recognition challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115 0 (3): 0 211--252, 2015
2015
-
[26]
Wndchrm--an open source utility for biological image analysis
Lior Shamir, Nikita Orlov, D Mark Eckley, Tomasz Macura, Josiah Johnston, and Ilya G Goldberg. Wndchrm--an open source utility for biological image analysis. Source code for biology and medicine, 3 0 (1): 0 13, 2008
2008
-
[27]
Early detection of radiographic knee osteoarthritis using computer-aided analysis
Lior Shamir, Shari M Ling, William Scott, Marc Hochberg, Luigi Ferrucci, and Ilya G Goldberg. Early detection of radiographic knee osteoarthritis using computer-aided analysis. Osteoarthritis and Cartilage, 17 0 (10): 0 1307--1312, 2009 a
2009
-
[28]
Knee x-ray image analysis method for automated detection of osteoarthritis
Lior Shamir, Shari M Ling, William W Scott Jr, Angelo Bos, Nikita Orlov, Tomasz J Macura, D Mark Eckley, Luigi Ferrucci, and Ilya G Goldberg. Knee x-ray image analysis method for automated detection of osteoarthritis. IEEE Transactions on Biomedical Engineering, 56 0 (2): 0 40...
2009
-
[29]
Very deep convolutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014
2014 arXiv
-
[30]
Automatic knee osteoarthritis diagnosis from plain radiographs: a deep learning-based approach
Aleksei Tiulpin, J \'e r \^o me Thevenot, Esa Rahtu, Petri Lehenkari, and Simo Saarakkala. Automatic knee osteoarthritis diagnosis from plain radiographs: a deep learning-based approach. Scientific reports, 8 0 (1): 0 1727, 2018
2018
-
[31]
Computer-aided detection of breast masses: Four-view strategy for screening mammography
Jun Wei, Heang-Ping Chan, Chuan Zhou, Yi-Ta Wu, Berkman Sahiner, Lubomir M Hadjiiski, Marilyn A Roubidoux, and Mark A Helvie. Computer-aided detection of breast masses: Four-view strategy for screening mammography. Medical physics, 38 0 (4): 0 1867--1876, 2011
2011
-
[32]
The application of two-level attention models in deep convolutional neural network for fine-grained image classification
Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, and Zheng Zhang. The application of two-level attention models in deep convolutional neural network for fine-grained image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Re...
2015
-
[33]
Simple scoring system and artificial neural network for knee osteoarthritis risk prediction: a cross-sectional study
Tae Keun Yoo, Deok Won Kim, Soo Beom Choi, and Jee Soo Park. Simple scoring system and artificial neural network for knee osteoarthritis risk prediction: a cross-sectional study. PloS one, 11 0 (2): 0 e0148724, 2016
2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.