REVIEW 4 major objections 4 minor 33 references
The paper claims that a four-stage deep-learning pipeline, ARTSS, can score rheumatoid arthritis damage on hand X-rays with error below one Sharp point, and that its best model—a Vision Transformer—achieves a mean absolute error of 0.95 on
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
ARTSS, a deep learning pipeline for automated Sharp/van der Heijde rheumatoid arthritis scoring from hand X-rays, reports MAE 0.95 and 99% joint detection, but its key results table contains a mathematically impossible RMSE/MAE pair.
T0 review reviewed 2026-08-04 challenge →
load-bearing objection The padding-plus-masking idea for variable joint counts is genuinely worth knowing, but the paper's central performance table is internally impossible (RMSE below MAE) and the clinical claim about reader variability is never measured. the 4 major comments →
Automated Radiographic Total Sharp Score (ARTSS) in Rheumatoid Arthritis: A Solution to Reduce Inter-Intra Reader Variation and Enhancing Clinical Practice
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The paper's central claim is that a standardized, fully automated pipeline—ARTSS—can score structural damage in rheumatoid arthritis hand radiographs at a level useful for clinical practice. On 970 public hand X-rays scored by two radiologists, the pipeline aligns images, segments hands with U-Net, identifies 11 joints per hand with YOLOv7 (99% accuracy), and then regresses the average reader Total Sharp/van der Heijde score using a custom Vision Transformer. The authors report the ViT achieves a mean absolute error of 0.95, RMSE 0.93, and Huber loss 0.87 on 291 unseen subjects, with VGG16 and VGG19 the next-best alternatives. The framework is designed to include patients with deformed or di
What carries the argument
The central object is the Total Sharp/van der Heijde score (TSS), the standard radiographic measure of RA joint damage that combines erosion and joint-space narrowing across 16 hand joints. The mechanism that carries the argument is the variable-length joint-image sequence: detected joints are cropped, padded to a fixed maximum length, and masked during training so padded slots contribute no learning signal. That lets one fixed-size deep-learning regressor—a Vision Transformer built on image-patch attention—score patients whose hands have different numbers of visible joints, including the severe cases previous systems excluded. The Vision Transformer is what turns the padded joint sequence i
Load-bearing premise
The central claim rests on the reported test-set error metrics being computed and averaged correctly; the printed average RMSE of 0.93 is below the average MAE of 0.95 for the same model, which is impossible for any error distribution, so the headline numbers as printed are not self-consistent.
What would settle it
Re-run the Vision Transformer on the 291 test images from the public dataset and recompute MAE, RMSE, and Huber loss per fold. RMSE must be at least MAE for any error list, so a reproduced result with average RMSE below average MAE would show the published metrics cannot be genuine; matching the printed 0.95/0.93 pair would support them.
If this is right
- If the reported 0.95 mean absolute error reproduces in other datasets, ARTSS could act as an automated pre-screen or second reader for hand X-ray Sharp scoring.
- The pipeline would reduce dependence on expert radiologists for routine RA damage tracking, which matters in centers without specialty readers.
- Patients with severe erosion and disappearing joints could remain in automated scoring workflows instead of being excluded, broadening the population the model covers.
- Because the stages are modular, the same orientation-segmentation-detection-regression structure could be retrained for other joints or imaging protocols.
- A reproducible mean error below one Sharp point would put automated scoring near or within typical reader agreement, supporting the paper's claim of reduced variability.
Where Pith is reading between the lines
- The paper does not measure the two radiologists' disagreement on this dataset, so a direct comparison of ARTSS error to that measured inter-reader spread would test the headline claim of reduced variability.
- The padding-and-masking trick should transfer to other anatomical scoring tasks where landmarks disappear with disease severity, such as RA foot joints or osteoarthritic knees.
- A per-joint error analysis, rather than whole-hand TSS error, would show whether the model's remaining mistakes concentrate in eroded or vanished joints, indicating where human review still adds value.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents ARTSS, a four-stage deep-learning pipeline for automated Total Sharp/van der Heijde Score (TSS) assessment from hand X-rays: (I) orientation and preprocessing, (II) U-Net hand segmentation, (III) YOLOv7 joint identification, and (IV) TSS regression with several CNN/transformer backbones. The model is trained on 970 images using the average of two radiologists' TSS scores as ground truth, with 3-fold cross-validation and a 291-subject held-out test set. The authors report that a Vision Transformer achieves the best performance (Huber loss 0.87, MAE 0.95, RMSE 0.93). The paper's core clinical claim is that ARTSS reduces inter- and intra-reader variability and improves clinical practice.
Significance. If the reported accuracy is accurate and the pipeline is reproducible, automated TSS scoring would be valuable for RA monitoring, especially in resource-limited settings. The paper addresses a real clinical need, uses a relatively large public dataset, and introduces a plausible strategy for variable-length joint sequences. However, the quantitative claims are undermined by a mathematical impossibility in the reported metrics and the central clinical claim is asserted without measurement. The authors are transparent about some limitations (single center, hand-only images), but the internal metric inconsistency is a more fundamental issue. No code or evaluation scripts are provided, only a data link, so the reader cannot verify the reported numbers.
major comments (4)
- [Table 1 (ViT rows)] For any error vector, RMSE is always at least MAE. Table 1 reports ViT fold-1 RMSE=0.89 with MAE=0.99, and average RMSE=0.93 below average MAE=0.95. These values are mathematically impossible, indicating a bug in metric computation or inconsistent test sets. Since the abstract and discussion highlight the ViT MAE and Huber loss, the reliability of all metrics in Table 1 is now in question. Recompute all metrics and verify with code before resubmission.
- [Abstract; Discussion (first paragraph)] The central claim that ARTSS 'reduces inter- and intra-reader variability' is never measured. No ICC, kappa, per-reader difference, or comparison between model error and the two-reader disagreement is reported. Using the average of two radiologists as the training target does not demonstrate a variability reduction. The existing two independent readings should be used to compute inter-reader agreement (e.g., MAE or ICC) and compare it against the model's error; otherwise the clinical claim remains unsupported.
- [Methods (TSS target); Table 1] The manuscript never states the range or normalization of the TSS target. The hand-only TSS can reach hundreds of points, so an MAE of 0.95 is uninterpretable without knowing the scale. If predictions are on a normalized scale, the transformation must be specified; if raw points, the near-perfect MAE is not contextualized against human reader variability. This is required to evaluate the clinical relevance of the reported errors.
- [Results (Joint identification); Table 2] The '99% accuracy' for joint identification is unsubstantiated. The Results section mentions Figure 4 and MAP but reports no numeric value for accuracy or MAP. Table 2 lists 'This Study ... 99%' but no supporting result or metric definition appears in the text. For object detection, accuracy is ambiguous; the authors should report MAP and recall and define what 'accuracy' means.
minor comments (4)
- [Throughout] Typos: 'Total Sharo scoring' (Table 1 caption), 'Statical analysis' (Methods header), 'KED curves' (Figure 2 caption should be KDE), and duplicate 'proximal interphalangeal (PI), proximal interphalangeal (PIP)' in Joint Identification.
- [Methods (Dataset)] Dataset counts are inconsistent: 970 images are described, but later '1080 images' are used for joint annotation and '582 images' were processed to generate masks with '382 ground truth masks'. Reconcile these numbers.
- [Methods (Cross-validation)] The relation between the 3-fold cross-validation (452 training, 227 validation per fold) and the 291-subject external test is unclear. Clarify whether the 291 are excluded from all folds and how the folds are constructed.
- [Data and Code Availability] Only a data link is provided; no evaluation code is available. Given the reported metric inconsistencies, releasing the metric-computation script and predictions would substantially increase confidence.
Circularity Check
No significant circularity: TSS predictions are genuine held-out outputs; the only self-citation is minor and non-load-bearing, while the unsupported reader-variability claim is a missing-baseline issue, not circularity.
full rationale
The paper's central quantitative chain is a supervised pipeline: preprocess -> segment -> detect joints -> regress TSS. The TSS regression target is the average of two radiologists' scores, and the test metrics in Table 1 are computed on 291 unseen subjects, so the ViT MAE (0.95) and Huber loss (0.87) are genuine held-out predictions rather than fitted values. Nothing in the method defines the TSS prediction in terms of the reported test values; the models are trained on fold splits and evaluated on held-out data. Therefore the empirical core is self-contained and not circular. The abstract/discussion claim that ARTSS 'reduces inter- and intra-reader variability' is not derived from any measured reader-variability baseline; the only operationalization is that the ground truth itself is the two-reader average, so any smoothing is in the labels, not demonstrated by the system. This is an unsupported clinical claim / missing-baseline issue, not a circular step under the strict definitions. The only author self-citation is [33], a radiomics reproducibility paper cited in a forward-looking sentence about integrating radiomics; it is not load-bearing for any method or result. Separately, Table 1 contains an internal inconsistency (ViT fold-1 RMSE 0.89 < MAE 0.99 and average RMSE 0.93 < average MAE 0.95, impossible because RMSE>=MAE), which is a correctness/evaluation-pipeline concern, not a circularity. Under the required evidence standard, no step can be shown to reduce by construction to its input, so the score is low.
Axiom & Free-Parameter Ledger
free parameters (5)
- Huber loss delta (sigma)
- Augmentation hyperparameters =
rotation 10 deg, shifts 0.2, brightness 0.7-1.2, horizontal flip
- Padding length
- Joint detection thresholds
- Reorientation model outputs
axioms (4)
- domain assumption The average of two radiologists' TSS readings is an accurate, low-noise ground truth for joint damage.
- domain assumption The 382 manually selected segmentation masks correctly delineate the hand region.
- domain assumption The 291 external test subjects are patient-level independent from the training folds and drawn from the same distribution.
- domain assumption The van der Heijde scoring rules were applied correctly by both readers, including to joints that have disappeared.
Cite this review
Pith. "Pith review of Automated Radiographic Total Sharp Score (ARTSS) in Rheumatoid Arthritis: A Solution to Reduce Inter-Intra Reader Variation and Enhancing Clinical Practice." pith.science (2026). https://pith.science/paper/IVKJ4AR3
@misc{pith2026250906854,
author = {Pith},
title = {Pith review of: Automated Radiographic Total Sharp Score (ARTSS) in Rheumatoid Arthritis: A Solution to Reduce Inter-Intra Reader Variation and Enhancing Clinical Practice},
year = {2026},
howpublished = {\url{https://pith.science/paper/IVKJ4AR3}},
note = {Machine review of arXiv:2509.06854}
}
read the original abstract
Assessing the severity of rheumatoid arthritis (RA) using the Total Sharp/Van Der Heijde Score (TSS) is crucial, but manual scoring is often time-consuming and subjective. This study introduces an Automated Radiographic Sharp Scoring (ARTSS) framework that leverages deep learning to analyze full-hand X-ray images, aiming to reduce inter- and intra-observer variability. The research uniquely accommodates patients with joint disappearance and variable-length image sequences. We developed ARTSS using data from 970 patients, structured into four stages: I) Image pre-processing and re-orientation using ResNet50, II) Hand segmentation using UNet.3, III) Joint identification using YOLOv7, and IV) TSS prediction using models such as VGG16, VGG19, ResNet50, DenseNet201, EfficientNetB0, and Vision Transformer (ViT). We evaluated model performance with Intersection over Union (IoU), Mean Average Precision (MAP), mean absolute error (MAE), Root Mean Squared Error (RMSE), and Huber loss. The average TSS from two radiologists was used as the ground truth. Model training employed 3-fold cross-validation, with each fold consisting of 452 training and 227 validation samples, and external testing included 291 unseen subjects. Our joint identification model achieved 99% accuracy. The best-performing model, ViT, achieved a notably low Huber loss of 0.87 for TSS prediction. Our results demonstrate the potential of deep learning to automate RA scoring, which can significantly enhance clinical practice. Our approach addresses the challenge of joint disappearance and variable joint numbers, offers timesaving benefits, reduces inter- and intra-reader variability, improves radiologist accuracy, and aids rheumatologists in making more informed decisions.
Figures
Reference graph
Works this paper leans on
-
[1]
Rheumatoid Arthritis: A Brief Overview of the Treatment,
J. Bullock et al., "Rheumatoid Arthritis: A Brief Overview of the Treatment," (in eng), no. 1423-0151 (Electronic)
-
[2]
A. Strait et al., "Demographic Characteristics of Participants in Rheumatoid Arthritis Randomized Clinical Trials: A Systematic Review," (in eng), no. 2574-3805 (Electronic)
-
[3]
Inching closer to precision treatment for rheumatoid arthritis,
L. T. Donlin, "Inching closer to precision treatment for rheumatoid arthritis," Nature Medicine, vol. 28, no. 6, pp. 1129- 1131, 2022/06/01 2022, doi: 10.1038/s41591-022-01857-5
-
[4]
R. A.-O. Meehan, I. A. Amigues, and V . Knight, "Precision Medicine for Rheumatoid Arthritis: The Right Drug for the Right Patient-Companion Diagnostics. LID - 10.3390/diagnostics11081362 [doi] LID - 1362," (in eng), no. 2075-4418 (Print)
-
[5]
O. Alekseeva, D. Kudinsky, A. Smirnov, A. V olkov, and E. Nasonov, "AB1378 RELATION BETWEEN SHARP SCORE AND RADIOLOGICAL PROGRESSION IN PATIENTS WITH RHEUMATOID ARTHRITIS: A PROSPECTIVE OBSERV ATION," Annals of the Rheumatic Diseases, vol. 81, no. Suppl 1, pp. 1794-1794, 2022, doi: 10.1136/annrheumdis-2022-eular.5081
-
[6]
D. A.-O. X. van der Heijde, D. D. Gladman, A. Kavanaugh, and P. J. Mease, "Assessing structural damage progression in psoriatic arthritis and its role as an outcome in research," (in eng), no. 1478 -6362 (Electronic)
-
[7]
Interpreting radiographic data in rheumatoid arthritis,
P. A. Ory, "Interpreting radiographic data in rheumatoid arthritis," (in eng), no. 0003-4967 (Print)
-
[8]
Radiographic scoring methods as outcome measures in rheumatoid arthritis: properties and advantages,
S. Boini and F. Guillemin, "Radiographic scoring methods as outcome measures in rheumatoid arthritis: properties and advantages," (in eng), no. 0003-4967 (Print)
-
[9]
Artificial Intelligence and Deep Learning for Rheumatologists,
C. McMaster et al., "Artificial Intelligence and Deep Learning for Rheumatologists," Arthritis & Rheumatology, vol. 74, no. 12, pp. 1893-1905, 2022, doi: https://doi.org/10.1002/art.42296
-
[10]
R. K. Ahalya, S. A.-O. Umapathy, P. T. Krishnan, and A. N. Joseph Raj, "Automated evaluation of rheumatoid arthritis from hand radiographs using Machine Learning and deep learning techniques," (in eng), no. 2041 -3033 (Electronic)
work page 2041
-
[11]
S. A.-O. Honda, K. Yano, E. A.-O. X. Tanaka, K. Ikari, and M. A.-O. Harigai, "Development of a scoring model for the Sharp/van der Heijde score using convolutional neural networks and its clinical application," (in eng), no. 1462 -0332 (Electronic)
-
[12]
K. Izumi et al., "Ensemble detection of hand joint ankylosis and subluxation in radiographic images using deep neural networks," Scientific Reports, vol. 14, no. 1, p. 7696, 2024/04/02 2024, doi: 10.1038/s41598-024-58242-0
-
[13]
YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real- time object detectors,
C.-Y . Wang, A. Bochkovskiy, and H.-Y . M. Liao, "YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real- time object detectors," arXiv preprint arXiv:2207.02696, 2022
Pith/arXiv arXiv 2022
-
[14]
Very deep convolutional networks for large-scale image recognition,
K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014
Pith/arXiv arXiv 2014
-
[15]
Efficientnet: Rethinking model scaling for convolutional neural networks,
M. Tan and Q. Le, "Efficientnet: Rethinking model scaling for convolutional neural networks," in International conference on machine learning, 2019: PMLR, pp. 6105-6114
work page 2019
-
[16]
Deep residual learning for image recognition,
K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778
2016
-
[17]
Densely connected convolutional networks,
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700-4708
2017
-
[18]
An image is worth 16x16 words: Transformers for image recognition at scale,
A. Dosovitskiy et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010.11929, 2020
Pith/arXiv arXiv 2010
-
[19]
An Efficient CNN for Hand X-Ray Overall Scoring of Rheumatoid Arthritis,
Z. Wang, J. Liu, Z. Gu, and C. Li, "An Efficient CNN for Hand X-Ray Overall Scoring of Rheumatoid Arthritis," Complexity, vol. 2022, p. 5485606, 2022/02/23 2022, doi: 10.1155/2022/5485606
-
[20]
You only look once: Unified, real-time object detection,
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 779-788
2016
-
[21]
A survey on Image Data Augmentation for Deep Learning,
C. Shorten and T. M. Khoshgoftaar, "A survey on Image Data Augmentation for Deep Learning," Journal of Big Data, vol. 6, no. 1, p. 60, 2019/07/06 2019, doi: 10.1186/s40537-019-0197-0
-
[22]
Boundary IoU: Improving Object-Centric Image Segmentation Evaluation,
B. Cheng, R. Girshick, P. Dollár, A. C. Berg, and A. Kirillov, "Boundary IoU: Improving Object-Centric Image Segmentation Evaluation," p. arXiv:2103.16562doi: 10.48550/arXiv.2103.16562
-
[23]
End-to-end training of object class detectors for mean average precision
P. Henderson and V . Ferrari, "End-to-end training of object class detectors for mean average precision," p. arXiv:1607.03476doi: 10.48550/arXiv.1607.03476
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1607.03476
-
[24]
A statistical learning assessment of Huber regression,
Y . Feng and Q. Wu, "A statistical learning assessment of Huber regression," Journal of Approximation Theory, vol. 273, p. 105660, 2022/01/01/ 2022, doi: https://doi.org/10.1016/j.jat.2021.105660
-
[25]
K. A.-O. Radke et al., "Adaptive IoU Thresholding for Improving Small Object Detection: A Proof-of-Concept Study of Hand Erosions Classification of Patients with Rheumatic Arthritis on X-ray Images. LID - 10.3390/diagnostics13010104 [doi] LID - 104," (in eng), no. 2075-4418 (Print)
-
[26]
I. Dimitrovsky and L. Ericson, A multistage deep learning method for scoring radiographic hand and foot joint damage in rheumatoid arthritis. 2020
work page 2020
-
[27]
S. Honda, K. Yano, E. Tanaka, K. Ikari, and M. Harigai, "Development of a scoring model for the Sharp/van der Heijde score using convolutional neural networks and its clinical application," (in eng), Rheumatology (Oxford), Oct 11 2022, doi: 10.1093/rheumatology/keac586
-
[28]
Rheumatoid Arthritis: Automated Scoring of Radiographic Joint Damage
Y . M. Tan, R. Q. H. Chong, and C. A. Hargreaves, "Rheumatoid arthritis: Automated scoring of radiographic joint damage," arXiv preprint arXiv:2110.08812, 2021
work page internal anchor Pith review Pith/arXiv arXiv 2021
-
[29]
H. A.-O. Wang et al., "Deep Learning-Based Computer-Aided Diagnosis of Rheumatoid Arthritis with Hand X-ray Images Conforming to Modified Total Sharp/van der Heijde Score. LID - 10.3390/biomedicines10061355 [doi] LID - 1355," (in eng), no. 2227-9059 (Print)
-
[30]
K. Üreten and H. H. Maraş, "Automated Classification of Rheumatoid Arthritis, Osteoarthritis, and Normal Hand Radiographs with Deep Learning Methods," (in eng), J Digit Imaging, vol. 35, no. 2, pp. 193-199, Apr 2022, doi: 10.1007/s10278-021-00564-w
-
[31]
Deep learning-based joint detection in Rheumatoid arthritis hand radiographs,
D. L. Fung et al., "Deep learning-based joint detection in Rheumatoid arthritis hand radiographs," (in eng), AMIA Jt Summits Transl Sci Proc, vol. 2023, pp. 206-215, 2023
work page 2023
-
[32]
Panomics for Precision Medicine,
C. Sandhu, A. Qureshi, and A. Emili, "Panomics for Precision Medicine," (in eng), no. 1471-499X (Electronic)
-
[33]
H. Moradmand, S. M. R. Aghamiri, and R. Ghaderi, "Impact of image preprocessing methods on reproducibility of radiomic features in multimodal magnetic resonance imaging in glioblastoma," Journal of Applied Clinical Medical Physics, vol. 21, no. 1, pp. 179-190, 2020, doi: https://doi.org/10.1002/acm2.12795
This paper was first reviewed by deepseek-v4-flash on August 4, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.