{"id":"929daf1f-7d95-4e66-9038-8a5892a24671","arxiv_id":"1908.08840","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Automated knee osteoarthritis grading with jointly trained classification-regression CNNs reaches 64.6% accuracy and ordinal regression 64.3%, outperforming WNDCHRM but not the cited Siamese CNN baseline.","lead":"This study trains deep neural networks to find knee joints in X-ray images and to grade osteoarthritis severity. The best models reach about 64% accuracy on a five-level grading scale, better than older feature-based methods but below some previously published deep learning results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Omitted published Tiulpin baseline (66.7% OAI accuracy) directly challenges the 'outperform existing approaches' claim; without a common-benchmark comparison, the central superiority claim is unsubstantiated.","rationale":"The reader's weakest_assumption—that continuous-scale quantification relies on equidistant KL grades—is a valid conceptual critique, and the paper itself acknowledges KL categories are not equidistant (Section 5.2.5). However, I see it as secondary because all compared methods use the same 0–4 encoding, so the relative MSE comparisons remain meaningful as an ordinal metric; the issue primarily undermines the 'continuous scale' framing, not the algorithmic ranking. The more load-bearing threat to the central 'outperform existing approaches' claim is the omission of the Tiulpin et al. Siamese CNN, which the paper itself reports as achieving 66.7% accuracy on OAI (Section 2.2.2), higher than the proposed 64.6%. Since the abstract and conclusion make state-of-the-art claims without benchmarking against this published method, the main claim is not established. The 'on par with radiologic reliability readings' conclusion (Section 6) is also unsupported—the paper lists human-level accuracy as future work—but this is a separate overclaim; the baseline omission is more fundamental to the core technical claim. The proposed concrete test—a head-to-head comparison on a common split with repeated seeds—would settle whether the proposed method actually advances the state of the art. I would keep the CONDITIONAL verdict, adding the condition that the authors must run this comparison or soften the state-of-the-art claim.","tokens_in":50607,"tokens_out":9206,"duration_ms":91154,"concrete_test":"Run a head-to-head comparison on a common benchmark: obtain or reimplement the Tiulpin et al. Siamese CNN, train it on the same OAI (or OAI+MOST) training split used in this paper, and evaluate both models on the same test split; report multi-class accuracy and MSE, plus mean±std over at least five random initializations. If the proposed model does not exceed the re-evaluated Tiulpin accuracy on this common benchmark, the claim of outperforming existing approaches is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the jointly trained CNN and ordinal regression 'outperform existing approaches' is not supported by the paper's comparisons. Section 2.2.2 cites Tiulpin et al. [44], a deep Siamese CNN that achieves 66.7% average multi-class accuracy on the entire OAI dataset, which is higher than the proposed 64.6%. Yet Table 43 compares only against WNDCHRM and fine-tuned BVLC CaffeNet, and no error bars or repeated-seed statistics are reported. Because the abstract and conclusion make state-of-the-art claims, the omission of a published stronger baseline is a load-bearing gap: if Tiulpin's method also outperforms the proposed method on a common evaluation protocol, the claim of 'best automatic quantification' is false. The paper should either include this baseline in the comparison or explicitly limit the claim to the methods actually tested.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript, written in thesis-chapter style, proposes and evaluates convolutional-neural-network methods for automatic knee osteoarthritis severity grading from X-ray images. The contributions include an FCN for knee joint localization, CNN classifiers and regressors trained from scratch, a jointly trained multi-objective network combining cross-entropy and MSE losses, and an ordinal regression variant with fixed grade weights. Experiments use the public OAI and MOST datasets; the final comparison (Table 43) reports that the jointly trained CNN reaches 64.6% multi-class accuracy and the ordinal regression reaches MSE 0.480, outperforming the WNDCHRM baseline and a fine-tuned BVLC CaffeNet.","tokens_in":50804,"tokens_out":5951,"duration_ms":51935,"significance":"If the results are reproducible and the comparisons are made on a common benchmark, the paper would provide a useful demonstration that multi-objective training and ordinal regression improve automatic KL grading over handcrafted-feature baselines. Strengths include the systematic progression from handcrafted features to transfer learning to from-scratch training, the use of two public datasets, the explicit error analysis, and the description of an end-to-end pipeline. The central claim of outperforming existing approaches is not currently established, however, because a published baseline (Tiulpin et al. [44]) cited in the paper itself reports higher accuracy, and because no variance or significance information is provided.","major_comments":[{"comment":"The paper's claim to 'outperform existing approaches' (abstract and Section 6) is not supported by the comparisons in Table 43. Section 2.2.2 cites Tiulpin et al. [44] with 66.7% average multi-class accuracy on the entire OAI dataset, which is higher than the 64.6% reported here for the jointly trained CNN. Table 43 only includes WNDCHRM and a fine-tuned BVLC CaffeNet, and no explicit comparison to [44] is provided. Please either run the proposed method and the Tiulpin et al. method on the same evaluation protocol, or revise the claims to state superiority only over the methods actually tested.","section":"Section 2.2.2 / Table 43"},{"comment":"The ordinal regression results are reported inconsistently. The text (Section 5.2.6, Results) states that after rounding the ordinal output, classification accuracy is 61.8% and MSE is 0.504, while the classification branch itself reaches 64.3%. Table 43 lists 'Ordinal Regression 64.3% 0.480', which does not match either quantity as described. Clarify which outputs and rounding procedures the table entries refer to, and correct the numbers in the abstract and conclusion if needed.","section":"Section 5.2.6 / Table 43"},{"comment":"The key accuracy and MSE differences (e.g., 61.8% vs 64.6%, and regression MSE 0.574 vs 0.507) are reported for single training runs, with no confidence intervals, repeated-seed statistics, or significance tests. Without such information, the improvement from joint training and ordinal regression cannot be distinguished from random variation. Please provide means and standard deviations over multiple runs or another appropriate statistical assessment.","section":"Section 5.2.4/5.2.5 (Tables 39-43)"},{"comment":"The continuous-scale quantification claim depends on treating KL grades 0-4 as equidistant integer targets (fixed weights [0,1,2,3,4] in the ordinal regression and integer labels in MSE). The paper itself notes that KL grades are not equidistant, citing references [22,81,25,24,82]. The reported MSE therefore measures deviation on this arbitrary integer scale, not on a validated continuous severity scale. Please either validate the scale (e.g., against quantitative joint-space-width measurements or OARSI grades) or clearly limit the continuous-scale claim to a modeling choice.","section":"Section 5.2.5 Discussion / 5.2.6"},{"comment":"The proposed end-to-end diagnostic system is described as combining the FCN localization and the jointly trained CNN, but no experiments evaluate the full pipeline. The results in Table 43 appear to be obtained from cropped knee images, and it is not stated whether these crops come from automatic localization or manual annotation (Section 5.2.4 describes both but Table 43 does not specify). The paper should report end-to-end performance with automatic localization, or explicitly state that the system was not evaluated as a whole.","section":"Section 5.3"}],"minor_comments":[{"comment":"There are inconsistent network descriptions; for example, Table 33 lists conv4-1 with 128 kernels but an output shape of 96, and several architecture tables use inconsistent naming conventions (e.g., 'Conv2 1' vs 'conv2-1').","section":"Throughout (architectures)"},{"comment":"The manuscript uses 'WNDCHARM' and 'WNDCHRM' interchangeably (e.g., Section 5.1.1 vs Table 31); please standardize the spelling.","section":"Throughout"},{"comment":"There are typos such as 'purpotedly' and non-standard ligatures in 'off-the-shelf'; a careful proofread is needed.","section":"Section 2.2.1"},{"comment":"The opening of Section 5.2.5 refers to 'Section 5.4.5' instead of 'Section 5.2.5'.","section":"Section 5.2.5"},{"comment":"Figure 51 is captioned 'The CNN configuration for ordinal regression' but appears to be a plot of test accuracy over epochs; the caption and the figure content should be matched.","section":"Figure 51"}],"recommendation":"major_revision","confidential_remarks":"The omission of the Tiulpin et al. comparison is particularly serious given that the paper cites this work and then makes a state-of-the-art claim. The manuscript also reads as an unrevised thesis chapter rather than a journal article; a thorough rewriting is needed. That said, the empirical work is substantial and the identified issues appear addressable within a major revision, so rejection is not warranted at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is largely a thesis chapter that consolidates the authors' own earlier papers (ICPR 2016, MLDM 2017, MIDL 2019) into one pipeline: FCN-based knee localization, CNNs trained from scratch, joint classification+regression loss, and an ordinal regression variant. The experimental work is real and reproducible in principle: public OAI/MOST data, clear architectures, and a sensible lightweight design (~2.9M parameters). The best numbers (64.6% multi-class accuracy for the jointly trained net; 0.480 MSE for ordinal regression) are plausibly better than the WNDCHRM and fine-tuned BVLC CaffeNet baselines tested. Credit where due: the paper is honest about KL grades not being equidistant, and it shows a systematic effort in hyperparameter tuning and error analysis.\n\nThe soft spots are real but mostly fixable. The biggest is that the \"outperform existing approaches\" claim ignores Tiulpin et al. 2018, which the paper itself cites as achieving 66.7% accuracy on the full OAI dataset. That number beats the proposed 64.6%. No error bars, no repeated-seed statistics, and no comparison on a common protocol are given, so the superiority claim is unsubstantiated. The paper should either include that baseline or explicitly scope the claim to the methods actually tested. There are also internal inconsistencies: the text says ordinal regression rounding gives 61.8% accuracy / 0.504 MSE, while Table 43 reports 64.3% / 0.480. Minor, but confusing. The continuous-scale quantification rests on treating KL grades as equidistant [0,1,2,3,4], which the paper itself acknowledges is clinically false; the MSE numbers should be read as ordinal-metric, not true continuous severity. Finally, the conclusion's \"on par with radiologic reliability\" is not supported by any reader study.\n\nNone of this is fatal. The core engineering story -- joint loss and ordinal regression improve over plain classification/regression on these datasets -- holds up in the comparisons actually run. The paper is a useful consolidation for anyone working on automated KL grading, and the architecture details are worth having in the literature.\n\nMy recommendation: send it to peer review, but require the authors to add the Tiulpin baseline or change the claim, add basic uncertainty estimates, and resolve the rounding inconsistencies. I would not cite it as state-of-the-art, but I might cite the ordinal regression adaptation.","headline":"A competent consolidation of prior knee-OA deep learning work with useful practical comparisons, but the central state-of-the-art claim is weakened by an omitted cited baseline and missing error bars.","tokens_in":51349,"tokens_out":2074,"would_cite":false,"duration_ms":19979,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that convolutional neural networks can automate knee osteoarthritis grading from X-rays, with a jointly trained classifier and regressor reaching 64.6% multi-class accuracy and a mean-squared error of 0.480, on par with…","keywords":["knee osteoarthritis","Kellgren-Lawrence grading","convolutional neural networks","ordinal regression","multi-objective learning","X-ray image classification","fully convolutional networks","OAI dataset"],"falsifier":"Take a test set of OAI or MOST radiographs that also have OARSI sub-scores for joint space narrowing and osteophytes, and compare the ordinal-regression predictions against the measured sub-scores: if a model whose grade weights are proportional to measured joint space width achieves lower MSE than the fixed [0,1,2,3,4] model, the equidistant-spacing assumption behind the paper's continuous claim is false.","tokens_in":50398,"feed_emoji":"🦴","tokens_out":4723,"duration_ms":48378,"temperature":0.7,"pith_summary":"This paper argues that convolutional neural networks can take over the two jobs a radiologist does when grading knee osteoarthritis on X-rays: finding the knee joint and rating its severity on the Kellgren-Lawrence 0-4 scale. It claims that a lightweight CNN trained from scratch on public knee radiographs, optimizing a weighted mix of classification and regression losses, reaches 64.6% multi-class accuracy, and an ordinal-regression variant reaches 0.480 mean-squared error. These numbers beat both a handcrafted-feature classifier (WNDCHRM, 34.8% accuracy, MSE 2.112) and a fine-tuned large CNN (BVLC CaffeNet, 57.6%, MSE 0.836). The paper further claims the full pipeline, a fully convolutional network for localizing joints plus the trained CNN, performs on par with radiologic reliability readings, the accepted gold standard. If right, this means automatic KL grading on standard public datasets is no longer a proof of concept but a practical diagnostic support tool.","feed_headline":"Joint-trained CNN lifts automatic knee OA grading to 64.6%","feed_subtitle":"Ordinal regression cuts mean-squared error to 0.480 on public knee X-rays, matching expert reliability.","key_machinery":"The central mechanism is multi-objective convolutional learning: a single lightweight CNN of about 2.9 million parameters, built from blocks of cascaded 3x3 convolutions, batch normalization, and max pooling, ends in a shared fully connected layer that branches into a softmax head for the five KL classes and a linear head for a continuous severity value. The loss is a weighted sum of categorical cross-entropy and mean-squared error, with the ratio tuned to 0.5; cross-entropy preserves discrete grade information while MSE injects ordering and distance information between grades. The ordinal-regression variant replaces the regression branch with a dot product of the softmax probabilities against fixed weights [0,1,2,3,4], so the regression target is the expected grade under the class distribution, and this is the mechanism that produces the paper's lowest reported MSE.","core_discovery":"On the paper's own terms, the discovery is that feature learning with CNNs, rather than handcrafted image features, is the effective route to fine-grained knee OA quantification, and that the best quantification comes from training the network to predict KL grade as both a discrete class and a continuous value at once. The jointly trained CNN predicts a discrete grade through a softmax head and a continuous value through a linear regression head, with the loss a weighted sum of categorical cross-entropy and mean-squared error; it reaches 64.6% multi-class accuracy and MSE 0.507 on the regression head. A variant that treats the softmax head as a hidden layer and multiplies its probabilities by fixed weights [0,1,2,3,4], the ordinal-regression approach, reaches MSE 0.480 while keeping classification accuracy near 64.3%, which the paper reports as the best continuous-scale quantification. The claim is that this accuracy is on par with the reliability of expert radiologic readings, making the automatic system a credible replacement for manual KL scoring.","pith_inferences":["If KL grades are not equidistant, as the paper itself notes, the fixed weights [0,1,2,3,4] impose a spacing the data may not support; a natural extension is to learn those weights from joint-space-width or osteophyte measurements instead of fixing them.","The same weighted-loss recipe should transfer to other ordinal medical scales, such as Alzheimer's or cancer staging, where the disease is progressive but labels are discrete; the paper sketches this direction but does not test it.","Because the paper reports only MSE against integer KL labels, a stronger test of continuous quantification would compare predictions to OARSI sub-scores such as joint space narrowing and osteophytes; if those correlations are high, the continuous claim is independently supported.","The 64.6% accuracy may be partly dataset-specific: OAI and MOST share imaging protocols and reading standards, so a test on an external multi-center radiograph collection would reveal whether the learned features generalize."],"forward_implications":["An end-to-end pipeline using a fully convolutional network for knee localization plus the jointly trained CNN can produce KL grades and continuous severity scores directly from raw X-rays, removing manual ROI cropping.","The localizer alone detects knee joints at 100% accuracy for Jaccard index >= 0.5 over the combined OAI-MOST test set, so downstream grading need not depend on hand-drawn regions.","Training a CNN from scratch outperforms fine-tuning ImageNet-pretrained networks on this task (61.8% versus 57.6% accuracy), suggesting that domain-specific features learned on medical images outweigh transfer from natural images when enough data are available.","Joint training with two losses improves both class accuracy and regression MSE over training either head alone, making multi-objective convolutional learning a practical recipe for progressive-disease grading.","The system's accuracy is on par with radiologic reliability readings, implying automatic KL grading could serve as a second reader or initial screening tool in clinical workflows."],"supporting_citations":[{"why":"Provides the template-matching localization baseline and the early-detection framing this work starts from and then outperforms.","marker":"[3]"},{"why":"Supplies the WNDCHRM open-source classifier whose handcrafted features are the main baseline (34.8% accuracy).","marker":"[4]"},{"why":"Reports the deep Siamese CNN state-of-the-art accuracy of 66.7% on OAI that this paper's results are compared against.","marker":"[44]"},{"why":"Previous conference work that introduced regression loss for knee OA grading and the argument that continuous evaluation is more appropriate than categorical accuracy.","marker":"[45]"},{"why":"Provides the ordinal classification formulation, softmax probabilities multiplied by fixed weights, that the paper adapts for its ordinal-regression head.","marker":"[85]"},{"why":"Cited as evidence that KL categories are not equidistant, the premise motivating continuous-scale quantification.","marker":"[22]"},{"why":"Earlier FCN-based detection approach extended here for automatic localization of the knee joint region.","marker":"[58]"}],"fun_headline_variants":["CNN feature learning outperforms handcrafted for knee OA grading","Ordinal regression nets MSE 0.480 for knee OA severity from X-rays","Joint CNN classification-regression boosts continuous knee OA scoring","Deep learning matches expert-level reliability in knee OA grading","Multi-objective CNN loss refines KL grade quantification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim assumes KL grades 0 through 4 are equally spaced points on a continuous severity scale; the paper itself notes they are not equidistant, so the reported continuous MSE depends on that spacing.","fun_headline_variants_meta":{"raw":{"variants":["CNN feature learning outperforms handcrafted for knee OA grading","Ordinal regression nets MSE 0.480 for knee OA severity from X-rays","Joint CNN classification-regression boosts continuous knee OA scoring","Deep learning matches expert-level reliability in knee OA grading","Multi-objective CNN loss refines KL grade quantification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1627,"prompt_tokens":1043,"completion_tokens":584,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":500}},"tokens_in":659,"tokens_out":584,"duration_ms":6207,"temperature":1.0,"reasoning_tokens":500,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:28:11.474098+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a test set of OAI or MOST radiographs that also have OARSI sub-scores for joint space narrowing and osteophytes, and compare the ordinal-regression predictions against the measured sub-scores: if a model whose grade weights are proportional to measured joint space width achieves lower MSE than the fixed [0,1,2,3,4] model, the equidistant-spacing assumption behind the paper's continuous claim is false.","supporting_citations":[{"cited_title":"Scientific reports 8(1) (2018) 1727","cited_arxiv_id":null,"evidence_quote":"Reports the deep Siamese CNN state-of-the-art accuracy of 66.7% on OAI that this paper's results are compared against."},{"cited_title":"In: Pattern Recognition (ICPR), 2016 23rd International Conference on, IEEE (2016) 1195– 1200","cited_arxiv_id":null,"evidence_quote":"Previous conference work that introduced regression loss for knee OA grading and the argument that continuous evaluation is more appropriate than categorical accuracy."},{"cited_title":"Osteoarthritis and Cartilage 5(4) (1997) 241–250","cited_arxiv_id":null,"evidence_quote":"Cited as evidence that KL categories are not equidistant, the premise motivating continuous-scale quantification."},{"cited_title":"In: International Conference on Machine Learning and Data Mining in Pattern Recognition, Springer (2017) 376–390","cited_arxiv_id":null,"evidence_quote":"Earlier FCN-based detection approach extended here for automatic localization of the knee joint region."}],"review_version":1}