{"id":"6da706d2-399d-41aa-94ad-3ccd7a36de88","arxiv_id":"2412.07526","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An ensemble of ten fine-tuned convolutional networks, fused by a shallow neural network, reaches 72% test accuracy on five-class knee osteoarthritis grading from X-rays.","lead":"This paper combined ten pretrained image-classification networks into an ensemble to grade knee osteoarthritis severity from X-rays, reaching 72% accuracy on a five-level scale. The study is a benchmark comparison aimed at showing that learned ensembles beat single models for automated knee OA grading.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 0.72 accuracy may be inflated by an image-level split that allows the same participant's knees to appear in both training and test sets; a subject-exclusive split is required.","rationale":"The reader's weakest assumption identifies exactly the same issue: the image-level split risks same-patient leakage. I examined the full text for any mitigating statement and found none; Section 2.1 only gives counts and ratios. The proposed test directly checks whether the central claim survives a subject-exclusive split. Other concerns, such as the missing majority-class baseline or the lack of code, are real but less decisive: they affect interpretation and reproducibility, not the validity of the accuracy number as a generalization estimate. Because the concern is specific, testable, and fixable, the conditional verdict remains appropriate rather than a rejection. If the authors can provide a subject-exclusive split and the accuracy holds, the paper's central claim would be supported; if not, the headline claim would need to be revised. Hence I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":6083,"tokens_out":2823,"duration_ms":29587,"concrete_test":"Obtain the participant IDs for all 8,260 images (available from OAI metadata) and recompute the 7:1:2 split with a constraint that all images from one participant are assigned to the same fold. Retrain the best single model (DenseNet-161 with weighted sampling) and the KneeXNet ensemble on this subject-exclusive split and report test accuracy and per-class F1. Additionally, report how many participants in the original split appeared in both training and test sets. If subject-exclusive accuracy drops materially below 0.72, the reported result does not measure generalization to new patients.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 2.1, the authors state that the OAI dataset comprises 8,260 knee X-ray images from 4,796 participants and that they split the data into training, validation, and test sets in a 7:1:2 ratio, but they never state that all images from a single participant were kept in the same split. With roughly 1.7 images per participant on average, a random image-level split almost certainly places images from the same participant in both training and test sets, whether these are left/right knees or longitudinal follow-ups. KL grade is strongly correlated between the two knees of a person and over time, so models can exploit patient-specific features rather than learning to generalize to new people. The central claim is that KneeXNet achieves 0.72 test accuracy as a measure of generalization, so this leakage would make the reported number optimistic. The paper also releases no code and no split indices, so the reader cannot verify whether participant-level separation was enforced. This is the most load-bearing weakness because it directly undermines the validity of the headline result, while the other omissions (missing majority-class baseline, no confidence intervals) are secondary. If a subject-exclusive split is provided and accuracy remains approximately 0.72, the central claim would stand; if not, the claimed advantage over prior work would not be established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates ten ImageNet-pretrained CNN architectures (ResNet, VGG, MobileNet, DenseNet, EfficientNet, GoogLeNet) for five-class Kellgren-Lawrence knee osteoarthritis grading on the OAI dataset, first with cross-entropy loss and then with inverse-frequency weighted sampling. It then combines the outputs of the ten models by majority voting and by a two-layer fully connected network, reporting a best test accuracy of 0.72 ± 0.01 for the FCN-based ensemble, called KneeXNet. The paper also includes Smooth-GradCAM++ visualizations of the best single model, DenseNet-161.","tokens_in":6299,"tokens_out":2040,"duration_ms":19960,"significance":"If the reported results hold under a proper evaluation protocol, the contribution is a moderate, incremental improvement in knee OA classification accuracy over a known three-DenseNet ensemble (0.71) and over the best individual model (0.69–0.70), together with a systematic comparison of ten architectures. The manuscript does not provide code or split indices, and the evaluation protocol has a serious potential leakage issue that affects the credibility of the headline number. The explainability visualizations are a useful but minor addition.","major_comments":[{"comment":"The data split is described at the image level (8,260 images from 4,796 participants split 7:1:2), but the paper never states that all images from a single participant were assigned to the same split. Because the same participant contributes on average 1.72 images (left and right knees, and possibly follow-up visits), a random image-level split will almost certainly place images of the same participant in both the training and test sets. KL grade is strongly correlated within a participant, so the reported 0.72 test accuracy could reflect memorization of participant-specific features rather than generalization to new individuals. This is load-bearing for the central claim, and the paper must either confirm that an exclusive participant-level split was used or re-run the experiments with such a split and report the resulting accuracies.","section":"Section 2.1"},{"comment":"The Discussion states that the best ensemble 'fused the outputs of models trained with and without the weighted sampling strategy,' but no such combined ensemble is described in the experimental sections. Section 3.3 reports an ensemble of the ten Experiment 1 models (accuracy 0.72 ± 0.01), while Section 3.4 reports an ensemble of the ten Experiment 2 models (accuracy 0.70 ± 0.01). The paper must clarify which configuration constitutes KneeXNet and, if the Discussion refers to a fourth ensemble not described in the methods, provide the full experimental details.","section":"Section 3.3 vs. Section 4"},{"comment":"No majority-class or random-chance baseline is reported. Given the class imbalance shown in Fig. 1, a trivial classifier that always predicts the majority class may already achieve a substantial accuracy, and the 0.72 figure has little meaning without this reference. Reporting per-class accuracy alongside the F1 scores would also help interpret the 0.72 overall accuracy, which is dominated by the larger classes.","section":"Section 3.1 / Fig. 1"}],"minor_comments":[{"comment":"The two-layer fully connected network is said to be trained 'with the same training parameters as the baseline models,' but the baseline description includes batch size 28 and a learning-rate schedule; it is unclear whether the FCN uses the same batch size and schedule or just the same optimizer and number of epochs.","section":"Section 2.3"},{"comment":"Standard deviations are reported for the FCN-based ensembles but not for the majority-voting ensembles. Since the voting ensembles were run once, the reader cannot assess the variability of the reported 0.70 and 0.64 accuracies.","section":"Sections 3.3 and 3.4"},{"comment":"There is a typo: 'hypotetised' should be 'hypothesized'.","section":"Section 4"},{"comment":"Reference [1] cites a Mendeley dataset from 2018, while the text says the study used the Osteoarthritis Initiative (OAI) dataset. The relationship between the OAI and the cited Mendeley entry should be clarified, as the reader currently cannot identify the exact data source.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is more in line with a workshop or conference contribution than a full journal paper in its current form. The participant-level split issue is the key technical gate; if the authors can show subject-exclusive separation and consistent results, the claim would be acceptable as a modest empirical contribution. The internal inconsistency between the Discussion and the experimental sections about the ensemble composition also needs resolution before the paper can be judged."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, incremental benchmark paper whose headline number (0.72 accuracy) is only trustworthy if the split is patient-exclusive, and the paper never says it is.\n\nWhat's actually new: the specific ensemble of ten ImageNet-pretrained CNNs fused by a two-layer FCN on OAI KL grading, reporting 0.72 ± 0.01, about one point above the cited 3-DenseNet ensemble and two to three points above their best single model. The paper does useful housekeeping: systematic comparison of ten architectures, weighted sampling for class imbalance, majority-voting and learned-fusion comparison, Smooth-GradCAM++ visualizations, and per-class F1 scores that honestly show Class 1 (Doubtful) remains near 0.2. That's an honest empirical contribution, modest in scope.\n\nThe load-bearing problem is the data split. The OAI data has 8,260 images from 4,796 participants; the paper splits images 7:1:2 but never states that all images from one participant stay in the same split. With about 1.7 images per person, a random image-level split almost guarantees the same participant's knees appear in both training and test. KL grades are highly correlated within a person, so the model can memorize patient-specific features and the 0.72 number would not measure generalization to new people. The authors release no code and no split indices, so the reader can't check. This is not a minor omission; it directly determines whether the central claim stands.\n\nSecondary issues: no majority-class baseline, so 0.72 is hard to interpret against the class imbalance; several results (e.g., individual model accuracies in Fig. 2) lack standard deviations; and the ensemble strategy was chosen after looking at test results, with no correction for multiple comparisons. These are fixable and less damaging.\n\nOn balance, the paper deserves a serious referee, but the referee should require a subject-exclusive split (or proof that the split is already participant-disjoint), a majority-class baseline, and code or split indices. If the accuracy holds under that split, the result is a useful data point; if not, the claimed advantage over prior work is not established.","headline":"Solid incremental benchmark whose 0.72 accuracy depends on a patient-exclusive split the paper never confirms.","tokens_in":6868,"tokens_out":1694,"would_cite":false,"duration_ms":15889,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An ensemble of ten fine-tuned CNNs grades knee osteoarthritis with 72 percent accuracy.","keywords":["knee osteoarthritis","Kellgren-Lawrence grading","ensemble learning","deep learning","X-ray classification","class imbalance","weighted sampling","Smooth-GradCAM++"],"falsifier":"Re-run KneeXNet's same training protocol but split the data by participant, ensuring that no person's images appear in more than one of the training, validation, and test sets. If the test accuracy falls below about 0.70 or the ensemble no longer beats the best single model, the reported 0.72 does not measure generalization to new patients.","tokens_in":5858,"feed_emoji":"🦴","tokens_out":10184,"duration_ms":87823,"temperature":0.7,"pith_summary":"This paper is trying to establish that an automated system can grade knee osteoarthritis severity from X-rays more reliably than any single deep network, using a learned ensemble of ten standard convolutional networks. On a public knee X-ray dataset labeled with the five-grade Kellgren-Lawrence scale, the best individual model reaches 0.69 accuracy, and a weighted-sampling strategy pushes the best single model (DenseNet-161) to 0.70. The proposed ensemble, KneeXNet, feeds the ten models' output scores into a shallow two-layer neural network and reaches 0.72 test accuracy, beating majority voting and a previously reported three-DenseNet ensemble at 0.71. If this holds on new patients, the result matters because automated grading could reduce radiologist workload in settings where expert assessment is scarce.","feed_headline":"Ten-model ensemble grades knee osteoarthritis at 72 percent accuracy","feed_subtitle":"Fusing ten fine-tuned CNNs beats every single model on five-class Kellgren-Lawrence grading.","key_machinery":"The load-bearing mechanism is the learned fuser. Each of the ten component networks outputs five logits, these are concatenated into one 50-dimensional input vector per image, and a shallow two-layer fully connected network is trained with cross-entropy loss to combine them; majority voting over the same models is the weaker alternative it is compared against. Weighted sampling, which samples classes inversely proportional to their frequency, is the supporting device for imbalanced training data, and Smooth-GradCAM++ is the supporting device for explainability, producing heatmaps over the joint-space region.","core_discovery":"The authors report that KneeXNet, built by concatenating the five-class logits of ten fine-tuned CNNs (ResNet-18/34/50, VGG-16/19, MobileNet, DenseNet-121/161, EfficientNet, GoogLeNet) into a 50-dimensional vector and training a two-layer fully connected network on that vector, achieves a test accuracy of 0.72±0.01 on five-class Kellgren-Lawrence grading. Per-class F1 scores are 0.82, 0.23, 0.71, 0.84, and 0.88. This is two points above the best single model with weighted sampling and one point above the cited three-DenseNet-121 ensemble. Weighted sampling is shown to improve the F1 score of the minority Doubtful grade by up to 0.14 while leaving overall accuracy roughly unchanged, and Smooth-GradCAM++ visualizations indicate that the best model's attention concentrates on the joint-space region, consistent with radiographic evidence of osteoarthritis.","pith_inferences":["Editorial inference: A patient-level split, where all images from each participant stay in one partition, would establish whether the 0.72 accuracy transfers to people the model has not seen; this is the natural next validation for the approach.","Editorial inference: The paper's own observation that Kellgren-Lawrence grades have fuzzy boundaries suggests ordinal regression or a three-class collapse would likely be clinically more useful than five-class classification, a direction the authors mention but do not test.","Editorial inference: The same logit-concatenation fuser is task-agnostic, so the ensemble design could be applied directly to other ordinal medical imaging tasks with class imbalance, such as diabetic retinopathy or hip osteoarthritis grading."],"forward_implications":["Knee OA grading on this dataset can be automated to match or beat the best single-network results without a radiologist in the loop.","A learned fusion of ten diverse CNN architectures is more accurate than majority voting and than a single architecture trained with the same data.","Weighted sampling improves the per-class F1 score for the hardest Kellgren-Lawrence grade (Doubtful) even when it does not raise overall accuracy, so class-balance corrections should be judged per class.","Future knee OA grading systems on the same public dataset need to exceed the 0.72 KneeXNet result to claim an improvement over this ensemble approach."],"supporting_citations":[{"why":"Supplies the knee X-ray images with Kellgren-Lawrence grades that form the training, validation, and test sets.","marker":"[2]"},{"why":"Reports a 0.71 accuracy for a three-DenseNet-121 ensemble on the same dataset; this is the baseline KneeXNet claims to outperform.","marker":"[12]"},{"why":"Defines the residual network family used as component models in the ensemble.","marker":"[6]"},{"why":"Defines DenseNet-121 and DenseNet-161, the architectures of the best single model and of the cited comparison ensemble.","marker":"[8]"},{"why":"Provides the pretrained weights used to initialize all ten fine-tuned networks.","marker":"[17]"},{"why":"Supplies Smooth-GradCAM++, the visualization method used to locate decision-influencing regions.","marker":"[14]"},{"why":"Defines the five-grade Kellgren-Lawrence scale that serves as the classification target.","marker":"[9]"}],"fun_headline_variants":["KneeXNet: 10-model ensemble hits 72% on OA grading","10 CNN ensemble bests single models for knee OA at 72%","Ensemble of 10 CNNs scores 72% on knee OA severity","KneeXNet ensemble: ten models, 72% accuracy on OA grading","Ten CNN ensemble grades knee OA with 72% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the data split separated people, not just images: if the same person's left and right knee X-rays appear in both training and test sets, the reported 0.72 accuracy reflects image-level similarity rather than the model's ability to grade a new person's knee.","fun_headline_variants_meta":{"raw":{"variants":["KneeXNet: 10-model ensemble hits 72% on OA grading","10 CNN ensemble bests single models for knee OA at 72%","Ensemble of 10 CNNs scores 72% on knee OA severity","KneeXNet ensemble: ten models, 72% accuracy on OA grading","Ten CNN ensemble grades knee OA with 72% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000526,"raw_usage":{"total_tokens":2528,"prompt_tokens":920,"completion_tokens":1608,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":1510}},"tokens_in":536,"tokens_out":1608,"duration_ms":9948,"temperature":1.0,"reasoning_tokens":1510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:44:54.632033+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run KneeXNet's same training protocol but split the data by participant, ensuring that no person's images appear in more than one of the training, validation, and test sets. If the test accuracy falls below about 0.70 or the ensemble no longer beats the best single model, the reported 0.72 does not measure generalization to new patients.","supporting_citations":[{"cited_title":"Comput- erized Medical Imaging and Graphics 75, 84–92 (2019)","cited_arxiv_id":null,"evidence_quote":"Supplies the knee X-ray images with Kellgren-Lawrence grades that form the training, validation, and test sets."},{"cited_title":"In: Recent Trends in Analysis of Images, Social Networks and Texts","cited_arxiv_id":null,"evidence_quote":"Reports a 0.71 accuracy for a three-DenseNet-121 ensemble on the same dataset; this is the baseline KneeXNet claims to outperform."},{"cited_title":"Annals of the rheumatic diseases16(4), 494–502 (1957).https://doi.org/10.1136/ard","cited_arxiv_id":null,"evidence_quote":"Defines the five-grade Kellgren-Lawrence scale that serves as the classification target."}],"review_version":1}