{"id":"78e3d8c3-6d5a-491d-9ec5-5f21cf564a01","arxiv_id":"1908.08873","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Statistical models built from patient questionnaire data predict knee osteoarthritis severity (KL grade) with RMSE around 0.94 to 0.98, close to the 0.77 RMSE of a convolutional neural network trained on X-ray images.","lead":"This paper compares computer models that predict knee osteoarthritis severity: one set uses patient questionnaire data, another uses X-ray images. The questionnaire-based models perform nearly as well as the image-based model on the same patients, suggesting that simple clinical information might substitute for imaging in some settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Statistical models include X-ray-derived baseline OA variables (P01LXRKOA/P01RXRKOA), so the questionnaire-vs-imaging comparison may not be a fair test.","rationale":"The reader's weakest assumption correctly identifies the load-bearing issue: including P01LXRKOA and P01RXRKOA in the statistical models makes the central questionnaire-versus-imaging comparison questionable. The manuscript itself describes these variables as reporting baseline radiographic OA status, so this is not an external accusation but a direct reading of the paper's own text. My stress-test pass confirms the concern is concrete and testable. If the codebook shows these variables are pure questionnaire items that do not encode any X-ray-based information, the concern would not land and the paper's comparison would be more defensible; however, the current description is sufficient to cast serious doubt on the central claim. The reader's other criticisms (missing uncertainty quantification, ordinal outcome treated as continuous, patient exclusion) are valid but secondary. I therefore agree with the REJECT verdict and would keep it unchanged. The recommended concrete check is essential: without it, the central finding cannot be trusted as stated.","tokens_in":11102,"tokens_out":3838,"duration_ms":39626,"concrete_test":"Use the OAI codebook or data dictionary to check whether P01LXRKOA and P01RXRKOA are derived from radiographic readings or from patient self-report of a prior X-ray diagnosis. Then refit the Elastic Net, Random Forest, and linear mixed models on the same training/validation split with these two variables removed, and recompute the overall validation RMSE. If the RMSE increases materially or the statistical models no longer appear comparable to the CNN, the paper's main claim is unsupported as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that patient questionnaire data can predict KOA severity about as well as X-ray-based CNNs. For that comparison to be meaningful, the statistical models must be trained on non-imaging patient data. The Discussion, however, identifies P01LXRKOA and P01RXRKOA as the strongest predictors and describes them as 'knee baseline radiographic OA status' for the left and right knee. These variable names literally encode X-ray knee OA, and the manuscript's own description indicates they carry radiographic information. If these variables are X-ray-derived, or even if they are patient-reported answers to a question such as 'has a doctor told you an X-ray shows knee osteoarthritis?', then the Elastic Net, Random Forest, and LMM models are not using questionnaire data alone. They are using information that already reflects radiographic assessment, which makes the comparison between 'patient data' and 'X-ray image' models partially circular. The RMSE values in Table 3 would then describe a comparison between an imaging-informed statistical model and an imaging-based CNN, not the low-cost questionnaire substitute claimed in the Abstract and Discussion. This is the single most load-bearing concern because it directly undermines the paper's primary contribution. Secondary issues such as reporting RMSE without uncertainty and treating an ordinal outcome as continuous would also need attention, but they would not by themselves invalidate the comparison if the predictor set were clean.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Using baseline data from the Osteoarthritis Initiative (OAI), the authors train Elastic Net, Random Forest, and linear mixed-effect models on participant questionnaire/clinical variables to predict Kellgren-Lawrence (KL) severity grades, and compare their RMSEs with a CNN trained on knee X-ray images. They report RMSEs of 0.973 (EN), 0.943 (RF), 0.978 (LMM), and 0.770 (CNN), and conclude that statistical models using patient questionnaire data achieve accuracy comparable to X-ray-image models. They also highlight the strongest predictive variables as potential monitoring targets. The paper's central claim is that a low-cost, non-imaging patient assessment can substitute for radiographic severity grading, while also yielding interpretable predictors.","tokens_in":11376,"tokens_out":6848,"duration_ms":72718,"significance":"If the comparison were valid, this would be a practically useful and novel result: questionnaire-based prediction of KL grade could support screening and monitoring without immediate X-ray imaging. The paper has several strengths: it uses a large public dataset, applies a deliberate training/validation split, uses LMM to account for the hierarchical structure of knees within patients, and attempts to identify interpretable predictors. However, the current analysis does not establish the central claim because the statistical models may include variables carrying radiographic information, the CNN benchmark is quoted from prior work by the same group without clear evidence of a matched evaluation protocol, and the RMSE comparisons lack uncertainty quantification.","major_comments":[{"comment":"The paper's central claim depends on the statistical models using only patient questionnaire data, but the strongest predictors are described as 'knee baseline radiographic OA status for the right or left knee' (P01LXRKOA, P01RXRKOA). These names and the Discussion wording indicate that the models receive information about X-ray-derived OA status. If P01LXRKOA/P01RXRKOA are imaging-derived, or even if they are patient reports of a prior radiographic diagnosis, the statistical models are not using questionnaire/symptom data alone, and the comparison with the X-ray CNN is partly circular. Please define these variables precisely from the OAI codebook, state whether they encode radiographic assessment, and if they do, refit the EN, RF, and LMM models without them and report the resulting RMSEs. This is necessary to support the Abstract and Discussion claims about questionnaire-only prediction.","section":"Discussion; Results (Table 3)"},{"comment":"The CNN RMSE of 0.770 is quoted from the authors' earlier publications (refs 26 and 27) rather than computed within the same evaluation pipeline described here. The text states that the same validation patients were used, but it does not demonstrate that the CNN was retrained on the same training set of 2,105 patients used for the statistical models, nor that its hyperparameters were selected without using the validation set. Without a fully matched training and validation protocol, Table 3 may compare models evaluated under different conditions, so the conclusion that the models are comparable is not yet supported. Please clarify whether the CNN result was recomputed for this paper, and report the exact split and model selection procedure used.","section":"Methods (Convolution Neural Network); Results (Table 3)"},{"comment":"All RMSE values are reported as single point estimates with no confidence intervals, bootstrap estimates, or significance tests. The observed differences between the CNN and the statistical models (e.g., 0.770 vs 0.943) could be within sampling variability, especially given the small numbers of patients at KL grades 3 and 4 and the large per-level RMSE differences in the table. Please provide uncertainty intervals (e.g., bootstrap over subjects, respecting the two-knee clustering) and, if appropriate, a paired comparison on the common validation set, so that the claim of 'comparable' accuracy is quantitatively supported.","section":"Results (Table 3)"}],"minor_comments":[{"comment":"The notation in Eq. (2) is internally inconsistent: if i indexes patients and j indexes knees within a patient, then m should be the number of patients and n_i should be 2 for both knees; the text's statement that 'm = 2 representing left and right knee' is incorrect and should be corrected.","section":"Methods (Linear Mixed Effect Model), Eq. (2)"},{"comment":"The threshold for dropping categorical variables is described as 'very low frequency ... less than 5% into one category in a binary variable'; please report the exact criterion and, ideally, a sensitivity analysis showing that the main results are stable to reasonable changes in this threshold and in the 15% missingness threshold.","section":"Methods (Data pre-processing)"},{"comment":"The supplementary table listing candidate variables is not available in the posted version; because the central concern about P01LXRKOA/P01RXRKOA depends on their definition, the OAI codebook entries and the full candidate variable list should be included as supplementary material, along with any code or data-processing scripts used to create the analytic dataset.","section":"Data and Supplement"},{"comment":"There are several minor language issues, such as 'The red points represents RMSE' (should be 'represent') and 'patients share almost similar distribution' (should be 'have similar distributions'); these do not affect the science but should be corrected in revision.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The most important issue is to settle what P01LXRKOA and P01RXRKOA actually measure. If they are X-ray-derived baseline OA status, the current framing of the paper is not supportable; however, dropping these variables and refitting is a tractable revision, so I recommend major revision rather than rejection at this stage. The editor may also wish to ask the authors to provide outside confirmation or code/data release for the CNN baseline, since the comparison target is the authors' own previous work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version: the headline claim doesn't hold as stated. The statistical models that are supposed to be 'questionnaire-only' include P01LXRKOA and P01RXRKOA, which the authors themselves describe as 'baseline radiographic OA status' for each knee. Whatever those variables are precisely, they carry information from a prior X-ray assessment, so the RMSE comparison of ~0.97/0.94 against the CNN's 0.77 is not a fair test of text versus image. That's the load-bearing problem.\n\nWhat's actually new and useful: this is the first paper I know of that tries to predict KL grade from patient-reported signs/symptoms/medication on the same OAI validation set as a CNN, and it's a sensible modeling exercise. The use of a linear mixed model to account for the correlation between knees is appropriate and the cross-validation sensitivity check is a good instinct. Identifying which symptoms and functional limitations track severity is clinically interesting even if the quantitative claim is inflated.\n\nThe soft spots beyond the circularity: RMSEs are point estimates with no intervals or tests; the ordinal outcome is treated as continuous, justified only by saying ordinal regression performed poorly; and the CNN result is taken from the same group's earlier papers (refs 26,27), not retrained here. Those are secondary but would matter in revision.\n\nI don't think the paper is a waste of time. The analysis is transparent, the data is public, and the question is genuinely relevant. But as submitted, the abstract and discussion overstate what has been shown. A revision that removes or reclassifies the imaging-derived variables (or reports results both with and without them), adds confidence intervals for RMSE, and either justifies the continuous treatment or uses ordinal methods would be worth another look. I'd send it to peer review, mainly because the central idea is worth scrutinizing and the authors seem capable of fixing it.","headline":"The paper's central comparison is compromised because the statistical models include baseline radiographic OA variables (P01LXRKOA/P01RXRKOA), so the 'questionnaire vs X-ray' framing is not clean; still, the modeling exercise and mixed-effects approach give it some value.","tokens_in":11905,"tokens_out":3192,"would_cite":false,"duration_ms":29515,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that patient questionnaire data—symptoms, medication use, and basic characteristics—can predict knee osteoarthritis severity nearly as well as a convolutional neural network trained on the knee X-rays themselves.","keywords":["knee osteoarthritis","Kellgren-Lawrence grade","patient questionnaire data","X-ray imaging","convolutional neural network","Elastic Net regression","Random Forest","linear mixed effects model"],"falsifier":"Look up the data dictionary definitions of P01LXRKOA and P01RXRKOA in the study's public dataset. If those fields are recorded by radiologists from X-rays rather than answered by patients, retrain the Elastic Net and Random Forest models without them; if the RMSE rises well above 0.97, the paper's questionnaire-data conclusion is partly an imaging conclusion.","tokens_in":10908,"feed_emoji":"🦴","tokens_out":7637,"duration_ms":71799,"temperature":0.7,"pith_summary":"The paper sets out to prove that a doctor's questionnaire—the patient's symptoms, functional limitations, medication history, and basic measures—can estimate the radiologist's Kellgren-Lawrence severity grade for knee osteoarthritis almost as accurately as a neural network that reads the knee X-ray. On a held-out validation set, Elastic Net and Random Forest models built from patient data reach root mean squared errors of 0.974 and 0.943, while a convolutional neural network trained only on paired X-rays reaches 0.77. A linear mixed model, which accounts for the fact that each patient contributes two knees, achieves essentially the same accuracy as Elastic Net while yielding more honest uncertainty estimates. The clinical stakes are that a cheap, non-imaging assessment could be used to monitor patients and flag early-stage disease before an X-ray is ordered.","feed_headline":"Questionnaire answers rival X-ray AI for knee OA severity","feed_subtitle":"Statistical models score RMSE 0.94–0.97 against 0.77 for an image CNN, close enough to flag early risk.","key_machinery":"The argument is carried by a fixed comparison protocol: the same 70/30 patient split, the same validation set, and RMSE on the continuous KL score as the single yardstick. The four models are Elastic Net regression, a penalized linear model that selects predictors; Random Forest, a bagged ensemble of trees; a linear mixed model with a subject-level random effect to absorb the correlation between left and right knee; and a five-layer convolutional network with batch normalization, max pooling, dropout, and L2 regularization that was trained from scratch on automatically localized knee joint images. The Elastic Net and Random Forest identify shared high-importance variables; the mixed model corrects the uncertainty estimates for the hierarchical data structure. What all four models share is the outcome: a five-point ordinal grade treated as a continuous response, which the paper acknowledges is a deliberate modeling choice after ordinal logistic regression performed poorly.","core_discovery":"The authors' central claim is that statistical models trained on patient assessment data predict KOA severity on the KL 0–4 scale with good accuracy, and that this accuracy is comparable to image-based deep learning. The RMSE values are 0.974 for Elastic Net, 0.943 for Random Forest, and 0.978 for the linear mixed model, versus 0.77 for the CNN; the paper reads the difference as considerably small and states that questionnaire data predict severity with good accuracy. The same validation split is used across all models, and the statistical models have the additional advantage of naming the predictors—prior knee surgery, pain medication, functional impairment, sex—that carry the signal. The paper also reports an intraclass correlation of 0.265, showing that the two knees of a patient are strongly correlated and that a mixed model is the appropriate inferential frame. The authors themselves flag that the KL grade is subjective and not a gold standard, which sets a ceiling on how well any model can agree with a single rater.","pith_inferences":["The central comparison would be cleaner if the two strongest predictors—P01LXRKOA and P01RXRKOA, named as baseline radiographic OA status for each knee—were excluded; their labels suggest they encode X-ray findings, so a questionnaire-only model without them may be a fairer test of the paper's headline claim.","A natural extension the paper does not run is a combined model: the same validation set and outcome could be used to test whether adding questionnaire variables to the X-ray CNN pushes RMSE below 0.77, which would tell clinicians whether the two data sources carry complementary information.","The per-level RMSE pattern predicts that in a population with more severe OA—where grades 3 and 4 are common—the overall comparability would likely shrink; the questionnaire models' weakness at high grades would matter more in a tertiary-care setting than in a community screening cohort.","Because the paper reports only a single random split, a bootstrap or repeated-split estimate of the RMSE difference would show whether the 0.17–0.20 gap between CNN and questionnaire models is stable or within noise."],"forward_implications":["If the comparison holds, routine clinical questionnaires could serve as a low-cost screening step that identifies patients warranting an X-ray, rather than imaging everyone first.","The variables the models highlight—previous knee surgery, pain medication, functional knee limitations, and patient sex—could be tracked over time as an early-warning checklist for KOA progression.","Because the linear mixed model captures 26.5% of variance at the patient level, analyses that treat the two knees as independent will overstate confidence; future knee-level studies should model the nested structure.","On the hardest-to-predict cases (KL grades 3 and 4, where training data are scarce), the CNN's per-level RMSE is much lower than the questionnaires', so comparable overall does not mean interchangeable at the severe end."],"supporting_citations":[{"why":"Supplies the Kellgren-Lawrence 0–4 ordinal grading scheme that all four models predict.","marker":"[14]"},{"why":"Identifies the public longitudinal study dataset whose baseline records supply both the questionnaire variables and the paired knee X-rays.","marker":"[23]"},{"why":"Provides the image-based CNN prediction results that the questionnaire models are compared against.","marker":"[26]"},{"why":"Provides the knee-joint localization and CNN regression approach used to produce the X-ray benchmark.","marker":"[27]"},{"why":"Supplies the broader deep-learning baseline for automatic KOA severity from plain radiographs cited in the comparison.","marker":"[28]"},{"why":"Provides the Elastic Net regularization method used to select predictors and build the questionnaire-based regression.","marker":"[30]"},{"why":"Provides the Random Forest ensemble method used as the second questionnaire-based predictive model.","marker":"[31]"},{"why":"Provides the linear mixed effects model formulation used to account for correlation between the two knees of a patient.","marker":"[36]"}],"fun_headline_variants":["Surveys rival X-ray AI for knee OA severity prediction","Patient data matches deep learning for knee OA grading","Knee OA severity from questionnaires: comparable to X-ray CNN","No X-ray needed? Questionnaires predict knee OA severity as well","Questionnaire models match imaging CNN for knee OA severity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparability claim assumes that P01LXRKOA and P01RXRKOA—the two variables with the highest predictive weight—are genuinely patient-reported questionnaire items, when their names describe baseline radiographic OA status of the knee.","fun_headline_variants_meta":{"raw":{"variants":["Surveys rival X-ray AI for knee OA severity prediction","Patient data matches deep learning for knee OA grading","Knee OA severity from questionnaires: comparable to X-ray CNN","No X-ray needed? Questionnaires predict knee OA severity as well","Questionnaire models match imaging CNN for knee OA severity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1415,"prompt_tokens":980,"completion_tokens":435,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":353}},"tokens_in":596,"tokens_out":435,"duration_ms":4653,"temperature":1.0,"reasoning_tokens":353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:27:03.773035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Look up the data dictionary definitions of P01LXRKOA and P01RXRKOA in the study's public dataset. If those fields are recorded by radiologists from X-rays rather than answered by patients, retrain the Elastic Net and Random Forest models without them; if the RMSE rises well above 0.97, the paper's questionnaire-data conclusion is partly an imaging conclusion.","supporting_citations":[{"cited_title":"& Lawrence, J","cited_arxiv_id":null,"evidence_quote":"Supplies the Kellgren-Lawrence 0–4 ordinal grading scheme that all four models predict."},{"cited_title":"& Hunter, D","cited_arxiv_id":null,"evidence_quote":"Identifies the public longitudinal study dataset whose baseline records supply both the questionnaire variables and the paired knee X-rays."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the image-based CNN prediction results that the questionnaire models are compared against."},{"cited_title":"& O’Connor, N","cited_arxiv_id":null,"evidence_quote":"Provides the knee-joint localization and CNN regression approach used to produce the X-ray benchmark."},{"cited_title":"& Hastie, T","cited_arxiv_id":null,"evidence_quote":"Provides the Elastic Net regularization method used to select predictors and build the questionnaire-based regression."},{"cited_title":"Random forests","cited_arxiv_id":null,"evidence_quote":"Provides the Random Forest ensemble method used as the second questionnaire-based predictive model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the linear mixed effects model formulation used to account for correlation between the two knees of a patient."}],"review_version":1}