{"id":"a0dec0f8-eee3-4ba0-bf14-783afe6204f5","arxiv_id":"1908.09251","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Machine learning models trained on Danish registry data predict biologic discontinuation in psoriasis with about 80% accuracy and treatment duration within 4.5 months, though without external validation.","lead":"The paper trains seven machine learning models on 681 Danish psoriasis patients to predict whether a biologic drug will be stopped and how long it lasts, reporting 82% accuracy and a mean prediction error of 4.5 months. It matters because it suggests routine clinic variables might be enough to personalize biologic treatment decisions, but the model has not been validated outside the registry it was trained on.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Repeated treatment series from the same patient leak across random CV folds; patient-level cross-validation is needed before the 82% accuracy claim can be accepted.","rationale":"The reader's weakest assumption—that treatment series from the same patient are inappropriately treated as independent in cross-validation—is the most load-bearing risk to the paper's central claim. The manuscript describes random partitioning of examples without any patient-level grouping, and the data description explicitly shows that most patients contribute multiple series. This is a concrete, testable flaw: regrouped CV would settle whether the 82% accuracy and 0.935 correlation generalize beyond patient memorization. The paper has some genuine strengths: it uses a real registry cohort, evaluates several algorithms, and reports clinically interpretable outputs, and its stated limitations acknowledge the need for external validation. No ad hominem concerns arise. Because the reader already flagged this exact issue and recommended a conditional verdict, my stress test does not change the verdict; it strengthens the rationale for requiring patient-level cross-validation before the result is accepted.","tokens_in":8265,"tokens_out":2274,"duration_ms":25534,"concrete_test":"Repeat the 5-fold cross-validation with patient-level grouping: assign all treatment series from a given patient to the same fold (e.g., group k-fold by patient ID), retrain the GLM and duration model, and compare the resulting accuracy, MAE, and Pearson correlation with Table 2 and Figure 2. If the GLM accuracy drops by more than about 5 percentage points, or the duration correlation falls substantially below 0.935, the reported performance is inflated by within-patient leakage and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline accuracy (Table 2, GLM accuracy 0.815) is obtained by 5-fold cross-validation that, per Methods, 'randomly partitions a set of examples into 5 (non-overlapping) sets.' The unit of analysis is the treatment series, not the patient. Table 1 reports that 433 of 681 patients contributed more than one treatment series, so the same patient's multiple series can fall on both sides of the training/test split. Because the outcome (discontinuation cause and duration) and key predictor 'previous biologic use' are strongly patient-dependent, random series-level splitting leaks patient-specific information into training and inflates the reported accuracy, the duration MAE of 4.5 months, and the r=0.935 correlation. The paper's stated limitations (small sample, single country, retrospective design) do not address this clustering issue, and no patient-level or group-based validation is described. If the true generalization accuracy is materially lower under patient-level CV, the central claim that routine variables can predict individual biologic outcomes is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes a machine learning analysis of the Danish Dermbio registry (681 psoriasis patients) with the goal of predicting, at the individual level, the risk and cause of biologic discontinuation and the duration of treatment. Seven supervised learning algorithms are compared using 5-fold cross-validation; the authors report accuracies between 0.73 and 0.82, with a generalized linear model (GLM) achieving 0.815, and a separate GLM yielding a mean absolute error of 4.5 months and a Pearson correlation of 0.935 for treatment duration. The paper additionally presents an 'input optimization' profile of a patient with a high chance of continuing treatment, and concludes that machine learning predicts outcomes with accuracy exceeding 80% based on a small set of routine variables.","tokens_in":8475,"tokens_out":5494,"duration_ms":57371,"significance":"If the reported performance estimates were obtained under a valid evaluation protocol, the work would be clinically useful because it relies on variables routinely available in practice and offers interpretable model output for patient communication. The comparison of seven algorithms on a real-world registry and the use of cross-validation in principle are strengths. However, the current analysis does not establish that the claimed predictive performance generalizes to new patients. The paper provides no code, no data availability statement, and no reproducibility artifacts. The central accuracy claim is therefore not yet supported.","major_comments":[{"comment":"The 5-fold cross-validation randomly partitions treatment series, not patients, yet Table 1 shows that 433 of 681 patients contributed more than one treatment series. This allows the same patient's multiple series to appear in both training and test folds, which can inflate the reported accuracies (Table 2) and the duration correlation through patient-level leakage. The authors should re-run the analysis with patient-level or group-based cross-validation and report whether the 0.815 accuracy and the 0.935 correlation hold under that more appropriate evaluation scheme.","section":"Methods, Machine Learning"},{"comment":"The preprocessing section states that predictors with missing values were filled with 'Null or appropriate value indicating unavailability of the data.' Given that Table 1 reports baseline PASI completeness of only 8.25%, DLQI completeness of 38.18%, and weight completeness of 57.27%, this crude handling can create artificial patterns (missingness indicators) that drive predictions, and it does not reflect how the model would be used with incomplete real-world data. The authors need to specify the imputation procedure precisely and perform sensitivity analyses under multiple missing-data assumptions.","section":"Methods, Data preprocessing and feature engineering"},{"comment":"The prediction of treatment duration is presented with MAE of 4.5 months and Pearson correlation 0.935, but the text does not state whether the duration GLM was evaluated on held-out data or on the same cohort used for fitting. If the latter, the reported agreement is in-sample and does not support the claim that treatment duration can be predicted for new patients. Moreover, treatment series that are censored (patients still on therapy at the last observation) are not discussed; treating censored durations as complete outcomes biases the error metrics. The authors should evaluate the duration model with patient-level cross-validation and address censoring, for example by using time-to-event modeling or by explicitly excluding ongoing series.","section":"Results, Predicting the treatment length"},{"comment":"The 'input optimization' profile in the Results states '≥23 year-old at the time of diagnosis', which is inconsistent with the abstract's characterization of the profile as 'early-onset' psoriasis, since early onset typically refers to onset before age 40 or a similar threshold. In addition, the profile is derived by inverting the same fitted GLM and is presented without any validation or confidence interval; the claim that such a patient has a ≤10% discontinuation risk is not an independent finding. The abstract profile and the Results profile must be reconciled, and the profile should be treated as an exploratory model-based description unless it is externally validated.","section":"Results, Input optimization reveals patient profile"},{"comment":"Feature selection via PCA is described as a preprocessing step before model training. If PCA is applied to the full dataset before the cross-validation split, it leaks information from the test folds into the training procedure, which can inflate accuracy. The authors should clarify that PCA, and any other preprocessing steps, are performed inside each training fold, or use a pipeline that prevents information from test examples from entering the feature-selection step.","section":"Methods, Data preprocessing and feature engineering"}],"minor_comments":[{"comment":"The affiliation contains a typo: 'Canda' should be 'Canada'.","section":"Title page"},{"comment":"The accuracy formula is written for binary classification (TP+TN over the total), but the outcome appears to be multinomial with at least three classes (lack of efficacy, adverse event, other, possibly with continuation as an additional class). The authors should define the multiclass accuracy used to produce Table 2.","section":"Methods, Performance analysis"},{"comment":"The text refers to 'GLA model' where 'GLM model' is intended.","section":"Results, Predicting the treatment length"},{"comment":"The total number of treatment series (rows) used for model training is not reported, only the number of patients. Please report the number of examples for each analysis.","section":"Results"},{"comment":"Runtime values (1.4 to 53 seconds) are reported without specifying the hardware and software versions; please provide this information or move runtimes to a supplementary table.","section":"Methods, Machine Learning"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim depends on the validity of the cross-validation, and the patient-clustering issue is substantial. The reported accuracy and correlation are likely optimistic because the same patient's multiple treatment series can appear in both training and test folds. I recommend requiring a re-analysis with patient-level cross-validation before publication, along with clarification of the missing-data handling and the duration-model evaluation. The clinical question is relevant, and the authors' approach of comparing several algorithms is reasonable, but the current evidence falls short of the strong claims made in the abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Off the record: this is a legitimate first application of standard machine learning to a psoriasis registry, and the paper is readable and honest about several limitations. But the central accuracy claim—82% for discontinuation risk—is probably overstated because the cross-validation splits treatment series, not patients. That is the main thing to know.\n\nWhat is genuinely new: the authors applied seven off-the-shelf classifiers to the Dermbio cohort and produced a patient profile for long-term biologic success (biologic-naive, under 49, early-onset plaque psoriasis, no PsA, weight <100 kg, moderate-to-severe disease). The algorithms are routine, but the empirical result on this real registry is new. The comparison across methods is sensible, and the authors are appropriately cautious about the retrospective, single-country design.\n\nThe soft spots are real. The most serious is the clustering issue. Table 1 shows 433 of 681 patients contributed more than one treatment series, and the 5-fold CV randomly partitions examples (treatment series), not patients. So the same patient's multiple series can appear in both training and test, leaking patient-specific information. That inflates the 0.815 accuracy, the 4.5-month MAE, and the r=0.935 correlation. This is not a minor technicality; it directly supports the headline claim. A patient-level split or a mixed-effects model would address it. Second, missingness is extreme—baseline PASI is only 8% complete—and the ad hoc Null filling likely turns 'missing' into a predictor. Third, the input-optimization profile is obtained by inverting the same fitted GLM, so presenting it as an independent finding overstates what it is: a description of the model's decision boundary.\n\nOn balance, the central argument does not hold up as presented. The paper would be worth serious refereeing if the authors redo the validation at the patient level, report how missingness is handled, and soften the conclusions. The topic is timely, the data are real, and the profile is clinically plausible.\n\nThis is a paper for dermatology and clinical-ML readers. I would not cite the numerical claims yet, but I would bring it to a reading group as a case study in cross-validation pitfalls. Send to peer review, but expect major revision.","headline":"A plausible first ML application to a psoriasis registry, but the 82% accuracy claim is likely inflated by patient-level clustering that the cross-validation ignores.","tokens_in":8999,"tokens_out":2956,"would_cite":false,"duration_ms":29345,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Routine clinical variables—age, weight, previous biologic use, disease activity—let a machine learning model predict psoriasis biologic discontinuation and its duration, with reported accuracy above 80%.","keywords":["psoriasis","biologic therapy","machine learning","drug discontinuation","treatment duration","generalized linear model","personalized medicine","registry data"],"falsifier":"Rerun the evaluation so that all treatment records from one patient stay in the same division of the data, never split between the records used to build the model and the records used to test it. If the 82% accuracy and 0.935 correlation drop substantially under that patient-level split, the reported individual-level performance is inflated; if they hold, the concern is answered.","tokens_in":8084,"feed_emoji":"💊","tokens_out":10025,"duration_ms":94880,"temperature":0.7,"pith_summary":"This paper tries to establish that long-term outcomes of biologic therapy in psoriasis are predictable for individual patients from routinely available clinical variables, not just from group-level statistics. Using a Danish registry cohort of 681 patients, the authors report that a generalized linear model predicts whether a treatment series ends in discontinuation—and whether the reason is lack of efficacy, adverse events, or something else—with 82% accuracy. The same model estimates how long a patient will stay on therapy, with a mean absolute error of 4.5 months and a strong correlation between predicted and actual durations. If correct, doctors could move from trial-and-error prescribing toward a patient-specific risk estimate, and patients could be told their absolute chance of long-term success. The paper also derives a profile of the patient most likely to continue a biologic for the long term.","feed_headline":"82% accuracy: machine learning forecasts psoriasis biologic outcomes","feed_subtitle":"Routine clinic data can predict drug discontinuation, its cause, and treatment length to within 4.5 months.","key_machinery":"The load-bearing mechanism is the generalized linear model (GLM), a regression framework that links input variables to a categorical or continuous outcome through a link function; here it classifies the cause of biologic discontinuation in one model and predicts treatment length in another. The inputs are 14 variables that a psoriasis clinic already records, including demographics, weight, comorbidity count, psoriatic arthritis status, methotrexate use, previous biologic exposure, baseline DLQI (dermatology life quality index) and PASI (psoriasis area and severity index), and the specific drug. Performance is estimated with 5-fold cross-validation, and agreement for treatment duration is checked with Bland-Altman analysis, mean absolute error, and Pearson correlation. An input-optimization step inverts the fitted model to identify the patient features that yield the lowest predicted withdrawal risk, producing the paper's 'ideal patient' profile, while the decision tree supplies readable rules such as infliximab being linked to inefficacy-related withdrawal within the first two years.","core_discovery":"The paper's central claim is that drug discontinuation—the real-world surrogate for treatment failure—is not random noise but a predictable function of a small set of clinical features. In the Danish registry cohort of 681 patients, a generalized linear model classified the outcome of a treatment series (continuation, discontinuation due to lack of efficacy, adverse event, or other cause) with 82% accuracy, and every one of the seven algorithms tested scored above 73%. A separate generalized linear model predicted the length of the treatment series with a mean absolute error of 4.5 months and a Pearson correlation of 0.935 between predicted and actual lengths. The input-optimization analysis concludes that a biologic-naive patient under 49 years old with early-onset plaque psoriasis, no psoriatic arthritis, weight under about 99 kg, and moderate-to-severe disease activity (DLQI ≥ 16, PASI ≥ 9.4) has at least a 90% chance of remaining on treatment—compared with population averages of roughly 20% discontinuation in the first year and 50% by year three. The authors present these results as evidence that machine learning can serve as a decision-support tool for communicating expected outcomes to patients.","pith_inferences":["Inference: If the data were split by patient rather than by treatment series, the reported accuracy and 0.935 correlation would likely drop, because 433 of 681 patients contributed multiple series; deployment should therefore validate at the patient level.","Inference: The ideal-patient profile is a falsifiable decision rule: a prospective cohort could check whether patients matching those criteria actually show one-year withdrawal risk of 10% or less, rather than the roughly 20% average.","Inference: The combination of a 0.935 Pearson correlation and a 4.5-month mean absolute error suggests the duration model ranks patients by persistence better than it fixes an exact stop date, so risk strata may be a safer clinical output than a point estimate."],"forward_implications":["A clinician could enter a patient's routine values and receive an absolute risk of stopping the biologic, plus the most likely reason, without interpreting odds ratios or hazard ratios.","A patient fitting the optimized profile can be told their chance of staying on treatment is at least 90%, well above the average population's roughly 80% one-year persistence.","The duration estimate, with a mean absolute error of 4.5 months, gives a concrete expectation for how long a prescribed biologic is likely to be continued.","Because model fitting and prediction take seconds on an ordinary laptop, the tool could be embedded in routine clinical workflows at minimal cost.","Retraining on local registry data would be needed in different health systems, since access, reimbursement, and guidelines influence drug survival."],"supporting_citations":[{"why":"Provides the registry data on safety, efficacy, and drug survival that form the training cohort.","marker":"1"},{"why":"Supports drug discontinuation as a real-world outcome and compares clinical-trial-eligible and ineligible patients.","marker":"3"},{"why":"Prior analysis of the same registry cohort that defines the dataset and baseline drug survival rates.","marker":"4"},{"why":"Earlier drug survival comparison for adalimumab, etanercept, and infliximab that motivates persistence as an outcome.","marker":"5"},{"why":"Surveys psoriasis registries worldwide and underpins the claim that combined registry data could be analyzed this way.","marker":"14"},{"why":"Supplies the Bland-Altman method used to quantify agreement between predicted and actual treatment lengths.","marker":"15"},{"why":"Establishes the generalized linear model framework that the main predictor is built on.","marker":"16"},{"why":"Meta-analysis of factors predicting biologic persistence, used to check the model's negative predictors and the general population baseline.","marker":"17"}],"fun_headline_variants":["ML predicts psoriasis biologic outcomes with 82% accuracy","Predicting psoriasis biologic failure: 82% accurate within 4.5 mo","Psoriasis biologic outcomes: ML at 82% accuracy, 4.5-month error","ML forecasts psoriasis biologic success: 82% accurate, 4.5-mo error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every recorded treatment series is independent of every other; because 433 of 681 patients have more than one series, a model could be learning patient identity rather than general rules when those series are split across the records used to build the model and the records used to test it.","fun_headline_variants_meta":{"raw":{"variants":["ML predicts psoriasis biologic outcomes with 82% accuracy","Predicting psoriasis biologic failure: 82% accurate within 4.5 mo","Psoriasis biologic outcomes: ML at 82% accuracy, 4.5-month error","ML forecasts psoriasis biologic success: 82% accurate, 4.5-mo error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001162,"raw_usage":{"total_tokens":4879,"prompt_tokens":1082,"completion_tokens":3797,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":698,"completion_tokens_details":{"reasoning_tokens":3712}},"tokens_in":698,"tokens_out":3797,"duration_ms":25038,"temperature":1.0,"reasoning_tokens":3712,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:16:36.317118+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the evaluation so that all treatment records from one patient stay in the same division of the data, never split between the records used to build the model and the records used to test it. If the 82% accuracy and 0.935 correlation drop substantially under that patient-level split, the reported individual-level performance is inflated; if they hold, the concern is answered.","supporting_citations":[],"review_version":1}