{"id":"9ddcf869-ad22-450b-9470-fb897d492608","arxiv_id":"2505.00032","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuned Llama 3.1 70B on text-transformed UK Biobank tabular data achieves 0.838 accuracy and 0.892 AUC for MDD diagnosis, beating XGBoost and other baselines.","lead":"This paper fine-tunes Llama 3.1 language models on text-converted UK Biobank records to diagnose major depressive disorder, reporting higher accuracy and AUC than traditional machine learning classifiers. The work demonstrates that LLMs can act as competitive classifiers on tabular medical data, although several evaluation issues limit the strength of the comparison.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported PPV/NPV in Table 3 imply a ~43.5% case-mix test set, not the 4.6% prevalence of the described UK Biobank split; headline accuracy is conditional on undisclosed rebalancing.","rationale":"The paper's central claim is high classification performance for MDD-LLM. The most load-bearing weakness is not only the label external validity identified by the reader, but an internal numerical inconsistency: Table 3's PPV/NPV values, combined with the reported sensitivity and specificity, imply a test-set prevalence around 43.5% rather than the 4.63% prevalence preserved by the described random patient-ID split. This means the headline accuracy, F1, PPV, and NPV are computed on an undisclosed case-enriched test set. Correcting to the natural prevalence would drop PPV to roughly 0.20, substantially changing the clinical interpretation. The label-validity concern remains valid but is secondary; even taking the F32/self-report label at face value, the metrics as reported cannot be reproduced from the stated experimental design. This strengthens the case for a conditional verdict: the authors must disclose the test-set construction and recompute all prevalence-dependent metrics at natural prevalence before the performance claims can be accepted.","tokens_in":11459,"tokens_out":7986,"duration_ms":84632,"concrete_test":"Run the evaluation script in the released GitHub repository and inspect how the test set is constructed. Then hold out a 20% patient-ID split with no resampling (prevalence ≈ 4.63%) and recompute ACC, PPV, and NPV for MDD-LLM 70B using the reported SENS/SPE. If PPV falls to about 0.20 while Table 3 reports 0.797, the headline metrics are artifacts of an undisclosed balanced test set and must be restated at natural prevalence.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 2.1 reports 12,715 MDD cases and 261,633 controls (4.63% prevalence). Section 2.4 says the dataset is split 80/20 by patient ID, which preserves that prevalence. However, Table 3 for MDD-LLM 70B reports SENS=0.8405, SPE=0.8358, PPV=0.7974, NPV=0.8721. Applying Bayes' theorem, PPV = SENS·p / (SENS·p + (1−SPE)·(1−p)), so the implied test-set prevalence is p ≈ 0.435. The same calculation for MDD-LLM 8B gives p ≈ 0.435, and the classical ML rows imply similarly case-enriched test sets (~0.45). This is impossible under a naturalistic 80/20 split; it indicates that the test set was balanced to about 43.5% cases, a rebalancing not disclosed anywhere in the Methods. With the actual UK Biobank prevalence, the same SENS/SPE would give PPV ≈ 0.20 and NPV ≈ 0.99, not 0.797/0.872. The abstract's headline accuracy and the PPV/NPV columns are therefore conditional on an artificial case mix, and the claim that MDD-LLM significantly outperforms baselines is not backed by correctly calibrated population-level metrics. This is an internal inconsistency, independent of the label-validity concern; it must be resolved before the central performance claim is accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MDD-LLM, a system that converts UK Biobank tabular data into text prompts and fine-tunes Llama 3.1 8B and 70B models with LoRA to predict major depressive disorder. The authors report that MDD-LLM 70B achieves accuracy 0.8378 and AUC 0.8919 (95% CI: 0.8799–0.9040), outperforming classical machine learning and deep learning baselines on a test set derived from an 80/20 patient-level split. They also compare prompt templates, fine-tuning methods, and robustness to missing features, and they provide interpretability examples. The central claim is that this is the first LLM-based solution fine-tuned on an extensive real-world dataset for MDD diagnosis.","tokens_in":11799,"tokens_out":4506,"duration_ms":47760,"significance":"If the performance claims held, the paper would provide a useful demonstration that LLMs can be applied to tabular clinical data for depression screening, with strengths including the large UK Biobank sample, public code release, exploration of prompt-format effects, and demonstration of robustness to missing features. The interpretability examples are a valuable direction for clinical adoption. However, the central performance claim is currently not trustworthy: the reported PPV/NPV values are internally inconsistent with the stated naturalistic prevalence, and the final model configuration was selected using the same test set on which performance is reported. The paper also overstates the accuracy improvement by not comparing to a trivial majority-class baseline. These issues must be resolved before the significance of the empirical results can be assessed.","major_comments":[{"comment":"","section":"Section 2.4 / Table 3"},{"comment":"","section":"Sections 3.3 and 3.4"},{"comment":"","section":"Section 3.2 / Table 3"},{"comment":"","section":"Section 2.1 / Section 6"}],"minor_comments":[{"comment":"","section":"Section 2.4"},{"comment":"","section":"Section 3.2"},{"comment":"","section":"Abstract and Table 3"},{"comment":"","section":"Throughout"},{"comment":"","section":"Section 3.5"}],"recommendation":"major_revision","confidential_remarks":"The Bayes-implied case-mix inconsistency in Table 3 is the most serious issue. If the authors cannot explain the discrepancy without disclosing an artificial test-set construction, the paper's central performance claim is not supportable. The test-set-based model selection is a second, independent problem that also needs to be fixed by redoing the evaluation with a proper validation split. I would not recommend rejection outright because the underlying idea and the large-scale experiment are potentially useful, but the current manuscript requires substantial methodological revision before the claims can be trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on arXiv:2505.00032. The paper applies LoRA fine-tuning of Llama 3.1 8B/70B to tabular UK Biobank data for MDD classification, with a large 274k cohort and extensive comparisons. That is a useful, well-scoped empirical contribution, and the missing-data robustness experiments are a nice addition. But there is a serious internal inconsistency you should know about before trusting any headline number.\n\nTable 3 reports MDD-LLM 70B: sensitivity 0.8405, specificity 0.8358, PPV 0.7974, NPV 0.8721. The UK Biobank split described in Section 2.4 preserves the 4.63% case prevalence. Using Bayes' theorem, those sensitivity/specificity numbers imply a test-set prevalence around 43.5% — that is a case-enriched test set, not a naturalistic one. The same holds for the 8B model and the classical baselines. The paper never discloses any rebalancing. This means the accuracy, PPV, and NPV figures are conditional on an artificially balanced test set, so the \"significantly outperforming\" claim is not supported as stated. The AUC may still be valid within the enriched sample (AUC is prevalence-independent), but it tells you about discrimination under an unrepresentative case mix, not real-world screening performance. The authors must explain this or the central claim falls.\n\nOther soft spots: the model configuration (Text Template, LoRA) was selected using the same test set (Sections 3.3 and 3.4), so there is some leakage of test information into the final result. The MDD label uses ICD-10 F32 plus self-report, which may capture help-seeking behavior rather than a stable diagnosis. The \"first LLM-based solution on extensive real-world data\" claim is plausible but not supported by a systematic literature search. Accuracy as a headline metric is also misleading at 4.6% prevalence — always predicting \"no\" gets 95.4%.\n\nWhat the paper does well: the scale, the breadth of baselines, the tabular-to-text prompt comparison, and the missing-feature robustness analysis are valuable. The code is available. This is a genuine empirical attempt to bring LLMs to a real psychiatric dataset.\n\nFor a referee: worth sending out, but only with a request to resolve the prevalence/PPV inconsistency, add calibration, and use a proper validation set. As it stands, I would not cite it or build on the quantitative claims.","headline":"A useful but sloppy empirical study: the AUC is plausible, but the reported PPV/NPV are mathematically inconsistent with the stated 4.6% prevalence, implying undisclosed test-set rebalancing.","tokens_in":12345,"tokens_out":3737,"would_cite":false,"duration_ms":32566,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning Llama 3.1 on 274,348 UK Biobank records yields an MDD classifier that reaches 0.8919 AUC and beats SVM, XGBoost, and MLP baselines.","keywords":["major depressive disorder","large language models","LLM fine-tuning","LoRA","UK Biobank","tabular data transformation","medical diagnosis","interpretability"],"falsifier":"Apply MDD-LLM and the best classical baseline (e.g., XGBoost) to a held-out cohort where MDD status is confirmed by structured clinical interview instead of treatment codes or self-report, and compare their AUCs on the same tabular fields. If the LLM's advantage shrinks or disappears on that cohort, the reported superiority is an artefact of the UK Biobank label rather than a general diagnostic ability.","tokens_in":11270,"feed_emoji":"🧠","tokens_out":7402,"duration_ms":67899,"temperature":0.7,"pith_summary":"MDD-LLM is a proposal to diagnose major depressive disorder by fine-tuning a general-purpose large language model on routine tabular health records that have been converted into plain-language prompts. Using 274,348 UK Biobank participants (12,715 with MDD), the fine-tuned 70B model reaches accuracy 0.8378 and area-under-the-curve (AUC) 0.8919 (95% CI 0.8799–0.9040), which the authors report as better than SVM, random forest, LightGBM, XGBoost, CatBoost, MLP, and ResNet1D trained on the same data. The paper also argues that the LLM approach tolerates missing features better than the classical models and can explain its predictions in dialogue, addressing two known barriers to deploying MDD classifiers in practice. The motivating problem is that current diagnosis relies on symptom scales requiring patient cooperation, and existing data-driven models are hard to interpret. If the result holds, it would provide a scalable, text-based path for depression diagnosis using data already present in biobank-style records.","feed_headline":"A 70B LLM beats classic models at depression diagnosis","feed_subtitle":"Fine-tuned on 274,348 UK Biobank records, it reaches 0.89 AUC and stays accurate when features are missing.","key_machinery":"The load-bearing machinery is the combination of three components: (1) a tabular-data-to-prompt transformation that turns each participant's fields into sentences such as \"Age is 60, sex is female, ...\"; (2) supervised fine-tuning of Llama 3.1 with LoRA, a parameter-efficient method that trains low-rank adapters (rank 8, alpha 16) rather than all model weights; and (3) an instruction that forces the model to answer \"Yes\" or \"No\" together with a probability, which is then used as a risk score for AUC evaluation. The language-model cross-entropy loss is the training objective, and the probability of the predicted token sequence, normalized across classes, supplies the confidence score. This design lets the model exploit pretrained linguistic and medical prior knowledge and keeps fine-tuning feasible on four H100 GPUs.","core_discovery":"The paper's central claim is that a parameter-efficiently fine-tuned LLM, given only text renderings of tabular health-record fields, is a superior MDD classifier to the classical machine and deep learning models usually applied to such data. MDD-LLM (70B) obtains ACC 0.8378, F1 0.8184, and AUC 0.8919 (95% CI 0.8799–0.9040); MDD-LLM (8B) obtains ACC 0.7904, F1 0.7627, and AUC 0.8566, both outperforming all tested baselines. The authors additionally report that LoRA fine-tuning is slightly more accurate and faster than QLoRA (40 vs 55 minutes) at the cost of more GPU memory; that Text Template and GPT-generated prompts outperform a List Template; and that keeping only 40% of features reduces accuracy by 9.24% and F1 by 9.50%, whereas XGBoost and MLP degrade much more sharply. The interpretability experiments show the model can state reasons for a yes/no prediction in free text. The conclusion positions MDD-LLM as the first LLM-based solution fine-tuned on an extensive real-world dataset for MDD diagnosis.","pith_inferences":["The UK Biobank MDD label is based on ICD-10 F32 codes plus self-reported conditions, so part of the AUC may reflect predicting who sought help or received a depression code, not necessarily a stable MDD phenotype; validation on interview-confirmed cases would settle this.","The same prompt-conversion and LoRA fine-tuning recipe could be applied to other ICD-10-defined psychiatric conditions in biobank-scale cohorts, though there is no evidence yet that the accuracy gain transfers.","Because the model produces free-text explanations, one could audit the features it cites and compare them with epidemiological risk factors, which might expose label bias or spurious correlations.","The baseline comparisons are against classical models trained from scratch on the same tabular fields; a fairer decomposition of the gain would compare MDD-LLM with an LLM of similar size fine-tuned on the same prompts, isolating the contribution of the base model's pretraining."],"forward_implications":["Depression diagnosis could be run directly on routine electronic health-record fields, without requiring a patient to complete PHQ-9 or similar scales during the diagnostic step.","Because the model keeps much of its accuracy when 60% of features are missing, LLM-based diagnosis is more usable with incomplete clinical records than gradient-boosted trees or MLPs.","A clinician-facing tool could ask the model to justify each prediction in natural language, turning a binary classifier into an auditable dialogue.","Scaling the base model from 8B to 70B parameters raises accuracy by about 4.7 percentage points and AUC by about 3.5 points, suggesting that larger foundation models are a direct path to better MDD diagnosis.","Fine-tuning with LoRA on the 8B model completes in 40 minutes on four H100 GPUs, so the approach is reproducible at moderate hardware cost."],"supporting_citations":[{"why":"Supplies the entire 274,348-participant cohort and all phenotype fields used for training and testing.","marker":"[28]"},{"why":"Defines the ICD-10 F32 depressive-episode code used to construct the MDD ground-truth label.","marker":"[30]"},{"why":"Provides the LoRA low-rank adaptation method used to fine-tune Llama 3.1.","marker":"[31]"},{"why":"Provides the QLoRA method used as the comparison fine-tuning strategy.","marker":"[32]"},{"why":"Supplies the Llama 3.1 8B and 70B base models that are fine-tuned into MDD-LLM.","marker":"[35]"}],"fun_headline_variants":["LLM diagnoses depression from health records with 0.89 AUC","Fine-tuned 70B LLM beats ML baselines in depression screening","Depression diagnosis via LLM: 0.89 AUC on 274k UK Biobank samples","MDD-LLM: AI doctor outperforms classic models for depression detection","70B LLM achieves 0.89 AUC in depression diagnosis from tabular data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the UK Biobank label for MDD, built from ICD-10 code F32 plus self-reported conditions in fields 20433 and 20434, is a true ground truth for major depressive disorder; if that label mostly captures a single depressive episode or help-seeking behaviour, the model's high accuracy partly measures consultation behaviour rather than the disease itself.","fun_headline_variants_meta":{"raw":{"variants":["LLM diagnoses depression from health records with 0.89 AUC","Fine-tuned 70B LLM beats ML baselines in depression screening","Depression diagnosis via LLM: 0.89 AUC on 274k UK Biobank samples","MDD-LLM: AI doctor outperforms classic models for depression detection","70B LLM achieves 0.89 AUC in depression diagnosis from tabular data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000695,"raw_usage":{"total_tokens":3212,"prompt_tokens":1084,"completion_tokens":2128,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":2021}},"tokens_in":700,"tokens_out":2128,"duration_ms":13423,"temperature":1.0,"reasoning_tokens":2021,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:47:57.780269+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply MDD-LLM and the best classical baseline (e.g., XGBoost) to a held-out cohort where MDD status is confirmed by structured clinical interview instead of treatment codes or self-report, and compare their AUCs on the same tabular fields. If the LLM's advantage shrinks or disappears on that cohort, the reported superiority is an artefact of the UK Biobank label rather than a general diagnostic ability.","supporting_citations":[{"cited_title":"The UK Biobank is a large-scale prospective cohort study that recruited over 500,000 individuals (aged 40-69) between 2006 and","cited_arxiv_id":null,"evidence_quote":"Supplies the entire 274,348-participant cohort and all phenotype fields used for training and testing."},{"cited_title":"QLoRA: Efficient Finetuning of Quantized LLMs","cited_arxiv_id":null,"evidence_quote":"Provides the QLoRA method used as the comparison fine-tuning strategy."}],"review_version":1}