REVIEW 5 major objections 5 minor 1 cited by
Explainable AI for Mental Health Emergency Returns: Integrating LLMs with Predictive Modeling
T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read LLM-extracted text features improve 30-day ED-return prediction (AUC 0.74 to 0.76) and let SHAP values be narrated in plain clinical language.
desk verdict Solid pipeline, but the all-cause ED-return outcome doesn't match the MH-return claim; a reanalysis is needed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a three-layer pipeline in which one LLM (LLaMA 3, 8B, prompted with ten examples) performs feature extraction and explanation generation. The first layer maps free-text chief complaints into the five categories Pain, Psychiatric, Injury, Infection, and Unclear, and maps heterogeneous social-history strings into standardized categories such as Alcohol Use, Housing Environment, and Exercise. The second layer trains standard classifiers, chiefly gradient-boosted trees, on structured electronic-health-record variables plus these LLM-derived features. The third layer combines SHAP values, patient-level input features, and cohort-level statistics into a prompt that asks the LLM to write a maximum-200-word narrative explaining the predicted risk. The identity that carries the argument is the reuse of the same few-shot prompting machinery for both input normalization and output explanation; the paper repeatedly grounds the narratives in SHAP rankings and population statistics rather than letting the LLM generate unsupported clinical claims.
What would settle it
Recompute the analysis with the outcome restricted to returns whose discharge diagnosis is a mental or behavioral F-code, or to returns linked to the index psychiatric complaint; if the LLM-feature gain in AUC (0.74 to 0.76) and AUC-PR (0.58 to 0.61) shrinks or disappears, the reported improvement is driven by general ED utilization rather than mental-health relapse. A second check is to give a new panel of clinicians the 100 generated narratives and ask them to flag any numerical inconsistency, testing whether the 99% accuracy survives outside the original reviewing pair.
Extended reading notes
Core claim
The paper's central discovery, stated on its own terms, is that a single transformer-based language model used with ten examples per category can beat both bag-of-words classifiers and a fine-tuned clinical BERT at classifying chief complaints (accuracy 0.882, F1 0.86, versus 0.59/0.53 for XGBoost and 0.63/0.59 for BlueBERT), while also standardizing social determinants of health with 0.95 accuracy and 0.96 weighted F1. Feeding those LLM-derived features into tree-based models lifts the best AUC from 0.74 to 0.76 and AUC-PR from 0.58 to 0.61. In the explainability layer, 99 of 100 sampled narratives were judged fully consistent with source records, population statistics, and SHAP-based feature rankings, with one minor numerical discrepancy and no moderate or severe errors. The paper's conclusion is that an integrated LLM-enhanced pipeline can make a predictive model both slightly more accurate and substantially more interpretable.
Load-bearing premise
The load-bearing premise is that 'return to the ED within 30 days' can stand in for 'return for a mental-health reason'; the paper uses the all-cause definition (Section 3.2), so if most returns are for unrelated medical complaints, the model may be predicting general care-seeking rather than psychiatric relapse.
Editorial extensions
If this is right
- Adding LLM-standardized chief complaints and social determinants of health lifts the best model's AUC from 0.74 to 0.76 and AUC-PR from 0.58 to 0.61, so text enrichment gives consistent but modest gains across model families.
- Chief complaint classification via ten-shot prompting (accuracy 0.882, F1 0.86) beats both bag-of-words classifiers and a fine-tuned clinical BERT, indicating useful signal can be extracted without task-specific fine-tuning.
- The explainability module ties each narrative to SHAP rankings and population statistics, and the 99-of-100 expert-reviewed accuracy indicates the generated explanations track the model's actual reasoning rather than free-standing text generation.
- Because the same LLM that standardizes input text also writes output narratives, the framework adds interpretability without a second, separately trained explanation system.
- The small size of the predictive gain implies the framework's main practical value is interpretability plus feature harmonization, not a large jump in raw prediction accuracy.
Reading between the lines
- The outcome label is any ED return within 30 days, not a mental-health-specific return; if the intended clinical target is psychiatric relapse, re-labeling with mental/behavioral F-code return diagnoses would be the decisive test of whether the reported AUC gain measures the right thing.
- The 99% explanation accuracy rests on 100 cases reviewed by two experts and, as the paper itself notes, was not tested for its effect on clinician trust or decisions; a prospective comparison against a SHAP-only arm would be a stronger test.
- The prompt-based framework is task-agnostic, so it likely transfers to other high-volume ED phenotypes such as sepsis or asthma revisits, where free-text triage and social context also matter.
- The paper's own stated limitations—single-center data, no formal clinician-trust evaluation, and computational latency—suggest that deployment would require site-specific prompt tuning and a workflow study before the narratives are used in live care.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a layered framework that combines structured EHR data with LLM-processed free-text features (chief complaints and social determinants of health) to predict 30-day emergency department returns among patients with mental health diagnoses, and that uses a second LLM step to convert SHAP values into narrative explanations. The authors report that LLaMA 3 (8B) with 10-shot learning outperforms traditional classifiers on chief complaint classification (accuracy 0.882, F1 0.86) and SDoH classification (weighted F1 up to 0.96), that adding LLM-extracted features improves XGBoost AUC from 0.74 to 0.76 and AUC-PR from 0.58 to 0.61, and that an LLM-based explainability framework produced narratives with no clinically significant errors in 99 of 100 reviewed cases. The paper includes prompt templates in an appendix and acknowledges limitations including single-center data and lack of clinician-behavior evaluation.
Significance. If the predictive and interpretability claims were fully supported, the framework would be a useful contribution to clinical decision support for ED return risk, especially because it explicitly addresses the gap between numeric SHAP values and clinician-facing narratives. The paper has several genuine strengths: it compares multiple classification strategies for chief complaints, provides full prompt templates in the appendix, deploys the LLM locally rather than through external APIs, and reports a structured expert review protocol for explanations. However, the current evidence is insufficient to establish the central claim that the model predicts 'mental health emergency returns': the outcome is all-cause ED return, the predictive gains are reported without uncertainty intervals, the SDoH classification evaluation lacks a described ground-truth annotation protocol, and the explainability success rate rests on 100 reviewed samples. These are load-bearing issues that require reanalysis or substantial revision before the conclusions are supported.
major comments (5)
- [Section 3.2, Table 5, title and abstract] The primary outcome is defined as 'whether a patient returned to the ED within 30 days of the index visit' without requiring the return to be mental-health related. The cohort is selected by an F-code on the index visit, but the label is all-cause return. Consequently the claim in the title, abstract, and Section 5 that the model predicts 'MH-related ED returns' or 'mental health emergency returns' is not supported by the outcome definition. Since SHAP analysis identifies 'number of visits in the past two months' and vital signs as top predictors, the observed AUC gain may reflect general healthcare utilization rather than psychiatric relapse. The authors should reanalyze with an outcome restricted to MH-related returns, or explicitly reframe the paper's claims to 'all-cause ED return among patients with mental health diagnoses.'
- [Section 4.2.2, Table 5] The reported improvement from adding LLM-extracted features (AUC 0.74 to 0.76; AUC-PR 0.58 to 0.61) is presented without confidence intervals, standard errors, or significance tests. Given the small absolute difference and the fact that several models (Neural Network, AdaBoost, Gradient Boosting, XGBoost) land at 0.75-0.76 AUC, the claim of 'consistent gains' is not quantitatively supported. Bootstrap confidence intervals on AUC and AUC-PR, or DeLong tests for correlated ROC curves, should be reported to establish that the difference is not chance.
- [Sections 3.4 and 4.1.2] The SDoH classification evaluation is under-specified: the manuscript does not describe how the reference labels were created, who annotated them, how many encounters were annotated per category, whether annotators were trained or blinded, or what inter-annotator agreement was. Without a described ground truth, the reported accuracies and F1 scores in Table 4 are not independently interpretable. Additionally, Table 4 reports per-category values such as Home_Environment accuracy 0.63 and F1 0.67, while the text states 'overall Accuracy of 0.95 and a weighted F1-Score of 0.96'; the relationship between these metrics should be clarified.
- [Sections 3.3.4 and 3.4] The few-shot examples are described as drawn from the training data (Section 3.3.4) or from 'real-world entries' (Section 3.4), and the 10-shot configuration is selected as best among 5-shot, 10-shot, and 20-shot. The manuscript does not state whether the test set was used at any point in selecting the number of shots or the specific examples. If test-set performance informed shot-count selection, the reported chief complaint accuracy of 0.882 could be optimistically biased. The authors should clarify the model-selection protocol and, ideally, use a separate validation set for shot-count and example selection.
- [Sections 3.6D and 4.3.1] The explainability validation is based on 100 randomly selected explanations, with one minor numerical error, yielding a claim of '99% accuracy.' This is not statistically robust: a binomial 95% confidence interval for 99/100 events spans roughly 94.7% to 100%, and the manuscript does not describe the sampling frame, blinding of expert reviewers, or a formal measure of reviewer agreement. Moreover, since the LLM is described as merely rephrasing SHAP outputs, the evaluation validates textual fidelity rather than clinical usefulness; no clinician decision-making or trust outcome is measured. These limitations should be acknowledged in the results and discussion, and the claim should be phrased with the appropriate uncertainty.
minor comments (5)
- [Figure labeling] Two different figures are labeled 'Figure 2': the explainability framework in Section 3.6 and the SHAP feature importance plot in Section 4.3.2. The figures should be renumbered sequentially.
- [Table 6 caption] The caption contains the typo 'expandability' instead of 'explainability.'
- [Table 4 and Section 4.1.2] The text says 'overall Accuracy of 0.95' for SDoH classification, but Table 4 reports category-specific accuracies ranging from 0.63 to 0.95. Please state clearly which rows or aggregation procedure give the overall figure.
- [Appendix prompts] Category label sets differ between the appendix prompts and Table 2. For example, the nutrition prompt uses 'Balanced Diet' and 'Unhealthy Diet,' while Table 2 uses 'Good Nutrition' and 'Poor Nutrition.' Align these vocabularies so the feature-generation process is reproducible.
- [Data and code availability] The data availability statement says data are available upon reasonable request, but no code or repository is provided. Given the emphasis on reproducibility of few-shot prompting and SHAP-to-narrative generation, a code repository with the exact prompts and preprocessing pipeline would strengthen the paper.
Circularity Check
Partial circularity in the explainability '99% accuracy' claim: the LLM is scored on the same SHAP values it was given as input; the predictive AUC gain is not circular.
-
self definitional
[Section 3.6D (Assessment Protocol), Section 4.3.1 (Clinical Validation), and Appendix A.9 prompt template]
"we assessed four dimensions: factual accuracy (numerical values, temporal relationships), clinical consistency (alignment with medical knowledge), logical coherence (internal consistency), and feature attribution accuracy (correspondence with SHAP values). ... Below are the top 10 features most responsible for this patient's classification: {features} ... Do not include SHAP values mid-sentence; instead, include them at the end of each item in parentheses (e.g., SHAP=0.245)."
The LLM-generated explanations are produced from a prompt that supplies the top features and instructs the LLM to state SHAP values in parentheses. The validation's 'feature attribution accuracy' is defined as correspondence with those same SHAP values. Therefore the reported '99% ... complete concordance with SHAP-derived feature rankings' is an instruction-following check on the input, not an independent empirical discovery that the LLM derived the attributions. The paper even concedes 'the LLM does not generate or infer new insights—it simply rephrases the SHAP outputs.' Expert review of clinical consistency and source records adds external validation, so this is partial rather than total circularity; the AUC improvement from LLM features remains a non-circular, independent result.
full rationale
The primary predictive modeling pipeline is self-contained and not circular: LLM few-shot extraction of chief complaints and SDoH features does not train on the outcome, the training/test split is respected, and the AUC gain (0.74 to 0.76) is an empirical comparison between feature sets. The self-citations (Refs. 28 and 32) are related-work or preprocessing references and are not load-bearing. The all-cause 30-day return outcome is a potential correctness/validity concern, but it is not a circularity argument. The one genuine circular element is in the interpretability evaluation: the LLM is fed the SHAP values and then 'feature attribution accuracy' is measured as agreement with those same SHAP values, making that submetric true by construction. Because the paper presents this as one of its two primary outcomes and reports a headline 99% accuracy, this partial by-construction claim warrants a moderate circularity score. The central predictive finding is independent, so a score of 6 reflects partial circularity rather than total derivation equivalence.
Assumptions & free parameters
free parameters (5)
- Number of few-shot examples =
10
- TF-IDF top features =
5000
- Categorical missingness exclusion threshold =
20%
- KNN imputation neighbors =
not reported
- Random oversampling ratio =
not reported
assumptions (4)
- domain assumption ICD-10-CM F-codes identify mental and behavioral disorder patients
- standard math SHAP values accurately attribute feature importance for the trained model
- domain assumption The 30-day return outcome is a valid proxy for mental health-related ED returns
- domain assumption LLM few-shot classification produces valid labels for chief complaints and SDoH fields
Cite this review
Pith. "Pith review of Explainable AI for Mental Health Emergency Returns: Integrating LLMs with Predictive Modeling." pith.science (2026). https://pith.science/paper/I5P4WB5B
@misc{pith2026250200025,
author = {Pith},
title = {Pith review of: Explainable AI for Mental Health Emergency Returns: Integrating LLMs with Predictive Modeling},
year = {2026},
howpublished = {\url{https://pith.science/paper/I5P4WB5B}},
note = {Machine review of arXiv:2502.00025}
}
read the original abstract
Importance: Emergency department (ED) returns for mental health conditions pose a major healthcare burden, with 24-27% of patients returning within 30 days. Traditional machine learning models for predicting these returns often lack interpretability for clinical use. Objective: To assess whether integrating large language models (LLMs) with machine learning improves predictive accuracy and clinical interpretability of ED mental health return risk models. Methods: This retrospective cohort study analyzed 42,464 ED visits for 27,904 unique mental health patients at an academic medical center in the Deep South from January 2018 to December 2022. Main Outcomes and Measures: Two primary outcomes were evaluated: (1) 30-day ED return prediction accuracy and (2) model interpretability using a novel LLM-enhanced framework integrating SHAP (SHapley Additive exPlanations) values with clinical knowledge. Results: For chief complaint classification, LLaMA 3 (8B) with 10-shot learning outperformed traditional models (accuracy: 0.882, F1-score: 0.86). In SDoH classification, LLM-based models achieved 0.95 accuracy and 0.96 F1-score, with Alcohol, Tobacco, and Substance Abuse performing best (F1: 0.96-0.89), while Exercise and Home Environment showed lower performance (F1: 0.70-0.67). The LLM-based interpretability framework achieved 99% accuracy in translating model predictions into clinically relevant explanations. LLM-extracted features improved XGBoost AUC from 0.74 to 0.76 and AUC-PR from 0.58 to 0.61. Conclusions and Relevance: Integrating LLMs with machine learning models yielded modest but consistent accuracy gains while significantly enhancing interpretability through automated, clinically relevant explanations. This approach provides a framework for translating predictive analytics into actionable clinical insights.
Forward citations
Cited by 1 Pith paper
-
HypoChainer: A Collaborative System Combining LLMs and Knowledge Graphs for Hypothesis-Driven Scientific Discovery
In a small user study and two case studies, a hypothesis-chain workflow grounded in knowledge graphs helped biomedical researchers construct and validate hypotheses from machine-learning predictions more effectively t...
Reference graph
Works this paper leans on
-
[2]
Related Work 2.1 Emergency Department Returns and MH Utilization EDs serve as a critical entry point for individuals experiencing acute MH crises, yet they are often ill-equipped to provide comprehensive psychiatric care [16]. Data from the National Hospital Ambulatory Medical Care Survey indicate that adults with MH disorders accounted for 52.9 ED visits...
work page 2017
-
[4]
Results 4.1 LLM features extraction performance results This section evaluates the performance of the LLM (Llama 3:8-billion) in feature extraction for chief complaint and SDoH classifications. Few-shot learning approaches are compared to traditional ML and pre-trained models. 4.1.1 Chief Complaint Classification The classification of chief complaints was...
-
[5]
Discussion This study introduces a layered clinical AI framework that advances both prediction accuracy and interpretability by integrating structured data, LLM-processed unstructured features, and narrative explanations. The first layer focuses on enriching the input data through LLM-driven processing of free-text fields, including chief complaints and S...
-
[7]
History of use, unsure if still using
Input: "History of use, unsure if still using" → Category: Unclear/Other Now classify: Input: "[SUBSTANCE_TEXT]" → Category: 10.6 A.6 Exercise Classification Prompt Classify the patient’s physical activity level into: - Regular Exercise - Sedentary Lifestyle - Occasional Activity - Limited Mobility - Unclear/Other Examples: 1. Input: "Walks daily for 30 m...
-
[8]
For each feature, explain how it affects risk using clinical terms (e.g., "frequent visits", "elevated heart rate"), and compare to population values if available. - Do not include SHAP values mid-sentence; instead, include them at the end of each item in parentheses (e.g., SHAP=0.245). 3. Group features with unclear or missing population data together in...
-
[9]
Institutional Review Board (IRB) Statement The Institutional Review Board (IRB) of the University of Alabama at Birmingham (UAB) determined that this study (IRB # IRB-300008858) does not meet the criteria for human subjects research and therefore does not require ethical approval. References 1. UHG, Avoidable Hospital Emergency Department Visits. 2019. 2....
work page 2019
-
[10]
Appendix A: Prompt Templates Used for LLM Classification 10.1 A.1 Chief Complaint Classification Prompt You are a medical classification assistant. Classify the following emergency department chief complaints into one of five categories: Pain, Psychiatric, Injury, Infection, or Unclear. Examples: 1. Chief Complaint: "I can't stop vomiting." → Category: In...
-
[23]
Zhou, S., et al., Large language models for disease diagnosis: A scoping review. npj Artificial Intelligence, 2025. 1(1): p. 1-17. 24. Nohara, Y ., et al., Explanation of machine learning models using shapley additive explanation and application for real data in hospital. Computer Methods and Programs in Biomedicine, 2022. 214: p. 106584. 25. Nordin, N., ...
arXiv 2025
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.