{"id":"44552d64-4bee-4acd-96aa-f7e14e888d1f","arxiv_id":"2502.00025","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LLM-extracted features improved XGBoost's 30-day ED return AUC from 0.74 to 0.76, and an LLM-based explainability layer translated SHAP values into clinician-readable narratives with 99% expert-rated accuracy on a 100-sample review.","lead":"This study combines large language models with traditional machine learning to predict whether mental health patients return to the emergency department within 30 days. The authors report modest accuracy gains and a new way to turn model explanations into plain-language clinical narratives.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The primary outcome is all-cause 30-day ED return, not mental-health-related return; without reanalysis using an MH-specific outcome, the central predictive claim is not established.","rationale":"I read the paper in good faith. The intended contribution is a framework that improves prediction of mental-health-related ED returns by adding LLM-processed chief complaint and SDoH features. The strongest empirical evidence for this is the AUC/AUC-PR increase in Table 5. That table's outcome, however, is defined in Section 3.2 as any ED return within 30 days. This is a mismatch with the stated psychiatric focus. The clinical question matters: predicting 'will this patient bounce back for any reason' is different from predicting 'will this patient have a psychiatric crisis.' The features added by the LLM—chief complaint categorization (Pain, Psychiatric, Injury, Infection, Unclear) and SDoH—may improve prediction of all-cause returns through general risk factors, not MH-specific relapse. The paper even includes SDoH and chief complaint categories like Injury, which are not MH-specific. The SHAP results emphasize prior visits and vital signs, which are generic utilization markers. Therefore, the reported gain could be the result of a broader, easier-to-predict outcome. This is not an internal inconsistency; the paper is transparent about the outcome. But it is a validity threat to the central claim. Reanalysis with an MH-specific return outcome, or at least a sensitivity analysis excluding returns with clearly non-psychiatric primary diagnoses, would settle it. The reader already flagged this as the weakest assumption, and I agree. Since the reader's verdict is CONDITIONAL—which appropriately demands such reanalysis before full acceptance—my stress-test does not shift the verdict. I did not find a more load-bearing concern; other issues (lack of error bars, possible shot-count selection on test data, unvalidated SDoH ground truth) are secondary to this outcome-label problem.","tokens_in":15436,"tokens_out":2086,"duration_ms":23067,"concrete_test":"Re-run the XGBoost comparison (with vs. without LLM-extracted features) using a restricted outcome: a 30-day ED return whose primary diagnosis at the return visit is a mental/behavioral disorder (any ICD-10 F-code, or the same F-code list used for cohort inclusion). Report AUC and AUC-PR with confidence intervals for both outcome definitions over the same train/test split. If the with-LLM AUC (0.76) does not meaningfully exceed the without-LLM AUC (0.74) under the MH-specific outcome, or if the gap falls within noise, the paper's central predictive claim is not supported for the claimed target.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that LLM-extracted features improve prediction of 'mental health emergency returns.' However, Section 3.2 defines the outcome as 'a binary indicator of whether a patient returned to the ED within 30 days of the index visit.' The cohort is restricted to patients with an ICD-10 F-code on the index visit, but the outcome does not require the return visit to be MH-related. A patient with depression who returns for chest pain, injury, or infection is counted as a positive. Because the model is trained and evaluated on this all-cause return label, the reported AUC gain (0.74 to 0.76) may reflect general healthcare utilization patterns—prior visit frequency, vitals, acuity—rather than psychiatric relapse. This is especially plausible given that the SHAP analysis identifies 'number of visits in the past two months' and vital signs as top predictors, which are generic utilization/severity markers. The framing throughout the paper, the title, and the discussion all claim MH-specific risk, so the outcome misalignment threatens the central clinical interpretation. If the improvement does not persist when the outcome is restricted to MH-related returns, the model does not do what the paper says it does. The reader's weakest assumption identifies exactly this issue, and I agree it is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a layered framework that combines structured EHR data with LLM-processed free-text features (chief complaints and social determinants of health) to predict 30-day emergency department returns among patients with mental health diagnoses, and that uses a second LLM step to convert SHAP values into narrative explanations. The authors report that LLaMA 3 (8B) with 10-shot learning outperforms traditional classifiers on chief complaint classification (accuracy 0.882, F1 0.86) and SDoH classification (weighted F1 up to 0.96), that adding LLM-extracted features improves XGBoost AUC from 0.74 to 0.76 and AUC-PR from 0.58 to 0.61, and that an LLM-based explainability framework produced narratives with no clinically significant errors in 99 of 100 reviewed cases. The paper includes prompt templates in an appendix and acknowledges limitations including single-center data and lack of clinician-behavior evaluation.","tokens_in":15713,"tokens_out":3478,"duration_ms":36505,"significance":"If the predictive and interpretability claims were fully supported, the framework would be a useful contribution to clinical decision support for ED return risk, especially because it explicitly addresses the gap between numeric SHAP values and clinician-facing narratives. The paper has several genuine strengths: it compares multiple classification strategies for chief complaints, provides full prompt templates in the appendix, deploys the LLM locally rather than through external APIs, and reports a structured expert review protocol for explanations. However, the current evidence is insufficient to establish the central claim that the model predicts 'mental health emergency returns': the outcome is all-cause ED return, the predictive gains are reported without uncertainty intervals, the SDoH classification evaluation lacks a described ground-truth annotation protocol, and the explainability success rate rests on 100 reviewed samples. These are load-bearing issues that require reanalysis or substantial revision before the conclusions are supported.","major_comments":[{"comment":"The primary outcome is defined as 'whether a patient returned to the ED within 30 days of the index visit' without requiring the return to be mental-health related. The cohort is selected by an F-code on the index visit, but the label is all-cause return. Consequently the claim in the title, abstract, and Section 5 that the model predicts 'MH-related ED returns' or 'mental health emergency returns' is not supported by the outcome definition. Since SHAP analysis identifies 'number of visits in the past two months' and vital signs as top predictors, the observed AUC gain may reflect general healthcare utilization rather than psychiatric relapse. The authors should reanalyze with an outcome restricted to MH-related returns, or explicitly reframe the paper's claims to 'all-cause ED return among patients with mental health diagnoses.'","section":"Section 3.2, Table 5, title and abstract"},{"comment":"The reported improvement from adding LLM-extracted features (AUC 0.74 to 0.76; AUC-PR 0.58 to 0.61) is presented without confidence intervals, standard errors, or significance tests. Given the small absolute difference and the fact that several models (Neural Network, AdaBoost, Gradient Boosting, XGBoost) land at 0.75-0.76 AUC, the claim of 'consistent gains' is not quantitatively supported. Bootstrap confidence intervals on AUC and AUC-PR, or DeLong tests for correlated ROC curves, should be reported to establish that the difference is not chance.","section":"Section 4.2.2, Table 5"},{"comment":"The SDoH classification evaluation is under-specified: the manuscript does not describe how the reference labels were created, who annotated them, how many encounters were annotated per category, whether annotators were trained or blinded, or what inter-annotator agreement was. Without a described ground truth, the reported accuracies and F1 scores in Table 4 are not independently interpretable. Additionally, Table 4 reports per-category values such as Home_Environment accuracy 0.63 and F1 0.67, while the text states 'overall Accuracy of 0.95 and a weighted F1-Score of 0.96'; the relationship between these metrics should be clarified.","section":"Sections 3.4 and 4.1.2"},{"comment":"The few-shot examples are described as drawn from the training data (Section 3.3.4) or from 'real-world entries' (Section 3.4), and the 10-shot configuration is selected as best among 5-shot, 10-shot, and 20-shot. The manuscript does not state whether the test set was used at any point in selecting the number of shots or the specific examples. If test-set performance informed shot-count selection, the reported chief complaint accuracy of 0.882 could be optimistically biased. The authors should clarify the model-selection protocol and, ideally, use a separate validation set for shot-count and example selection.","section":"Sections 3.3.4 and 3.4"},{"comment":"The explainability validation is based on 100 randomly selected explanations, with one minor numerical error, yielding a claim of '99% accuracy.' This is not statistically robust: a binomial 95% confidence interval for 99/100 events spans roughly 94.7% to 100%, and the manuscript does not describe the sampling frame, blinding of expert reviewers, or a formal measure of reviewer agreement. Moreover, since the LLM is described as merely rephrasing SHAP outputs, the evaluation validates textual fidelity rather than clinical usefulness; no clinician decision-making or trust outcome is measured. These limitations should be acknowledged in the results and discussion, and the claim should be phrased with the appropriate uncertainty.","section":"Sections 3.6D and 4.3.1"}],"minor_comments":[{"comment":"Two different figures are labeled 'Figure 2': the explainability framework in Section 3.6 and the SHAP feature importance plot in Section 4.3.2. The figures should be renumbered sequentially.","section":"Figure labeling"},{"comment":"The caption contains the typo 'expandability' instead of 'explainability.'","section":"Table 6 caption"},{"comment":"The text says 'overall Accuracy of 0.95' for SDoH classification, but Table 4 reports category-specific accuracies ranging from 0.63 to 0.95. Please state clearly which rows or aggregation procedure give the overall figure.","section":"Table 4 and Section 4.1.2"},{"comment":"Category label sets differ between the appendix prompts and Table 2. For example, the nutrition prompt uses 'Balanced Diet' and 'Unhealthy Diet,' while Table 2 uses 'Good Nutrition' and 'Poor Nutrition.' Align these vocabularies so the feature-generation process is reproducible.","section":"Appendix prompts"},{"comment":"The data availability statement says data are available upon reasonable request, but no code or repository is provided. Given the emphasis on reproducibility of few-shot prompting and SHAP-to-narrative generation, a code repository with the exact prompts and preprocessing pipeline would strengthen the paper.","section":"Data and code availability"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the outcome definition: the paper's central claim concerns mental-health-related ED returns, but the outcome is all-cause returns within 30 days. This is fixable by reanalysis or by honestly reframing the paper, so I do not recommend rejection. The lack of confidence intervals and the under-specified SDoH annotation protocol are also substantive but addressable. I would advise the editor to require the reanalysis or reframing as a condition of revision, and to ask for a clearer separation of validation data in the few-shot selection process."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a clean implementation of a plausible pipeline—LLM few-shot extraction of chief complaints and SDoH, combined with SHAP-to-narrative explainability—on a large retrospective cohort (42k visits). The writing is clear, the framework is honest about what the LLM does (it rephrases SHAP values, not adding new inference), and they report limitations like single center and no clinician trust testing. References are appropriate; the novelty claim is modest and accurate. Credit where due: the 10-shot LLaMA classification of chief complaints beats TF-IDF/BlueBERT baselines, and the improvement in AUC-PR from 0.58 to 0.61 is consistent across boosting models, even if small.\n\nThe soft spot is load-bearing. Section 3.2 defines the outcome as any return to the ED within 30 days of the index visit. The cohort is restricted to patients with an F-code on the index visit, but the return itself is not required to be mental-health-related. So a patient with depression who returns for chest pain or an injury counts as a positive. The title, abstract, and discussion repeatedly frame this as MH-specific risk prediction, and the SHAP results emphasize non-specific utilization markers (prior visit count, heart rate). You can't tell whether the LLM-extracted features are capturing psychiatric relapse or general healthcare-seeking behavior. This needs reanalysis with a mental-health-related return label (e.g., F-code on the return visit) or at least a sensitivity analysis excluding non-MH returns.\n\nOther issues are more minor. The AUC gain lacks confidence intervals; with ~42k visits the gain might be statistically significant, but you'd want to see error bars. The SDoH classification evaluation is under-specified—no description of how ground-truth labels were obtained for the 0.95 accuracy/F1 claims. The 5/10/20-shot comparison is reported but the choice of 10-shot is not explicitly justified on a validation set, so the possibility of test-set selection is open (though not demonstrated). The explainability validation is only 100 explanations, and 99% accuracy is a thin basis for the 'clinically accurate' claim, though the authors do acknowledge that clinician decision-making wasn't tested.\n\nIs the central idea salvageable? Yes. The framework is sensible and the feature extraction results are promising. But the outcome definition is the kind of thing a referee would catch immediately, and it's fixable. I'd recommend sending to review with a major-revision decision: require the MH-specific outcome reanalysis, confidence intervals, and a proper SDoH annotation protocol. If the predictive gain disappears under the correct outcome, the paper still has value as an explainability framework, but it can't make the predictive claim it currently makes.","headline":"Solid pipeline, but the all-cause ED-return outcome doesn't match the MH-return claim; a reanalysis is needed.","tokens_in":16233,"tokens_out":3513,"would_cite":false,"duration_ms":31500,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-extracted text features improve 30-day ED-return prediction (AUC 0.74 to 0.76) and let SHAP values be narrated in plain clinical language.","keywords":["Emergency Department","30-Day ED Return","Large Language Models","Explainable AI","SHAP","Social Determinants of Health","XGBoost","Mental Health"],"falsifier":"Recompute the analysis with the outcome restricted to returns whose discharge diagnosis is a mental or behavioral F-code, or to returns linked to the index psychiatric complaint; if the LLM-feature gain in AUC (0.74 to 0.76) and AUC-PR (0.58 to 0.61) shrinks or disappears, the reported improvement is driven by general ED utilization rather than mental-health relapse. A second check is to give a new panel of clinicians the 100 generated narratives and ask them to flag any numerical inconsistency, testing whether the 99% accuracy survives outside the original reviewing pair.","tokens_in":15280,"feed_emoji":"🧠","tokens_out":12207,"duration_ms":108566,"temperature":0.7,"pith_summary":"This paper is trying to establish that one large language model can do two jobs for a clinical prediction task: turn messy free-text chief complaints and social-history fields into standardized features, and turn SHAP attributions into short plain-language narratives that clinicians can interpret. The reported payoff is a consistent but modest gain in predicting 30-day emergency department returns among mental health patients, with XGBoost AUC rising from 0.74 to 0.76 and AUC-PR from 0.58 to 0.61, plus near-perfect explanation quality in a 100-case expert review. A sympathetic reader would take the central contribution to be the layered framework itself: enrich inputs with an LLM, predict with a traditional model, and explain with the same LLM. The paper also claims that the LLM explanations do not invent new findings; they rephrase SHAP outputs and population statistics in clinically coherent terms.","feed_headline":"LLM features nudge ED-return AUC from 0.74 to 0.76 and narrate SHAP","feed_subtitle":"A 10-shot LLM both standardizes clinical free text and turns model attributions into clinician-ready narratives.","key_machinery":"The central object is a three-layer pipeline in which one LLM (LLaMA 3, 8B, prompted with ten examples) performs feature extraction and explanation generation. The first layer maps free-text chief complaints into the five categories Pain, Psychiatric, Injury, Infection, and Unclear, and maps heterogeneous social-history strings into standardized categories such as Alcohol Use, Housing Environment, and Exercise. The second layer trains standard classifiers, chiefly gradient-boosted trees, on structured electronic-health-record variables plus these LLM-derived features. The third layer combines SHAP values, patient-level input features, and cohort-level statistics into a prompt that asks the LLM to write a maximum-200-word narrative explaining the predicted risk. The identity that carries the argument is the reuse of the same few-shot prompting machinery for both input normalization and output explanation; the paper repeatedly grounds the narratives in SHAP rankings and population statistics rather than letting the LLM generate unsupported clinical claims.","core_discovery":"The paper's central discovery, stated on its own terms, is that a single transformer-based language model used with ten examples per category can beat both bag-of-words classifiers and a fine-tuned clinical BERT at classifying chief complaints (accuracy 0.882, F1 0.86, versus 0.59/0.53 for XGBoost and 0.63/0.59 for BlueBERT), while also standardizing social determinants of health with 0.95 accuracy and 0.96 weighted F1. Feeding those LLM-derived features into tree-based models lifts the best AUC from 0.74 to 0.76 and AUC-PR from 0.58 to 0.61. In the explainability layer, 99 of 100 sampled narratives were judged fully consistent with source records, population statistics, and SHAP-based feature rankings, with one minor numerical discrepancy and no moderate or severe errors. The paper's conclusion is that an integrated LLM-enhanced pipeline can make a predictive model both slightly more accurate and substantially more interpretable.","pith_inferences":["The outcome label is any ED return within 30 days, not a mental-health-specific return; if the intended clinical target is psychiatric relapse, re-labeling with mental/behavioral F-code return diagnoses would be the decisive test of whether the reported AUC gain measures the right thing.","The 99% explanation accuracy rests on 100 cases reviewed by two experts and, as the paper itself notes, was not tested for its effect on clinician trust or decisions; a prospective comparison against a SHAP-only arm would be a stronger test.","The prompt-based framework is task-agnostic, so it likely transfers to other high-volume ED phenotypes such as sepsis or asthma revisits, where free-text triage and social context also matter.","The paper's own stated limitations—single-center data, no formal clinician-trust evaluation, and computational latency—suggest that deployment would require site-specific prompt tuning and a workflow study before the narratives are used in live care."],"forward_implications":["Adding LLM-standardized chief complaints and social determinants of health lifts the best model's AUC from 0.74 to 0.76 and AUC-PR from 0.58 to 0.61, so text enrichment gives consistent but modest gains across model families.","Chief complaint classification via ten-shot prompting (accuracy 0.882, F1 0.86) beats both bag-of-words classifiers and a fine-tuned clinical BERT, indicating useful signal can be extracted without task-specific fine-tuning.","The explainability module ties each narrative to SHAP rankings and population statistics, and the 99-of-100 expert-reviewed accuracy indicates the generated explanations track the model's actual reasoning rather than free-standing text generation.","Because the same LLM that standardizes input text also writes output narratives, the framework adds interpretability without a second, separately trained explanation system.","The small size of the predictive gain implies the framework's main practical value is interpretability plus feature harmonization, not a large jump in raw prediction accuracy."],"supporting_citations":[{"why":"This taxonomy supplies the five chief-complaint categories (Pain, Psychiatric, Injury, Infection, Unclear) that the LLM classifier is built on.","marker":"[33]"},{"why":"This supplies the gradient-boosted tree algorithm used as the main predictive model and as a chief-complaint classification baseline.","marker":"[35]"},{"why":"This supplies the LLaMA 3 8B model used for few-shot clinical text classification and narrative explanation generation.","marker":"[44]"},{"why":"This supplies the SHAP attribution method whose per-feature contributions are the input to the LLM explanation module.","marker":"[45]"},{"why":"This documents the limited interpretability of existing ED-return machine learning models, the gap the proposed framework targets.","marker":"[10]"},{"why":"This provides the clinical BERT baseline that the few-shot LLM chief-complaint classifier must outperform.","marker":"[42]"}],"fun_headline_variants":["LLM text features nudge ED-return AUC to 0.76 and narrate SHAP","10-shot LLM beats clinical BERT on ED text, then explains model","LLM does triage, SDoH, and SHAP narration—99% accurate","LLM+ML: AUC 0.74→0.76, explanations 99% accurate","LLM text lifts ED-return AUC to 0.76; SHAP stories 99% accurate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that 'return to the ED within 30 days' can stand in for 'return for a mental-health reason'; the paper uses the all-cause definition (Section 3.2), so if most returns are for unrelated medical complaints, the model may be predicting general care-seeking rather than psychiatric relapse.","fun_headline_variants_meta":{"raw":{"variants":["LLM text features nudge ED-return AUC to 0.76 and narrate SHAP","10-shot LLM beats clinical BERT on ED text, then explains model","LLM does triage, SDoH, and SHAP narration—99% accurate","LLM+ML: AUC 0.74→0.76, explanations 99% accurate","LLM text lifts ED-return AUC to 0.76; SHAP stories 99% accurate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001969,"raw_usage":{"total_tokens":7788,"prompt_tokens":1135,"completion_tokens":6653,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":751,"completion_tokens_details":{"reasoning_tokens":6533}},"tokens_in":751,"tokens_out":6653,"duration_ms":50416,"temperature":1.0,"reasoning_tokens":6533,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:22:27.341619+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the analysis with the outcome restricted to returns whose discharge diagnosis is a mental or behavioral F-code, or to returns linked to the index psychiatric complaint; if the LLM-feature gain in AUC (0.74 to 0.76) and AUC-PR (0.58 to 0.61) shrinks or disappears, the reported improvement is driven by general ED utilization rather than mental-health relapse. A second check is to give a new panel of clinicians the 100 generated narratives and ask them to flag any numerical inconsistency, testing whether the 99% accuracy survives outside the original reviewing pair.","supporting_citations":[{"cited_title":"I can't stop vomiting","cited_arxiv_id":null,"evidence_quote":"This documents the limited interpretability of existing ED-return machine learning models, the gap the proposed framework targets."}],"review_version":1}