Pith. sign in

REVIEW 3 major objections 5 minor 41 references

Multi-Label Classification with Generative AI Models in Healthcare: A Case Study of Suicidality and Risk Factors

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned GPT-3.5 and guideline-prompted GPT-4.5 can classify multiple co-occurring suicide risk factors from psychiatric intake notes, with GPT-4.5 more balanced on rare label combinations.

desk verdict A solid clinical NLP evaluation with a useful label-set metric, but the headline claim about GPT-4.5's rare-label advantage rests on a few dozen notes and an unexplained GPT-4.5-Tune. read the letter →

arxiv 2507.17009 v1 pith:R6DQP3E2 submitted 2025-07-22 cs.CL cs.IRq-bio.QM

Ming Huang , Zehan Li , Yan Hu , Wanjing Wang , Andrew Wen , Scott Lane , Salih Selek , Lokesh Shahani
show 4 more authors
Rodrigo Machado-Vieira Jair Soares Hua Xu Hongfang Liu
This is my paper
classification cs.CLcs.IRq-bio.QM
keywords multi-labelclassificationgenerativelargelanguagemodelssuicidalityclinicalnoteselectronichealthrecordslabel-setevaluationerroranalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Suicide risk is rarely a single yes-or-no; a patient's notes can mention suicidal ideation, a past suicide attempt, exposure to someone else's suicide, and non-suicidal self-injury, in any combination. The paper tries to show that generative large language models can assign all of these labels at once from unstructured psychiatric intake notes, rather than running separate binary screens. On a 500-note annotated corpus, fine-tuned GPT-3.5 reached 0.94 partial-match accuracy and 0.91 F1, while GPT-4.5 with a guideline prompt was close on overall accuracy and clearly better on rare or minority label combinations, which the authors read as more balanced behavior. The paper also contributes evaluation tools — label-set-level exact- and partial-match metrics and a multi-label confusion matrix — that turn partial successes and systematic error patterns into something measurable. If the approach holds, it offers a path from narrative EHR text to structured tables for suicide research and intervention planning.

What carries the argument

The load-bearing object is the label powerset: with four binary labels (SI, SA, ES, NSSI), each note maps to one of 16 possible label sets, and evaluating at this level turns multi-label classification into multi-class classification over the 16 combinations. Around that object the paper builds two pieces of machinery. The first is a family of label-set-level metrics, including a partial-match score that credits a prediction for the labels it gets right inside a partially wrong combination, with precision and recall that split the penalty equally between hallucinated extra labels and omitted true labels. The second is a 16-by-16 confusion matrix over the label powerset, with binary and semantic codes on both axes, which localizes each error as a hallucination, an omission, or a mix of both at the model, label-set, and single-label level.

What would settle it

Collect or resample enough data so each rare label set has at least 100 instances, rerun the same five model variants, and compute macro-averaged label-set F1 with bootstrap confidence intervals. If GPT-4.5-Guide's 10-point lead over GPT-3.5-Tune shrinks below the interval or flips sign, the paper's balanced-robustness claim fails. A cheaper first check is to bootstrap the existing 500 predictions and ask whether the macro-F1 difference is distinguishable from noise.

Watch

Extended reading notes

Core claim

The central claim is that a single generative-model pass can do multi-label classification of suicidality-related factors from psychiatric notes as well as or better than prior approaches, and that how you measure it changes which model wins. Fine-tuned GPT-3.5 has the best instance-level scores on the 500-note corpus, with 0.78 exact-match and 0.94 partial-match accuracy and 0.91 micro-F1 on instances, labels, and label sets. GPT-4.5 with guided prompting trails by only 0.01 on accuracy but beats GPT-3.5-Tune by 0.10–0.11 on macro-averaged label-set F1, which the authors interpret as more balanced performance on rare combinations such as NSSI-only or exposure-to-suicide sets. Error analysis through a 16-by-16 confusion matrix over the label powerset reveals that the model tends to over-label rather than under-label: 80 of 86 errors in the top-right corner are hallucinations, SA-only notes are frequently upgraded to SI&SA, and NSSI-only notes often pick up a hallucinated SA label.

Load-bearing premise

The load-bearing premise is that the 500-note sample, which contains only about 11 notes with non-suicidal self-injury as the sole label and 22 notes mentioning exposure to suicide, is large enough to make the rare-label-set comparisons stable; if those tiny subgroups are noisy, the claimed GPT-4.5 advantage could disappear.

Editorial extensions

If this is right

  • A single generative model pass can replace separate binary screens for ideation, attempts, exposure, and self-injury, turning psychiatric intake notes directly into structured labels.
  • When rare label combinations are clinically important, GPT-4.5-Guide is the preferred model even though fine-tuned GPT-3.5 has slightly higher overall accuracy, because the macro-F1 gap of 10–11 points reflects better rare-set behavior.
  • Systematic error patterns imply that predicted suicide-attempt labels need a closer look: nearly half of SA-only notes are mislabeled as SI&SA, and NSSI-only notes tend to acquire a hallucinated SA label.
  • The label-set-level exact- and partial-match metrics and the multi-label confusion matrix are transferable to other multi-label clinical text tasks, giving researchers a shared way to compare generative models on rare combinations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test: add a post-hoc correction that strips hallucinated SA labels from NSSI-only predictions and see whether GPT-4.5-Guide's exact-match accuracy then catches up to GPT-3.5-Tune.
  • The 10–11 point macro-F1 advantage is computed from very small subgroups (11 NSSI-only notes, 22 exposure-positive notes), so I would not treat it as stable until it is replicated on a larger or multi-hospital corpus.
  • The same powerset confusion matrix could be applied to other clinical multi-label extraction tasks, such as social-determinants-of-health coding, to check whether the over-labeling bias is a general property of generative models or specific to suicidality notes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper develops a generative-LLM multi-label classification pipeline for detecting four suicidality-related factors (SI, SA, ES, NSSI) in 500 psychiatric initial evaluation notes. It compares GPT-3.5 (zero-shot, guided, fine-tuned), GPT-4-Guide, and GPT-4.5-Guide, using 5-fold cross-validation with 3 repeats. It proposes label-set-level exact/partial match metrics and a powerset confusion matrix. Main reported findings are that fine-tuned GPT-3.5 achieves the best overall accuracy (0.94 partial match) and that GPT-4.5-Guide achieves the best macro-F1 (0.89 partial match), which the authors interpret as more balanced and robust performance across rare label sets.

Significance. The study addresses a real clinical modeling gap by treating suicidality detection as multi-label rather than binary, with careful annotation (IAA κ=0.95), repeated cross-validation, and a transparent error analysis that yields actionable clinical error patterns (e.g., SA-only notes misclassified as SI&SA). The proposed metrics are definitions applied to observed predictions rather than fitted parameters, so there is no circularity in the evaluation. If the rare-label superiority claim is statistically substantiated, the work would be a useful reference for generative multi-label classification in EHRs. The significance, however, depends on resolving the model-label inconsistency and the lack of inferential statistics detailed below.

major comments (3)
  1. [Results, Label Set Performance Analysis; Methods, Multi-label Classification] The Results and Discussion report a model 'GPT-4.5-Tune' that achieves the best F1 in 8 of 14 label sets (e.g., 1-0-1-0 and 0-0-0-0), and the Discussion attributes the best performance in 8 label sets to it. Yet the Methods section only describes fine-tuning GPT-3.5; no fine-tuning of GPT-4.5 is specified in the experiments. This is not a cosmetic inconsistency: the abstract's claim that 'GPT-4.5 with guided prompting showed superior performance across label sets' is contradicted by the Discussion's attribution of the best label-set performance to GPT-4.5-Tune. Please either add a complete description of the GPT-4.5 fine-tuning experiment (data, hyperparameters, folds) or relabel/remove these results and revise the corresponding conclusions.
  2. [Results, Model Performance Analysis; Discussion] The statement that GPT-4.5-Guide's macro-averaged F1 is 'significantly better' than GPT-3.5-Tune by 0.10–0.11 is not supported by any inferential statistic. No confidence intervals, bootstrap, or paired test is reported, despite the availability of 3 repeats across 5 folds. Because macro-F1 averages over the 14 label sets with equal weight, and rare label sets have very small denominators (e.g., 0-0-0-1 has only 11 true instances; ES-positive label sets total 22 instances), the claimed advantage may be driven by a handful of notes. Please report per-label-set variability and a note-level bootstrap or equivalent significance test before claiming that GPT-4.5-Guide is more balanced and robust on rare label sets.
  3. [Results, Label Set Performance Analysis] The label-set-level F1 scores for rare sets (e.g., 0-0-0-1, F1=0.49–0.59 for the best models) are based on 11 true instances, and the reported ±0.00 or ±0.02 repeat standard deviations do not capture sampling variability across different note samples; one or two changed predictions would move these F1 values by several points. Thus the conclusion that 'all models failed to perform well for 0-0-0-1' and that GPT-4.5-Guide is superior on rare sets is currently based on unstable estimates. Please qualify these claims or provide interval estimates that account for the small support.
minor comments (5)
  1. [Significance Statement] The Significance Statement says the study performs 'multi-class classification,' but the task is multi-label classification; please correct this.
  2. [Results, Label Set Performance Analysis] The exact-match and partial-match F1 formulas for label sets are defined only in the Methods; for readability, the Results should include a brief definition or a pointer to the relevant equations.
  3. [Methods, Performance Evaluation] The partial-match metrics are a form of example-based accuracy or Jaccard-style scoring; the paper should position them against existing multi-label evaluation metrics (e.g., in Zhang and Zhou 2013) rather than presenting them as entirely novel without contrast.
  4. [References] Reference 34 ('Fu, S. University of Minnesota (2021)') is incomplete; a full citation is needed.
  5. [General] No data or code availability statement is included; given the reproducibility emphasis of the paper, please add one.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all results are empirical evaluations against an independent gold standard, with no fitted parameter renamed as a prediction.

full rationale

This paper reports an empirical comparison of generative LLMs for multi-label classification of suicidality-related factors in psychiatric notes. There is no derivation chain in which an output is defined in terms of an input, and no fitted parameter is later relabeled as a prediction. Model performance is measured against independently annotated labels with reported inter-annotator agreement (Cohen's kappa 0.90-1.00), and the metrics, including the proposed label-set-level exact and partial match scores, are definitions applied to observed predictions and gold-standard labels; they do not feed back into any model constant or evaluation setting. The claim that GPT-4.5-Guide is more balanced and robust relies on computed macro-F1 values, which is an empirical claim rather than a circular one. The appearance of 'GPT-4.5-Tune' in the label-set results, despite the Methods describing fine-tuning only for GPT-3.5, is a reporting inconsistency or potential experimental omission, but it does not constitute circularity because it does not make any result equivalent to its own input. No load-bearing self-citation was found; the cited prior work of the same group is used as contextual comparison, not as the justification for the present results. Therefore the paper is self-contained against external benchmarks for circularity purposes, and the score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims are empirical and rest on data quality, sample size, and model stability assumptions rather than on fitted parameters or invented constructs. No free parameters are fit; the main fragility is the very small number of rare label-set instances.

assumptions (4)
  • domain assumption The 500 annotated IPE notes provide a sufficiently large and representative sample for stable MLC performance estimates, especially for rare label sets.
    The strongest comparative claim about performance on rare label sets rests on 11 NSSI-only and 22 ES-positive notes; these tiny counts make macro-F1 estimates high-variance.
  • domain assumption The two annotators' labels (kappa 0.95) with psychiatric adjudication are an accurate gold standard for the four SrF categories.
    All reported accuracies and F1 scores treat this manual labeling as ground truth; disagreements were settled by a 4-psychiatrist panel, but the adjudication process is not detailed.
  • domain assumption GPT-3.5, GPT-4, and GPT-4.5 behave as stable, well-defined functions across repeated API calls with the prompts and settings used.
    Reproducibility of the numbers depends on proprietary model versions and decoding parameters that are not frozen or fully specified in the paper.
  • domain assumption Document-level presence or absence of SI, SA, ES, and NSSI is the clinically appropriate target, and the four classes are mutually distinguishable as defined.
    The entire MLC evaluation assumes these four binary labels are the right semantic units and that the notes contain sufficient evidence to assign them.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Label Classification with Generative AI Models in Healthcare: A Case Study of Suicidality and Risk Factors." pith.science (2026). https://pith.science/paper/R6DQP3E2

@misc{pith2026250717009,
  author       = {Pith},
  title        = {Pith review of: Multi-Label Classification with Generative AI Models in Healthcare: A Case Study of Suicidality and Risk Factors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R6DQP3E2}},
  note         = {Machine review of arXiv:2507.17009}
}
read the original abstract

Suicide remains a pressing global health crisis, with over 720,000 deaths annually and millions more affected by suicide ideation (SI) and suicide attempts (SA). Early identification of suicidality-related factors (SrFs), including SI, SA, exposure to suicide (ES), and non-suicidal self-injury (NSSI), is critical for timely intervention. While prior studies have applied AI to detect SrFs in clinical notes, most treat suicidality as a binary classification task, overlooking the complexity of cooccurring risk factors. This study explores the use of generative large language models (LLMs), specifically GPT-3.5 and GPT-4.5, for multi-label classification (MLC) of SrFs from psychiatric electronic health records (EHRs). We present a novel end to end generative MLC pipeline and introduce advanced evaluation methods, including label set level metrics and a multilabel confusion matrix for error analysis. Finetuned GPT-3.5 achieved top performance with 0.94 partial match accuracy and 0.91 F1 score, while GPT-4.5 with guided prompting showed superior performance across label sets, including rare or minority label sets, indicating a more balanced and robust performance. Our findings reveal systematic error patterns, such as the conflation of SI and SA, and highlight the models tendency toward cautious over labeling. This work not only demonstrates the feasibility of using generative AI for complex clinical classification tasks but also provides a blueprint for structuring unstructured EHR data to support large scale clinical research and evidence based medicine.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 37 canonical work pages

  1. [1]

    End-to-end generative MLC pipeline with LLMs. We demonstrate that fine-tuned GPT-3.5 and guideline prompted GPT-4.5 can achieve top accuracy of 0.93-0.94 and F1 scores of 0.88–0.91 on a MLC task with four labels despite severe class imbalance

    work page

  2. [2]

    Label-set–level evaluation. Beyond conventional micro/macro metrics, we introduce exact- and partial-match label-set metrics that quantify performance on clinically meaningful label combinations, capturing errors that standard label-level statistics obscure

    work page

  3. [3]

    We extend the classical confusion matrix to the power- set setting, enabling granular inspection of hallucination (false-positive) versus omission (false- negative) patterns

    Multi-label confusion-matrix error analysis. We extend the classical confusion matrix to the power- set setting, enabling granular inspection of hallucination (false-positive) versus omission (false- negative) patterns. For instance, we reveal that nearly half of SA-only notes are mislabelled as SI & SA, spotlighting a systematic tendency to conflate idea...

    work page

  4. [4]

    Suicidal Ideation

    Blueprint for structuring narrative EHR data. Our workflow enables the conversion of unstructured clinical text into structured tables, facilitating large-scale observational studies and downstream causal analyses in healthcare research. Results In the following, we present our results on the comparison of GPT models with relevant training strategies for ...

    work page

  5. [5]

    Suicide worldwide in 2019: global health estimates, (2021)

    Organization, W.H. Suicide worldwide in 2019: global health estimates, (2021)

    work page 2021

  6. [6]

    World Health Organization. Suicide. (2024)

    work page 2024

  7. [7]

    Preventing suicide: A global imperative, (2014)

    Organization, W.H. Preventing suicide: A global imperative, (2014)

    work page 2014

  8. [8]

    Suicide Data and Statistics

    Centers for Disease Control and Prevention. Suicide Data and Statistics. (2024)

    work page 2024

Show all 41 references
  1. [9]

    & Pirkis, J

    Hawton, K. & Pirkis, J. Preventing suicide: a call to action. The Lancet Public Health 9, e825-e830 (2024)

    2024

  2. [10]

    Predicting suicide following self-harm: systematic review of risk factors and risk scales

    Chan, M.K.Y., et al. Predicting suicide following self-harm: systematic review of risk factors and risk scales. British Journal of Psychiatry 209, 277-283 (2016)

    2016

  3. [11]

    Association of suicidal behavior with exposure to suicide and suicide attempt: A systematic review and multilevel meta-analysis

    Hill, N.T., et al. Association of suicidal behavior with exposure to suicide and suicide attempt: A systematic review and multilevel meta-analysis. PLoS medicine 17, e1003074 (2020)

    2020

  4. [12]

    & Large, M.M

    McHugh, C.M., Corderoy, A., Ryan, C.J., Hickie, I.B. & Large, M.M. Association between suicidal ideation and suicide: meta-analyses of odds ratios, sensitivity, specificity and positive predictive value. BJPsych Open 5, e18 (2019)

    2019

  5. [13]

    Using weak supervision and deep learning to classify clinical notes for identification of current suicidal ideation

    Cusick, M., et al. Using weak supervision and deep learning to classify clinical notes for identification of current suicidal ideation. Journal of psychiatric research 136, 95-102 (2021)

    2021

  6. [14]

    Portability of natural language processing methods to detect suicidality from clinical text in US and UK electronic health records

    Cusick, M., et al. Portability of natural language processing methods to detect suicidality from clinical text in US and UK electronic health records. Journal of affective disorders reports 10, 100430 (2022)

    2022

  7. [15]

    Comparisons of deep learning and machine learning while using text mining methods to identify suicide attempts of patients with mood disorders

    Wang, X., et al. Comparisons of deep learning and machine learning while using text mining methods to identify suicide attempts of patients with mood disorders. Journal of affective disorders 317, 107-113 (2022)

    2022

  8. [16]

    Rawat, B.P.S., Kovaly, S., Pigeon, W.R. & Yu, H. Scan: Suicide attempt and ideation events dataset. in Proceedings of the conference. Association for Computational Linguistics. North American Chapter. Meeting, Vol. 2022 1029 (NIH Public Access, 2022)

    2022

  9. [17]

    & Hechter Rulin, C

    Xie, F., Ling Grant Deborah, S., Chang, J., Amundsen Britta, I. & Hechter Rulin, C. Identifying Suicidal Ideation and Attempt From Clinical Notes Within a Large Integrated Health Care System. The Permanente Journal 26, 85-93 (2022)

    2022

  10. [18]

    & Chan, L.F

    Nordin, N., Zainol, Z., Mohd Noor, M.H. & Chan, L.F. Suicidal behaviour prediction models using machine learning techniques: A systematic review. Artificial Intelligence in Medicine 132, 102395 (2022)

    2022

  11. [19]

    Large language models encode clinical knowledge

    Singhal, K., et al. Large language models encode clinical knowledge. Nature 620, 172-180 (2023)

    2023

  12. [20]

    Large language models in medicine

    Thirunavukarasu, A.J., et al. Large language models in medicine. Nature Medicine 29, 1930-1940 (2023)

    2023

  13. [21]

    Suicide Phenotyping from Clinical Notes in Safety-Net Psychiatric Hospital Using Multi-Label Classification with Pre-Trained Language Models

    Li, Z., et al. Suicide Phenotyping from Clinical Notes in Safety-Net Psychiatric Hospital Using Multi-Label Classification with Pre-Trained Language Models. in AMIA Summits Transl Sci Proc, Vol. In press (American Medical Informatics Association, Pittsburgh, PA, USA, 2025)

    2025

  14. [22]

    Large language models: A survey

    Minaee, S., et al. Large language models: A survey. arXiv preprint arXiv:2402.06196 (2024)

    2024 arXiv

  15. [23]

    & Rouhizadeh, M

    Zandbiglari, K., Kumar, S., Bilal, M., Goodin, A. & Rouhizadeh, M. Enhancing suicidal behavior detection in EHRs: A multi-label NLP framework with transformer models and semantic retrieval-based annotation. Journal of Biomedical Informatics 161, 104755 (2025)

    2025

  16. [24]

    Social determinants of health extraction from clinical notes across institutions using large language models

    Keloth, V.K., et al. Social determinants of health extraction from clinical notes across institutions using large language models. npj Digital Medicine 8, 1-13 (2025)

    2025

  17. [25]

    The interpersonal theory of suicide

    Van Orden, K.A., et al. The interpersonal theory of suicide. Psychological review 117, 575 (2010)

    2010

  18. [26]

    The interpersonal theory of suicide: A systematic review and meta- analysis of a decade of cross-national research

    Chu, C., et al. The interpersonal theory of suicide: A systematic review and meta- analysis of a decade of cross-national research. Psychological bulletin 143, 1313 (2017)

    2017

  19. [27]

    SMHD: a Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions

    Cohan, A., et al. SMHD: a Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions. 1485-1497 (Association for Computational Linguistics, Santa Fe, New Mexico, USA, 2018)

    2018

  20. [28]

    & Zhou, Z.-H

    Zhang, M.-L. & Zhou, Z.-H. A review on multi-label learning algorithms. IEEE transactions on knowledge and data engineering 26, 1819-1837 (2013)

    2013

  21. [29]

    & Tsang, I.W

    Liu, W., Wang, H., Shen, X. & Tsang, I.W. The emerging trends of multi-label learning. IEEE transactions on pattern analysis and machine intelligence 44, 7955- 7974 (2021)

    2021

  22. [30]

    & Kocev, D

    Bogatinovski, J., Todorovski, L., Džeroski, S. & Kocev, D. Comprehensive comparative study of multi-label classification methods. Expert Sys Appl 203, 117215 (2022)

    2022

  23. [31]

    Classifying patient portal messages using Convolutional Neural Networks

    Sulieman, L., et al. Classifying patient portal messages using Convolutional Neural Networks. Journal of biomedical informatics 74, 59-70 (2017)

    2017

  24. [32]

    Wu, X., Zhang, J. & Li, H. Text-to-table: A new way of information extraction. arXiv preprint arXiv:2109.02707 (2021)

    2021 arXiv

  25. [33]

    & Malik, M.K

    Nasar, Z., Jaffry, S.W. & Malik, M.K. Named entity recognition and relation extraction: State-of-the-art. ACM Computing Surveys (CSUR) 54, 1-39 (2021)

    2021

  26. [34]

    Clinical information extraction applications: a literature review

    Wang, Y., et al. Clinical information extraction applications: a literature review. Journal of biomedical informatics 77, 34-49 (2018)

    2018

  27. [35]

    Clinical concept extraction: a methodology review

    Fu, S., et al. Clinical concept extraction: a methodology review. Journal of biomedical informatics 109, 103526 (2020)

    2020

  28. [36]

    Improving large language models for clinical named entity recognition via prompt engineering

    Hu, Y., et al. Improving large language models for clinical named entity recognition via prompt engineering. J Am Med Inform Assoc 31, 1812-1820 (2024)

    2024

  29. [37]

    Structured information extraction from scientific text with large language models

    Dagdelen, J., et al. Structured information extraction from scientific text with large language models. Nature Communications 15, 1418 (2024)

    2024

  30. [38]

    University of Minnesota (2021)

    Fu, S. University of Minnesota (2021)

    2021

  31. [39]

    Meta-analysis of Cohen’s kappa

    Sun, S. Meta-analysis of Cohen’s kappa. Health Services and Outcomes Research Methodology 11, 145-163 (2011)

    2011

  32. [40]

    Language models are few-shot learners

    Brown, T.B. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)

    2020 arXiv

  33. [41]

    Gpt-4 technical report

    Achiam, J., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.