Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

A Multi-Stage Large Language Model Framework for Extracting Suicide-Related Social Determinants of Health

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a three-stage zero-shot LLM framework—retrieve relevant sentences, verify their relevance, then classify—can extract rare suicide-related social determinants of health from death investigation notes more accurately th

desk verdict Useful applied-NLP framework with a promising three-stage design, but the transparency numbers are built on the same data used for tuning, so the explainability claim needs a fresh held-out evaluation. read the letter →

arxiv 2508.05003 v1 pith:U7BTBV5W submitted 2025-08-07 cs.CL cs.AI

Song Wang , Yishu Wei , Haotian Ma , Max Lovitt , Kelly Deng , Yuan Meng , Zihan Xu , Jingze Zhang
show 5 more authors
Yunyu Xiao Ying Ding Xuhai Xu Joydeep Ghosh Yifan Peng
This is my paper
classification cs.CLcs.AI
keywords socialdeterminantsofhealthsuicidepreventionlargelanguagemodelszero-shotextractionclinicalNLPexplainabilityrelevanceverificationNVDRS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a zero-shot multi-stage LLM pipeline—sentence retrieval, relevance verification, and final classification—can extract rare suicide-related social determinants of health from unstructured death investigation notes better than fine-tuned models or single-pass LLM prompting, and that the intermediate stages make the model's reasoning inspectable. A sympathetic reader would care because suicide-related SDoH factors are long-tailed: the rare ones are hardest to detect by conventional classifiers, yet they may be the most clinically urgent. The paper reports that on 10 infrequent factors the framework's average F-1 beat fine-tuned BioBERT by 17.7%, GPT-3.5-turbo End-to-End by 4.8%, and GPT-3.5-turbo Chain-of-Thought by 8.8%, while also providing verified sentence-level evidence that can be shown to annotators or clinicians.

What carries the argument

The mechanism is a pipeline of three prompted decision stages plus an optional small fine-tuned verifier. First, NLTK splits the narrative and a GPT-3.5-turbo prompt selects sentences matching the target factor's definition. Second, a second model—GPT-3.5-turbo or fine-tuned FLAN-T5-base—independently verifies each retrieved sentence's relevance, filtering false positives. Third, the verified sentences are passed to the final extraction prompt, which returns True/False for whether the factor occurred within two weeks. The load-bearing idea is that constraining the model to produce and then re-check explicit textual evidence forces it to attend to rare factors and yields the verified sentence

What would settle it

Annotate a fresh set of sentences from NVDRS notes after freezing all prompts and the fine-tuned FLAN-T5 weights, then measure stage-2 verification accuracy against that set. If the fine-tuned verifier's accuracy no longer clearly beats the raw stage-1 retrieval accuracy, the relevance-verification and explainability claims are not established.

Watch

Extended reading notes

Core claim

The central claim is that decomposing zero-shot LLM extraction into three stages—(1) retrieve sentences plausibly relevant to a target SDoH factor, (2) have a separate model verify that relevance, (3) classify whether the factor occurred within two weeks before the suicide—improves accuracy and transparency relative to single-pass prompting or fine-tuned encoders. On the 10 infrequent SDoH factors, the framework's average F-1 exceeded fine-tuned BioBERT by 17.7%, GPT-3.5-turbo End-to-End by 4.8%, and GPT-3.5-turbo Chain-of-Thought by 8.8%, and it won on 9 of the 10 factors. On 8 frequent factors, the zero-shot pipeline still beat fine-tuned BioBERT by 4.0% average F-1 without ever being fine

Load-bearing premise

The results rest on the assumption that the curated 655-sentence gold-standard set was a true held-out test set, untouched by prompt selection and FLAN-T5 model selection; if that assumption fails, the reported relevance-accuracy and explainability gains are not independent measurements.

Editorial extensions

If this is right

  • For 9 of 10 infrequent SDoH factors, the three-stage zero-shot framework achieved higher F-1 than fine-tuned BioBERT, GPT-3.5-turbo End2End, and GPT-3.5-turbo CoT, with average gains of 17.7%, 4.8%, and 8.8% respectively.
  • The same zero-shot pipeline, without task-specific fine-tuning, beat fine-tuned BioBERT by 4.0% average F-1 on 8 frequent SDoH factors, suggesting reuse across factors could be cheap.
  • The relevance-verification stage improved intermediate evidence accuracy from 59.3% to 73.1%, and a fine-tuned FLAN-T5-base verifier reached 86.6% accuracy at lower inference cost than calling a large model twice.
  • If the model's verified evidence is shown to expert annotators, the pilot study indicates annotation time drops by roughly 62 seconds per incident with no loss in accuracy, pointing toward AI-assisted abstraction workflows.
  • The comparison with DeepSeek-R1 indicates a division of labor: reasoning models give higher precision, while the multi-stage framework gives higher recall, so deployment choices could depend on whether false positives or missed cases are more costly in a given surveillance setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the relevance-verification benefit replicates on fully held-out data, the same retrieve-verify-extract scaffold could be adapted to other long-tailed clinical extraction tasks—adverse events, elder abuse indicators, or housing instability—with minimal per-task prompt engineering and a small fine-tuned verifier.
  • The verified sentences provide a natural audit trail: a public-health abstractor could be shown exactly which sentence triggered a positive label, turning the model into an evidence-locating aid rather than an unexplained binary classifier.
  • An untested extension would be to use a stronger or reasoning-based LLM as the verifier in stage 2; the paper does not report whether a more powerful verifier would combine its precision advantage with the framework's recall advantage.
  • Because the two-week temporal restriction is encoded only in the final prompt, a testable next step would isolate temporal-reasoning errors explicitly—for example, by measuring how often the model correctly separates a factor that is mentioned but occurred earlier than two weeks before the incident.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes a multi-stage zero-shot LLM framework for extracting suicide-related social determinants of health (SDoH) factors from NVDRS death-investigation narratives. The pipeline consists of context retrieval, relevance verification (including a fine-tuned FLAN-T5-base examiner), and final SDoH factor extraction. The authors compare the framework with fine-tuned BioBERT, GPT-3.5-turbo End2End, GPT-3.5-turbo CoT, and DeepSeek-R1, reporting F-1 improvements on 10 infrequent and 8 frequent SDoH factors. They also evaluate the accuracy of the intermediate retrieval/verification stages as an explainability proxy, and they run a pilot user study to test whether the intermediate context helps human annotators. The central claims are that the staged design improves extraction accuracy over the baselines and that the intermediate outputs provide transparent and verifiable explanations.

Significance. If the reported results hold, the framework is of practical interest: it targets rare, long-tailed SDoH factors, offers a cheaper small-model verification stage, and attempts to make LLM predictions more transparent through explicit intermediate evidence. The paper provides a detailed architecture, prompts, data-preparation protocol, and supplementary per-factor scores, which are useful for replication. The main extraction comparisons are plausible and the pilot user study, despite its acknowledged within-subject limitation, is a valuable addition. However, the central explainability/transparency result is undermined by the use of the same curated gold set for prompt selection, FLAN-T5 checkpoint selection, and final evaluation. The extraction F-1 comparisons also lack confidence intervals and significance tests, so the magnitude of the claimed improvements is not fully established. The core idea is sound and the problems are addressable with a proper held-out evaluation protocol.

major comments (4)
  1. [§2.1.1, §2.1.2, §2.3, §3.4] The same 655-sentence gold set is used in three incompatible roles: §2.1.1 selects the context-retrieval prompt by F-1 on this set; §2.1.2 selects the fine-tuned FLAN-T5 examiner by F-1 'on the curated test set'; and §2.3/§3.4 report final stage-1/stage-2 accuracy on it. The reported improvement from 59.3% to 73.1% and the 86.6% FLAN-T5 accuracy are therefore selection-on-test estimates, not unbiased measurements. Since §2.6 defines explainability as the correctness of these intermediate outputs, the transparency benefit is not established. Please re-run with a true held-out set or nested cross-validation, and report development-set and test-set scores separately.
  2. [§2.1.2] No train/validation/test split is described for the FLAN-T5 fine-tuning. The text says the model is 'evaluated on the curated test set using F-1 scores, and the fine-tuned model with the highest F-1 score on the test set was selected as the examiner model.' Selecting a checkpoint on the same labels later used for reporting is overfitting to the evaluation set. Even if the FLAN-T5 training data were disjoint, the checkpoint selection invalidates the reported accuracy. Please specify the training/validation split and, ideally, freeze the examiner before touching the final test set.
  3. [§3.2 and Supplementary Tables 2–3] The headline improvements (e.g., 4.8% average F-1 gain over GPT-3.5-turbo End2End on infrequent factors) are reported without confidence intervals, bootstrap estimates, or significance tests. With 300 positive and 300 negative instances per factor for most comparisons, a 4.8% average difference may not be significant. Since the LLM baselines are stochastic, repeated sampling or at least paired bootstrap CIs are needed to support the claim that the framework 'outperformed all baseline methods in 9 out of 10 infrequent factors.'
  4. [§2.3 and §3.1] The 160 death-investigation notes used to build the 655-sentence relevance gold set appear to be a 'newly curated test set,' but the manuscript does not state whether these 160 notes overlap the balanced factor-extraction test set described in §2.3 and Supplementary Table 1. If the same notes contribute both relevance annotations and extraction labels, the extraction F-1 comparisons are also contaminated. Please explicitly confirm non-overlap, or evaluate with non-overlapping note sets.
minor comments (5)
  1. [§2.1.1 vs §2.3] Terminology is inconsistent: §2.1.1 refers to a 'curated gold standard development set,' while §2.3 calls the same 655-sentence resource a 'curated test set.' Use one term and clarify its role.
  2. [§3.4] The phrase 'improved overall relevant context retrieval accuracy from 59.3% to 73.1%, representing a 13.8% improvement' is ambiguous: 13.8 percentage points is not 13.8% relative improvement. Similarly, 'accuracy increased by 31%, reaching 86.6%' from 55.8% mixes absolute percentage points with relative increase. Please report both clearly.
  3. [Supplementary Table 4] The DeepSeek-R1 subset is very small for some factors (e.g., Other Addiction has only 3 positive and 1 negative instances). The per-factor F-1 scores in Figure 6 should be interpreted with extreme caution, and the sample sizes should be printed on the figure or stated in the caption.
  4. [§3.5 and Figure 8] The pilot user study reports an accuracy difference of 81.55% vs 83.33% and a time saving of 62.39 seconds, but no statistical tests or effect sizes are reported for these comparisons. Given the small sample (n=6, within-subject), please provide at least paired tests and confidence intervals.
  5. [Code Availability] The paper states code 'will be made publicly available,' but the repository is not yet accessible at review time. Please provide a working link or a complete appendix with prompts and fine-tuning details to support reproducibility.

Circularity Check

2 steps flagged · score 6.0 of 10

Same 655-sentence gold set is used for prompt selection, FLAN-T5 model selection, and final relevance-verification evaluation (§2.1.1, §2.1.2, §2.3), so the transparency/accuracy gains reported in §3.4 are selection-on-test estimates.

  1. fitted input called prediction [Section 2.1.1 and Section 2.3; results reported in Section 3.4]
    "we tested several variations and evaluated their performance on a curated gold standard development set, using F-1 scores ... The prompt that achieved the highest F-1 score was chosen for use in downstream experiments. ... This curated test set was then used to evaluate the effectiveness of both the context retrieval and relevance verification modules."

    The 'development set' in §2.1.1 is the same 655-sentence 'curated test set' described in §2.3. Choosing the context-retrieval prompt by highest F-1 on the exact sentences later used to report the 59.3% to 73.1% accuracy improvement means the stage-1 accuracy is a selection-maximized estimate, not an independent measurement of retrieval quality. The reported gain partly reflects fitting the prompt to the evaluation set, so the claimed explainability improvement is not established by these numbers.

  2. fitted input called prediction [Section 2.1.2 and Section 3.4]
    "The model was evaluated on the curated test set using F-1 scores, and the fine-tuned model with the highest F-1 score on the test set was selected as the examiner model. ... After fine-tuning, the accuracy increased by 31%, reaching 86.6%."

    The reported 86.6% accuracy is the accuracy of a FLAN-T5 checkpoint selected for the highest F-1 on the same 655-sentence curated test set used for final evaluation. Selecting the best checkpoint on the test set and then reporting accuracy on that test set is selection-on-test; the 55.8% to 86.6% gain is not an unbiased estimate of the fine-tuned model's verification accuracy. This directly undermines the transparency claim, since explainability is defined as the correctness of these intermediate outputs.

full rationale

The paper's central transparency/explainability claim is operationalized as the accuracy of the intermediate context-retrieval and relevance-verification modules (§2.6). Those accuracy numbers are not independent estimates: the same 655-sentence gold set is used (a) to select the context-retrieval prompt by F-1 (§2.1.1), (b) to select the fine-tuned FLAN-T5 checkpoint by test F-1 (§2.1.2), and (c) as the evaluation set for the stage-1/stage-2 and fine-tuned accuracy figures reported in §3.4 (59.3%→73.1% and 55.8%→86.6%). Because selection on the test set precedes the reported evaluation, these improvements are partly forced by construction rather than independent evidence. The factor-extraction F-1 comparisons against BioBERT, GPT-3.5-turbo, and DeepSeek-R1 use a separate balanced extraction test set and are less directly compromised; those results can stand on their own. Overall, partial circularity: the framework's headline accuracy/transparency gains reduce in part to fitting on the evaluation set.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The framework does not introduce physical entities or new mathematical postulates. The central empirical claims rest on the NVDRS data and the gold-standard relevance annotations. No numeric free parameters are reported for the zero-shot LLM stages; the fine-tuned FLAN-T5 involves trained weights, but the more serious tuning issue is the selection of prompts and of the FLAN-T5 checkpoint on the evaluation set itself, which is captured in red flags and the circularity score.

assumptions (3)
  • domain assumption The NVDRS coding manual's two-week temporal window is the correct target for factor extraction.
    Section 2.1.3 prompts the LLM to code factors that occurred 'within the two weeks before the suicide incident' based on the manual definitions; the paper assumes this manual is valid ground truth.
  • domain assumption The curated human-verified relevance labels are a valid gold standard for evaluating context retrieval and verification.
    Section 2.3 creates the gold-standard dataset using two annotators plus adjudication; the paper assumes these labels are correct despite the evaluation leakage described in the report.
  • domain assumption Balanced test sets provide a meaningful evaluation of extraction performance despite the long-tailed real-world distribution.
    Section 2.3 samples balanced positives and negatives per factor; the reported F-1, precision, and recall are unweighted by real prevalence, and the paper treats these as representative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Multi-Stage Large Language Model Framework for Extracting Suicide-Related Social Determinants of Health." pith.science (2026). https://pith.science/paper/U7BTBV5W

@misc{pith2026250805003,
  author       = {Pith},
  title        = {Pith review of: A Multi-Stage Large Language Model Framework for Extracting Suicide-Related Social Determinants of Health},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U7BTBV5W}},
  note         = {Machine review of arXiv:2508.05003}
}
read the original abstract

Background: Understanding social determinants of health (SDoH) factors contributing to suicide incidents is crucial for early intervention and prevention. However, data-driven approaches to this goal face challenges such as long-tailed factor distributions, analyzing pivotal stressors preceding suicide incidents, and limited model explainability. Methods: We present a multi-stage large language model framework to enhance SDoH factor extraction from unstructured text. Our approach was compared to other state-of-the-art language models (i.e., pre-trained BioBERT and GPT-3.5-turbo) and reasoning models (i.e., DeepSeek-R1). We also evaluated how the model's explanations help people annotate SDoH factors more quickly and accurately. The analysis included both automated comparisons and a pilot user study. Results: We show that our proposed framework demonstrated performance boosts in the overarching task of extracting SDoH factors and in the finer-grained tasks of retrieving relevant context. Additionally, we show that fine-tuning a smaller, task-specific model achieves comparable or better performance with reduced inference costs. The multi-stage design not only enhances extraction but also provides intermediate explanations, improving model explainability. Conclusions: Our approach improves both the accuracy and transparency of extracting suicide-related SDoH from unstructured texts. These advancements have the potential to support early identification of individuals at risk and inform more effective prevention strategies.

Figures

Figures reproduced from arXiv: 2508.05003 by the authors.

Figure 1
Figure 1. Overview of the proposed multi-stage large language model framework for extracting suicide￾related SDoH factors. the sentences that are relevant to ‘{TARGET_SOCIAL_FACTOR}’ based on the provided definition. Format your output (a list of relevant sentences) in a valid JSON payload with one key ‘Relevant’.[/INST] Here is your input: [CONTEXT]{INPUT_REPORT}[/CONTEXT] [{TARGET_SOCIAL_FACTOR}]{FACTOR_DEFINITION}[/{TARGET… view at source ↗
Figure 2
Figure 2. Questionnaire. 2.8 Statistics and Reproducibility All statistical analyses in this study were performed to evaluate the performance of the proposed multi￾stage LLM framework and baseline models for suicide-related SDoH factor extraction. Model performance was assessed using standard metrics, including precision, recall, and F-1 score, computed on balanced test sets comprising both positive and negative instances for… view at source ↗
Figure 3
Figure 3. Frequency distribution of suicide-related SDoH factors, with the frequent factors shown in red and infrequent factors in blue. 3.2 Comparison with baseline models 3.2.1 Extracting infrequent suicide-related SDoH factors We first evaluated the performance of our framework in extracting 10 infrequent suicide-related SDoH factors ( [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Performance comparisons between our proposed multi-stage framework and three baseline mod￾els in extracting the 10 infrequent suicide-related SDoH factors from death investigation notes. a, F-1 score comparisons. b, Precision score comparisons. c, Recall score comparis…
Figure 5
Figure 5. Figure 5: Performance comparisons between our proposed multi-stage framework and three baseline mod￾els in extracting the 8 frequent suicide-related SDoH factors from death investigation notes. a, F-1 score comparisons. b, Precision score comparisons. c, Recall score comparisons…
Figure 6
Figure 6. Figure 6: Performance comparisons between our proposed multi-stage framework, baseline models, and the DeepSeek-R1 reasoning model in extracting suicide-related SDoH factors from death investigation notes. a, F-1 score comparisons. b, Precision score comparisons. c, Recall score…
Figure 7
Figure 7. Figure 7: Accuracy score comparisons between the context retrieval in stage 1 and the relevance verification in stage 2, to examine the effectiveness of our multi-stage design in the proposed framework. Detailed scores can be found in Supplementary [PITH_FULL_IMAGE:figures/full…
Figure 8
Figure 8. Figure 8: Pilot user studies for suicide-related SDoH factor annotation. a, user study survey quantitative response distributions. b, the annotation accuracy comparisons between the control arm and the intervention arm. c, the annotation time comparisons between the control arm …

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A Survey of Large Language Models for Perception and Measurement of Human Psychology

    cs.CY 2026-05 unverdicted novelty 5.0 of 10

    A survey proposing a three-pillar framework to evaluate LLMs as tools for measuring latent psychological constructs and reviewing applications in personality and mental health.

Reference graph

Works this paper leans on

37 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Social determinants of mental health

    World Health Organization and Calouste Gulbenkian Foundation. Social determinants of mental health. Geneva, World Health Organization, 2014

    work page 2014

  2. [2]

    Social determinants of health—an approach taken at cdc

    Karen Hacker, John Auerbach, Robin Ikeda, Celeste Philip, and Debra Houry. Social determinants of health—an approach taken at cdc. Journal of Public Health Management and Practice , 28:589–594, 09 2022. doi: 10.1097/PHH.0000000000001626

    work page doi:10.1097/phh.0000000000001626 2022

  3. [3]

    Morin, Natalie M

    Shuhan Liu, Samuel B. Morin, Natalie M. Bourand, Isabella L. DeClue, Gustavo E. Delgado, Jiahe Fan, Sabrina K. Foster, Maaz S. Imam, Coulter B. Johnston, Franklin B. Joseph, Yihao Lu, Ujjwal Sehrawat, Li Chun Su, Ketaki Tavan, Kelly L. Zhang, Xingruo Zhang, Loren Saulsberry, and Robert D. Gibbons. Social vulnerability and risk of suicide in us adults, 201...

    work page arXiv 2016

  4. [4]

    Extract- ing social determinants of health from electronic health records using natural language processing: a systematic review

    Braja G Patra, Mohit M Sharma, Veer Vekaria, Prakash Adekkanattu, Olga V Patterson, Benjamin Glicksberg, Lauren A Lepow, Euijung Ryu, Joanna M Biernacka, Al’ona Furmanchuk, Thomas J George, William Hogan, Yonghui Wu, Xi Yang, Jiang Bian, Myrna Weissman, Priya Wickramaratne, J John Mann, Mark Olfson, Thomas R Campion, Mark Weiner, and Jyotishman Pathak. Ex...

    2021

  5. [5]

    Allen, Cara McDonnell, Jean-Paul Russo, Jonathan Cummins, Joshua R

    Tanja Magoc, Katie S. Allen, Cara McDonnell, Jean-Paul Russo, Jonathan Cummins, Joshua R. Vest, and Christopher A. Harle. Generalizability and portability of natural language processing system to extract individual social risk factors. International Journal of Medical Informatics, 177:105115, 2023. ISSN 1386-5056. doi: https://doi.org/10.1016/j.ijmedinf.2...

    work page arXiv 2023

  6. [6]

    Zhang, Lingyun Shi, Russell Richie, Haixia Liu, Andrew Tseng, Wei Quan, Neal Ryan, David Brent, and Fuchiang R

    Sifei Han, Robert F. Zhang, Lingyun Shi, Russell Richie, Haixia Liu, Andrew Tseng, Wei Quan, Neal Ryan, David Brent, and Fuchiang R. Tsui. Classifying social determinants of health from unstructured electronic health records using deep learning-based natural language processing. Journal of Biomedi- cal Informatics, 127:103984, 2022. ISSN 1532-0464. doi: h...

    arXiv 2022

  7. [7]

    An nlp approach to identify sdoh-related circumstance and suicide crisis from death investigation narratives

    Song Wang, Yifang Dang, Zhaoyi Sun, Ying Ding, Jyotishman Pathak, Cui Tao, Yunyu Xiao, and Yifan Peng. An nlp approach to identify sdoh-related circumstance and suicide crisis from death investigation narratives. Journal of the American Medical Informatics Association, 30(8):1408–1417, 04 2023. ISSN 1527-974X. doi: 10.1093/jamia/ocad068

    doi:10.1093/jamia/ocad068 2023

  8. [8]

    Gabriel, Onkar Litake, Sierra Simpson, Brittany N

    Rodney A. Gabriel, Onkar Litake, Sierra Simpson, Brittany N. Burton, Ruth S. Waterman, and Al- varo A. Macias. On the development and validation of large language model-based classifiers for identifying social determinants of health. Proceedings of the National Academy of Sciences, 121(39): e2320716121, 2024. doi: 10.1073/pnas.2320716121

    doi:10.1073/pnas.2320716121 2024

Show all 37 references
  1. [9]

    Surveillance for violent deaths - national violent death reporting system, 48 states, the district of columbia, and puerto rico, 2020.MMWR Surveill

    Grace S Liu, Brenda L Nguyen, Bridget H Lyons, Kameron J Sheats, Rebecca F Wilson, Carter J Betz, and Katherine A Fowler. Surveillance for violent deaths - national violent death reporting system, 48 states, the district of columbia, and puerto rico, 2020.MMWR Surveill. Summ.,...

    2020

  2. [10]

    Explainable deep learning for efficient and robust pattern recognition: A survey of recent de- velopments

    Xiao Bai, Xiang Wang, Xianglong Liu, Qiang Liu, Jingkuan Song, Nicu Sebe, and Been Kim. Explainable deep learning for efficient and robust pattern recognition: A survey of recent de- velopments. Pattern Recognition , 120:108102, 2021. ISSN 0031-3203. doi: https://doi. org/10.1...

    2021

  3. [11]

    Peeking inside the black-box: A survey on explainable artifi- cial intelligence (xai)

    Amina Adadi and Mohammed Berrada. Peeking inside the black-box: A survey on explainable artifi- cial intelligence (xai). IEEE Access, 6:52138–52160, 2018. doi: 10.1109/ACCESS.2018.2870052

    2018

  4. [12]

    Interpreting black- 18 box models: A review on explainable artificial intelligence

    Vikas Hassija, Vinay Chamola, Atmesh Mahapatra, Abhinandan Singal, Divyansh Goel, Kaizhu Huang, Simone Scardapane, Indro Spinelli, Mufti Mahmud, and Amir Hussain. Interpreting black- 18 box models: A review on explainable artificial intelligence. Cognitive Computation, 16, 08 ...

    2023 doi

  5. [13]

    Rethinking in- terpretability in the era of large language models, 2024

    Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. Rethinking in- terpretability in the era of large language models, 2024. URLhttps://arxiv.org/abs/2402. 01761

    2024

  6. [14]

    Large language models to identify social determinants of health in electronic health records

    Marco Guevara, Shan Chen, Spencer Thomas, Tafadzwa L Chaunzwa, Idalid Franco, Benjamin H Kann, Shalini Moningi, Jack M Qian, Madeleine Goldstein, Susan Harper, Hugo J W L Aerts, Paul J Catalano, Guergana K Savova, Raymond H Mak, and Danielle S Bitterman. Large language models ...

    2024

  7. [15]

    Sdoh- gpt: Using large language models to extract social determinants of health (sdoh), 2024

    Bernardo Consoli, Xizhi Wu, Song Wang, Xinyu Zhao, Yanshan Wang, Justin Rousseau, Tom Hartvigsen, Li Shen, Huanmei Wu, Yifan Peng, Qi Long, Tianlong Chen, and Ying Ding. Sdoh- gpt: Using large language models to extract social determinants of health (sdoh), 2024. URL https://a...

    2024 arXiv

  8. [16]

    Nltk: the natural language toolkit, 2006

    Steven Bird. Nltk: the natural language toolkit, 2006

    2006

  9. [17]

    Few-shot self-rationalization with natural language prompts

    Ana Marasovic, Iz Beltagy, Doug Downey, and Matthew Peters. Few-shot self-rationalization with natural language prompts. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Findings of the Association for Computational Linguistics: NAACL 2022 ...

    2022

  10. [18]

    Large language models are better reasoners with self-verification, 2023

    Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification, 2023. URL https: //arxiv.org/abs/2212.09561

    2023 arXiv

  11. [19]

    Self-verification improves few-shot clinical information extraction, 2023

    Zelalem Gero, Chandan Singh, Hao Cheng, Tristan Naumann, Michel Galley, Jianfeng Gao, and Hoifung Poon. Self-verification improves few-shot clinical information extraction, 2023. URL https://arxiv.org/abs/2306.00024

    2023 arXiv

  12. [20]

    Fine-tuned ’small’ llms (still) significantly outperform zero-shot generative ai models in text classification, 2024

    Martin Juan Jos ´e Bucher and Marco Martini. Fine-tuned ’small’ llms (still) significantly outperform zero-shot generative ai models in text classification, 2024. URL https://arxiv.org/abs/ 2406.08660

    2024 arXiv

  13. [21]

    Human still wins over llm: An empirical study of active learning on domain-specific annotation tasks, 2023

    Yuxuan Lu, Bingsheng Yao, Shao Zhang, Yun Wang, Peng Zhang, Tun Lu, Toby Jia-Jun Li, and Dakuo Wang. Human still wins over llm: An empirical study of active learning on domain-specific annotation tasks, 2023. URL https://arxiv.org/abs/2311.09825

    2023 arXiv

  14. [22]

    Language models for text classification: Is in- context learning enough?, 2024

    Aleksandra Edwards and Jose Camacho-Collados. Language models for text classification: Is in- context learning enough?, 2024. URL https://arxiv.org/abs/2403.17661

    2024 arXiv

  15. [23]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinso...

    2022 arXiv

  16. [24]

    Gpt-3.5-turbo model, 2023

    OpenAI. Gpt-3.5-turbo model, 2023. URL https://platform.openai.com/docs/ models/gpt-3-5

    2023

  17. [25]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903. 19

    2023 arXiv

  18. [26]

    Biobert: a pre-trained biomedical language representation model for biomedical text mining

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 09 2019. ISSN 1367-4803. doi: 10.1093/bioinformatics/btz682

    2019 doi

  19. [27]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  20. [28]

    Sandra G. Hart. Nasa-task load index (nasa-tlx); 20 years later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 50(9):904–908, 2006. doi: 10.1177/154193120605000909. URL https://doi.org/10.1177/154193120605000909

    2006 doi

  21. [29]

    Kortum Aaron Bangor and James T

    Philip T. Kortum Aaron Bangor and James T. Miller. An empirical evaluation of the system usability scale. International Journal of Human–Computer Interaction , 24(6):574–594, 2008. doi: 10.1080/ 10447310802205776. URL https://doi.org/10.1080/10447310802205776

    2008 doi

  22. [30]

    Hoffman, Shane T

    Robert R. Hoffman, Shane T. Mueller, Gary Klein, and Jordan Litman. Measures for explainable ai: Explanation goodness, user satisfaction, mental models, curiosity, trust, and human-ai perfor- mance. Frontiers in Computer Science , 5, 2023. ISSN 2624-9898. doi: 10.3389/fcomp.20...

    2023

  23. [31]

    Bisantz Jiun-Yin Jian and Colin G

    Ann M. Bisantz Jiun-Yin Jian and Colin G. Drury. Foundations for an empirically determined scale of trust in automated systems. International Journal of Cognitive Ergonomics, 4(1):53–71, 2000. doi: 10. 1207/S15327566IJCE0401\ 04. URL https://doi.org/10.1207/S15327566IJCE0401_ 04

    doi

  24. [32]

    Interrater reliability: The kappa statistic

    Mary McHugh. Interrater reliability: The kappa statistic. Biochemia medica : ˇcasopis Hrvatskoga druˇstva medicinskih biokemiˇcara / HDMB, 22:276–82, 10 2012. doi: 10.11613/BM.2012.031. 20

    2012 doi

  25. [33]

    Sys- tematic design and data-driven evaluation of social determinants of health ontology (sdoho)

    Yifang Dang, Fang Li, Xinyue Hu, Vipina K Keloth, Meng Zhang, Sunyang Fu, Muhammad F Amith, J Wilfred Fan, Jingcheng Du, Evan Yu, Hongfang Liu, Xiaoqian Jiang, Hua Xu, and Cui Tao. Sys- tematic design and data-driven evaluation of social determinants of health ontology (sdoho)...

    2023 doi

  26. [34]

    Towards faithful model explanation in NLP: A survey

    Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch. Towards faithful model explanation in NLP: A survey. Computational Linguistics, 50(2):657–723, June 2024. doi: 10.1162/coli a 00511. URL https://aclanthology.org/2024.cl-2.6/

    2024 doi

  27. [35]

    Explainability for large language models: A survey

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey. ACM Trans. Intell. Syst. Technol., 15(2), February 2024. ISSN 2157-6904. doi: 10.1145/3639372. URL https: //...

    2024 doi

  28. [36]

    Gary Charness, Uri Gneezy, and Michael A. Kuhn. Experimental methods: Between-subject and within-subject design. Journal of Economic Behavior & Organization, 81(1):1–8, 2012. ISSN 0167-

    2012

  29. [2681]

    URL https://www.sciencedirect

    doi: https://doi.org/10.1016/j.jebo.2011.08.009. URL https://www.sciencedirect. com/science/article/pii/S0167268111002289. 21 Supplementary Table 1: SDoH factor statistics in the training and test sets. Training Test Testing DeepSeek-R1 Factor Positive Negative Positive Negati...

    2011 doi

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.