Pith. sign in

REVIEW 4 major objections 6 minor 69 references

SynthEHR-Eviction: Enhancing Eviction SDoH Detection with LLM-Augmented Synthetic EHR Data

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that an LLM-driven pipeline can generate synthetic EHR notes labeled for eviction and related social risks, and that fine-tuned open models trained on this data reach Macro-F1 of 88.8% for eviction and 90.3% for other…

desk verdict A genuinely useful synthetic dataset and pipeline for eviction-related SDoH extraction, but the headline claims overstate what the evaluation actually supports; worth engaging, with fixes. read the letter →

arxiv 2507.07421 v1 pith:7DZ5EKDP submitted 2025-07-10 cs.CL cs.AI

Zonghai Yao , Youxia Zhao , Avijit Mitra , David A. Levy , Emily Druhl , Jack Tsai , Hong Yu This is my paper
classification cs.CLcs.AI
keywords evictionsocialdeterminantsofhealthelectronicrecordssyntheticdatalargelanguagemodelspromptoptimizationclinicalNLPfine-grainedannotation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Eviction is a social determinant of health that appears in clinical notes but is almost never recorded in structured EHR fields, so it escapes detection in routine care. This paper tries to establish that a mostly synthetic training dataset, built by an LLM-driven pipeline with human verification, is enough to make open-weights models detect eviction and related housing risks from free-text notes. On human-validated test sets spanning synthetic, MIMIC, and PMC notes, fine-tuned Qwen2.5-7B and LLaMA-3.1-8B reach Macro-F1 of 88.8% for eviction subcategories and 90.3% for other social determinants, edging past a prompt-optimized GPT-4o baseline (87.8% and 87.3%) while cutting annotation time by more than 80%. If true, the paper offers a low-cost, reproducible path to eviction-risk screening from existing EHR text.

What carries the argument

The central mechanism is a two-stage augmenter–annotator loop. The augmenter is a label-specific LLM prompt optimized against expert feedback; it rewrites real MIMIC social-history sections so they exhibit a target eviction or SDoH class, with human verification filtering outputs until accuracy exceeds 90%. The annotator is a DSPy program trained on a small human-validated set, using chain-of-thought reasoning and BootstrapFewShotWithRandomSearch to produce both a label and a rationale. The fine-tuning recipe then mixes 70% synthetic notes with 30% real PMC notes and includes the reasoning traces, which the paper shows is what lets small open models transfer to long, narrative clinical text.

What would settle it

Take a corpus of real clinical notes known to contain eviction documentation (for example, an unmodified EHR sample with verified eviction mentions), run the fine-tuned Qwen2.5-7B model on it, and compare F1 against the reported 88.8%; also check whether any PMC training notes overlap the PMC test set. A large drop or detectable overlap would show the reported scores overstate deployment readiness.

Watch

Extended reading notes

Core claim

The paper's central claim is that eviction, a social determinant of health almost never coded in structured EHR fields, can be detected from clinical free text using models trained on a mostly synthetic dataset produced by its SynthEHR-Eviction pipeline. The pipeline separates generation from verification: label-specific LLM 'augmenters' rewrite real MIMIC social-history sections into eviction-relevant notes under expert feedback, and DSPy-optimized 'annotators' label the notes with 14 fine-grained categories plus chain-of-thought rationales. Fine-tuned open models, especially Qwen2.5-7B and LLaMA-3.1-8B, reach Macro-F1 0.888 for eviction subcategories and 0.903 for other SDoH categories on human-validated test sets spanning synthetic, MIMIC, and PMC notes, slightly exceeding the GPT-4o-APO baseline (0.878 and 0.873) while reducing human annotation time from over 266 hours to under 6.

Load-bearing premise

The evaluation relies on test notes that human experts rewrote to inject eviction content, and the training set includes PMC notes drawn from the same source as the PMC test notes, so the reported F1 may not reflect performance on naturally occurring eviction documentation.

Editorial extensions

If this is right

  • Fine-tuned open-weights LLMs trained on the released dataset can be deployed at 3B scale with Macro-F1 near 0.85–0.89 on eviction tasks, making eviction screening feasible without proprietary APIs.
  • The 3,000–5,000 sample plateau suggests downstream users can reproduce the pipeline's gains without collecting tens of thousands of annotations.
  • Including reasoning traces in training data lifts small-model accuracy (LLaMA-3.2-3B: +3.8 Macro-F1), so interpretable rationales are a byproduct rather than a cost.
  • Replacing 30% of synthetic training notes with real PMC notes improves out-of-domain PMC Micro-F1 by up to +0.302, indicating synthetic-only training is insufficient for real-world transfer.
  • The same augmenter–annotator design is claimed to generalize to other under-coded SDoH such as food insecurity, utility shutoff, and intimate partner violence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference — If the pipeline generalizes as claimed, the largest practical impact may be surveillance: coupling an eviction-risk flag with a generated rationale could let health systems screen whole populations without waiting for structured Z-code documentation.
  • Inference — The rewritten-test design means real-world F1 is untested; a fair deployment test on unmodified notes with adjudicated eviction mentions would be the first check before clinical use.
  • Inference — The plateau at 3k–5k examples hints that the synthetic data is information-dense; testing whether the same plateau holds for other SDoH domains would show whether the 80% labor saving transfers.
  • Inference — Because the training and test PMC notes share a source, part of the reported generalization gain may be source-overlap rather than lexical diversity; constructing a held-out hospital system's notes would separate these effects.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces SynthEHR-Eviction, a pipeline that combines GPT-based augmentation, DSPy automatic prompt optimization, and human-in-the-loop validation to construct a 14-class dataset of eviction-related social determinants of health (SDoH). The authors fine-tune open LLMs (Qwen2.5, LLaMA-3.1, etc.) and report Macro-F1 scores of 88.8% for eviction subcategories and 90.3% for non-eviction subcategories averaged over synthetic, MIMIC, and PMC test sets, slightly exceeding a GPT-4o-APO baseline while reducing manual annotation effort by over 80%. The paper also presents class-wise analyses, reasoning-annotation ablations, and training-size scaling experiments.

Significance. If the generalization claim survives scrutiny, this is a useful resource: the pipeline is modular, the dataset is publicly released (except MIMIC, gated by DUA), and the use of five-run confidence intervals, reasoning annotations for small models, and systematic ablation of training size are strengths. The central comparison to GPT-4o-APO, however, rests on devsets that are human-rewritten to inject eviction content and on a hybrid training set that may overlap with the PMC test set; until those checks are performed, the real-world claim is not fully established.

major comments (4)
  1. [§4.3, Table 5] Section 4.3 states that the second fine-tuning phase enriches the training set by adding 30% PMC long notes, while Section 4.2.3 constructs the PMC devset from PMC-Patients case reports rewritten by humans to include targeted eviction-related scenarios. The paper does not state whether the PMC documents used for the 30% training replacement are disjoint from the PMC documents used to build the 48 PMC test instances. Table 5 reports that this addition improves Qwen2.5-7B's PMC Micro-F1 by +0.302, so this component is decisive for the PMC results. Since the three-way average weights PMC as one-third and the claimed outperformance margin over GPT-4o-APO in eviction Macro-F1 is about 0.01 (0.888 vs. 0.878), overlap in even a handful of the 48 PMC test notes could change the headline. Please verify disjointness or re-run the evaluation with a guaranteed-disjoint test set.
  2. [§4.1.1 and §4.2.3] The augmentation pipeline selects over 30,000 MIMIC discharge notes as raw input (Section 4.1.1), and the MIMIC devset consists of MIMIC notes that were rewritten by human experts to include eviction-related content (Section 4.2.3). The paper does not demonstrate that the raw notes used for augmentation are disjoint from the notes selected for rewriting into the MIMIC devset. If the same patient notes or even the same discharge summaries appear in both, the synthetic training data could have been generated from exactly the texts that later appear in the test set, inflating MIMIC F1. Please report document- and patient-level overlap between the augmentation corpus and the MIMIC test notes.
  3. [§4.2.3] The MIMIC and PMC devsets are human-rewritten notes rather than naturally occurring eviction documentation. The rewriting instructions in Table 8 ask annotators to preserve the original MIMIC documentation style and incorporate eviction-related circumstances, which produces a specific, possibly stereotyped, distribution of eviction language. The paper does not provide evidence that this distribution matches how eviction is actually documented in clinical notes, for example in the authors' prior VA corpus (reference 27). Without such evidence, the reported MIMIC/PMC F1 scores should be interpreted as performance on human-generated rewrites, not on naturalistic real-world notes. A direct comparison on unmodified eviction-containing notes would strengthen the claim.
  4. [§4.4.1] The description of the five-run procedure for GPT-4o and GPT-4o-mini states that one run used temperature=0 and four additional runs used temperature=0.5. This mixes deterministic and stochastic generation within the same five-run confidence interval, so the 95% intervals and p-values reported in Table 1 are not computed under a single well-defined sampling distribution. For a fair comparison with the fine-tuned models, which vary random seeds under fixed hyperparameters, either use temperature>0 for all runs of the closed models or report the deterministic run separately.
minor comments (6)
  1. [§2.5] There is a typo in the phrase 'black-box eviciton prediction'; it should be 'eviction prediction'.
  2. [Appendix, Table 8/9/13] Several typos appear in the appendix: 'happend' instead of 'happened' in Table 8, 'seires' instead of 'series' in Table 9, and 'Fountuantly' instead of 'Fortunately' in Table 13. A proofread pass is recommended.
  3. [§4.2.3] The title 'Trainset and Devset' is ambiguous; consider renaming it to 'DSPy Trainset and Devset' to distinguish from the later SFT training set.
  4. [Figure 2] The caption mentions 'Red solid and Red dashed horizontal lines' but does not clearly map which line corresponds to GPT-4o-APO versus GPT-4o-mini-APO; please clarify the legend and caption.
  5. [Table 4] The rows for 'Eviction absent' have empty cells for Test-Mimic and Test-PMC; use an explicit dash or 'not applicable' to indicate that the class is intentionally excluded from those sets.
  6. [§4.2.1] The phrase 'divided into two groups, corresponding to three separate annotators' is confusing; rephrase to describe the two task groups (eviction vs. non-eviction) handled by three annotators (binary, eviction multi-class, non-eviction multi-class).

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity: benchmark claims rest on human-validated external MIMIC/PMC test sets; self-citations are definitional context, not load-bearing.

full rationale

The paper makes no first-principles derivation; its claims are empirical. The eviction taxonomy and confusion-matrix metric are attributed to the authors' prior JAMIA paper (ref 27), but the definitions are restated in full (Tables 6-7) and the comparison to GPT-4o-APO is computed on human-validated Synth, MIMIC, and PMC devsets (Section 4.2.3, Table 4), so self-citation is not load-bearing. The Synth devset is generated by the same GPT-based augmenter family as the training data; the paper explicitly acknowledges this in Discussion ('The Synth dataset, generated using our LLM-based pipeline, closely mirrors the model's generation style and served as the primary training source, explaining its strong in-domain performance'), which is a limitation of in-domain evaluation, not a circular reduction of the headline claim, because the headline average is not exclusively Synth and external MIMIC/PMC results are reported separately. The possible lack of documented disjointness between the 30% PMC training notes (Section 4.3) and the PMC test notes (Section 4.2.3) is a reproducibility/leakage concern, not a demonstrated circularity; the paper's own limitation statement ('additional clinical validation will be necessary to establish the model's performance in real-world environments') appropriately flags external validity. No equation or construction in the paper makes a predicted quantity equal to an input quantity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on several assumptions about data representativeness and label quality, plus hand-tuned parameters (real-data mixing ratio, training sizes, acceptance threshold, temporal cutoff). These are the main things a reader must grant beyond the released artifacts.

free parameters (4)
  • Hybrid real-data mixing ratio = 30% PMC notes
    Introduced in phase 2 after synthetic-only training showed weak PMC generalization (Table 5); the ratio is a hand-tuned hyperparameter chosen on the evaluation domain, and the paper does not report sensitivity to it.
  • SFT training set sizes = 5,000 for LLMs (Step 2); 3,000 for BERT models (Step 2) and all models (Step 3)
    Chosen from the data-size experiments in Section 2.6 to maximize Macro-F1 on the devsets; 10,000 samples caused overfitting.
  • Augmenter acceptance threshold = 0.90 accuracy
    Algorithm 1 stops optimizing an augmenter only when human verification accuracy reaches 90%; this arbitrary threshold shapes the composition of the generated training data.
  • Temporal current/history cutoff = Within the current natural year vs more than one year ago
    This labeling rule defines Eviction present current vs history and comes from the authors' prior work (Yao et al., 2023, ref 27); it is a hand-set boundary, not a learned or standard threshold.
assumptions (5)
  • domain assumption MIMIC discharge notes with a 'social history' section are a suitable raw corpus for eviction-related SDoH generation and evaluation.
    Section 4.1.1; the selection of 30,000 notes is based on the authors' observation, not on a demonstrated match to eviction documentation patterns across institutions.
  • domain assumption Eviction maps to ICD-10-CM Z59.89 and the 14-class taxonomy is clinically valid.
    Section 4.1.2; the mapping is asserted without external coding validation, and the taxonomy is inherited from prior work (ref 27).
  • ad hoc to paper Human rewrites of MIMIC/PMC notes preserve the linguistic and narrative distribution of real eviction documentation.
    Section 4.2.3; experts are instructed to preserve structure, but the inserted eviction content is artificial and may be cleaner or more explicit than natural mentions.
  • domain assumption DSPy annotator self-consistency (three consistent passes) is sufficient quality control for the 8,000 SFT instances.
    Figure 4 and Section 4.3; only the DSPy trainset is human-verified, and the large-scale SFT labels come from model agreement without per-instance human review.
  • ad hoc to paper PMC notes used in the hybrid training set do not overlap with the PMC test set.
    Section 4.3 and Table 5; the paper never states that the 30% PMC training notes are disjoint from the PMC test notes, yet the main results depend on this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SynthEHR-Eviction: Enhancing Eviction SDoH Detection with LLM-Augmented Synthetic EHR Data." pith.science (2026). https://pith.science/paper/7DZ5EKDP

@misc{pith2026250707421,
  author       = {Pith},
  title        = {Pith review of: SynthEHR-Eviction: Enhancing Eviction SDoH Detection with LLM-Augmented Synthetic EHR Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7DZ5EKDP}},
  note         = {Machine review of arXiv:2507.07421}
}
read the original abstract

Eviction is a significant yet understudied social determinants of health (SDoH), linked to housing instability, unemployment, and mental health. While eviction appears in unstructured electronic health records (EHRs), it is rarely coded in structured fields, limiting downstream applications. We introduce SynthEHR-Eviction, a scalable pipeline combining LLMs, human-in-the-loop annotation, and automated prompt optimization (APO) to extract eviction statuses from clinical notes. Using this pipeline, we created the largest public eviction-related SDoH dataset to date, comprising 14 fine-grained categories. Fine-tuned LLMs (e.g., Qwen2.5, LLaMA3) trained on SynthEHR-Eviction achieved Macro-F1 scores of 88.8% (eviction) and 90.3% (other SDoH) on human validated data, outperforming GPT-4o-APO (87.8%, 87.3%), GPT-4o-mini-APO (69.1%, 78.1%), and BioBERT (60.7%, 68.3%), while enabling cost-effective deployment across various model sizes. The pipeline reduces annotation effort by over 80%, accelerates dataset creation, enables scalable eviction detection, and generalizes to other information extraction tasks.

Figures

Figures reproduced from arXiv: 2507.07421 by the authors.

Figure 1
Figure 1. Performance comparison of LLMs trained with and without reasoning annotations on the Eviction Multi-Class task. 2.5 Impact of Reasoning in Training Data To evaluate the effect of explicit reasoning annotations in training data, we prepared two versions of each model’s training set: one with reasoning explanations and one without, keeping the labels identical. As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Impact of training set size on model performance across eviction and non-eviction classification tasks. Each subplot illustrates how model performance varies with different training set sizes (x-axis) across four classification scenarios: eviction (top two rows) and non-eviction (bottom two rows), evaluated using two metrics—Macro-F1 (first and third rows) and Micro-F1 (second and fourth rows). The y-axis represents… view at source ↗
Figure 3
Figure 3. Relationships between Eviction and other housing and economic challenges, providing a comprehensive framework for understanding its broader implications within the SDoH context. 4.1 Data Augmentation Pipeline 4.1.1 Raw Note Extraction We selected over 30,000 discharge notes from the MIMIC-III and MIMIC-IV databases. The selection criteria focused on notes that contained a ”social history” section. This was based on … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: A complete DSPy-based system for training annotators to identify potential eviction cases from clinical notes. Upper Section (Training Process): A small-scale human-validated dataset feeds into the DSPy Program training process, which employs GPT-4 or GPT-4o-mini model…
Figure 5
Figure 5. Figure 5: Class Performance Across Different Devsets for Step 2: Eviction Multi-Class. Box plot revealing F1 scores across three datasets (Synth, MIMIC, and PMC) for all evaluated models, including GPT-4o-APO, GPT-4o-mini-APO, fine-tuned Qwen, Llama, SBDH-LLM, BERT, Clinical-Ber…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 56 canonical work pages

  1. [1]

    Whole person health: What it is and why it’s important

    National Center for Complementary and Integrative Health. Whole person health: What it is and why it’s important. https://www.nccih.nih.gov/health/whole-person-health-what-it-is-and-why-its-important (n.d.). Accessed: 2025-03-17

    work page 2025

  2. [2]

    G., Haun, J

    Bokhour, B. G., Haun, J. N., Hyde, J., Charns, M. & Kligler, B. Transforming the veterans affairs to a whole health system of care: time for action and research. Med. Care 58, 295–300 (2020)

    work page 2020

  3. [3]

    Organization, W. H. Social determinants of health. http://www.who.int/social determinants/sdh definition/en/ (n.d.). Accessed: 2025-03-17

    work page 2025

  4. [4]

    M., Gennuso, K

    Hood, C. M., Gennuso, K. P., Swain, G. R. & Catlin, B. B. County health rankings: relationships between determinant factors and health outcomes. Am. journal preventive medicine 50, 129–135 (2016)

    work page 2016

  5. [5]

    Craig, K. J. T. et al. Leveraging data and digital health technologies to assess and impact social determinants of health (sdoh): a state-of-the-art literature review. Online J. Public Heal. Informatics 13, E14 (2021)

    work page 2021

  6. [6]

    & Thapa, A

    Rangachari, P. & Thapa, A. Impact of hospital and health system initiatives to address social determinants of health (sdoh) in the united states: a scoping review of the peer-reviewed literature. BMC Heal. Serv. Res. 25, 342 (2025)

    work page 2025

  7. [7]

    & Rosenheck, R

    Tsai, J., Jones, N., Szymkowiak, D. & Rosenheck, R. A. Longitudinal study of the housing and mental health outcomes of tenants appearing in eviction court. Soc. psychiatry psychiatric epidemiology 56, 1679–1686 (2021)

    work page 2021

  8. [8]

    Chen, K. L. et al. Association of promoting housing affordability and stability with improved health outcomes: a systematic review. JAMA Netw. Open 5, e2239860–e2239860 (2022)

    work page 2022

Show all 69 references
  1. [9]

    & Dasilva, L

    Tsai, J., Szymkowiak, D., Beydoun, H. & Dasilva, L. Transportation as a determinant of health: Evaluating the veterans affairs rideshare program for homeless patients: Article examines the veterans affairs rideshare program for homeless patients. Heal. Aff. 43, 1090–1099 (2024)

    2024

  2. [10]

    L., Pyrialakou, D., Waldorf, B

    Losada-Rojas, L. L., Pyrialakou, D., Waldorf, B. S., Banda, J. A. & Gkritza, K. Understanding transportation-related health determinants and their interrelationships in rural us. J. Transp. & Heal. 38, 101887 (2024)

    2024

  3. [11]

    Guevara, M. et al. Large language models to identify social determinants of health in electronic health records. NPJ digital medicine 7, 6 (2024). 16/27

    2024

  4. [12]

    Eviction in the united states

    Wikipedia contributors. Eviction in the united states. https://en.wikipedia.org/wiki/Eviction in the United States (n.d.). Accessed: 2025-03-17

    2025

  5. [13]

    & Kimbro, R

    Desmond, M. & Kimbro, R. T. Eviction’s fallout: housing, hardship, and health. Soc. Forces94, 295–324 (2015)

    2015

  6. [14]

    Evicted: Poverty and Profit in the American City (Crown, New York City, USA, 2016)

    Desmond, M. Evicted: Poverty and Profit in the American City (Crown, New York City, USA, 2016)

    2016

  7. [15]

    & Huang, M

    Tsai, J. & Huang, M. Systematic review of psychosocial factors associated with evictions. Heal. Soc. Care Community 27, e1–e9 (2019)

    2019

  8. [16]

    & Desmond, M

    Hepburn, P., Louis, R. & Desmond, M. Racial and gender disparities among evicted americans. Sociol. Sci. 7, 649–662 (2020)

    2020

  9. [17]

    Theorizing pathways between eviction filings and increased mortality risk

    Tsai, J. Theorizing pathways between eviction filings and increased mortality risk. JAMA 331, 570–571, DOI: 10.1001/ jama.2023.27978 (2024)

    2024

  10. [18]

    Johnson, A. E. et al. Mimic-iii, a freely accessible critical care database. Sci. data 3, 1–9 (2016)

    2016

  11. [19]

    Johnson, A. et al. Mimic-iv. PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021) 49–55 (2020)

    2020

  12. [20]

    Improving the collection of social determinants of health (sdoh) data with icd-10-cm z codes (2023)

    for Medicare & Medicaid Services (CMS), C. Improving the collection of social determinants of health (sdoh) data with icd-10-cm z codes (2023). Retrieved from https://go.cms.gov/OMH

    2023

  13. [21]

    Sivarajkumar, S. et al. Clinical information retrieval: a literature review. J. healthcare informatics research8, 313–352 (2024)

    2024

  14. [22]

    Savova, G. K. et al. Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications. J. Am. Med. Informatics Assoc. 17, 507–513 (2010)

    2010

  15. [23]

    & Oleynik, M

    Hahn, U. & Oleynik, M. Medical information extraction in the age of deep learning. Yearb. medical informatics 29, 208–220 (2020)

    2020

  16. [24]

    Information extraction from clinical notes: Are we ready to switch to large language models? arXiv preprint arXiv:2411.10020 (2024)

    Hu, Y .et al. Information extraction from clinical notes: Are we ready to switch to large language models? arXiv preprint arXiv:2411.10020 (2024)

    2024 arXiv

  17. [25]

    Keloth, V . K.et al. Social determinants of health extraction from clinical notes across institutions using large language models. npj Digit. Medicine 8, 1–13 (2025)

    2025

  18. [26]

    Patra, B. G. et al. Extracting social determinants of health from electronic health records using natural language processing: a systematic review. J. Am. Med. Informatics Assoc. 28, 2716–2727, DOI: 10.1093/jamia/ocab170 (2021)

    2021 doi

  19. [27]

    Yao, Z. et al. Automated identification of eviction status from electronic health record notes. J. Am. Med. Informatics Assoc. 30, 1429–1437, DOI: 10.1093/jamia/ocad081 (2023)

    2023 doi

  20. [28]

    Ramnath, K. et al. A systematic survey of automatic prompt optimization techniques. arXiv preprint arXiv:2502.16923 (2025)

    2025

  21. [29]

    Khattab, O. et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714 (2023)

    2023 arXiv

  22. [30]

    C., Mallinson, J., Adamek, J., Malmi, E

    Magister, L. C., Mallinson, J., Adamek, J., Malmi, E. & Severyn, A. Teaching small language models to reason. arXiv preprint arXiv:2212.08410 (2022)

    2022 arXiv

  23. [31]

    Zhao, Z., Jin, Q., Chen, F., Peng, T. & Yu, S. A large-scale dataset of patient summaries for retrieval-based clinical decision support systems. Sci. data 10, 909 (2023)

    2023

  24. [32]

    & Gershenson, C

    Desmond, M. & Gershenson, C. Who gets evicted? assessing individual, neighborhood, and network factors. Soc. Sci. Res. 62, 362–377 (2017)

    2017

  25. [33]

    & Roberts, K

    Hsu, E. & Roberts, K. Leveraging large language models for knowledge-free weak supervision in clinical natural language processing. Sci. reports 15, 8241 (2025)

    2025

  26. [34]

    & Ratner, A

    Zhang, J., Hsieh, C.-Y ., Yu, Y ., Zhang, C. & Ratner, A. A survey on programmatic weak supervision. arXiv preprint arXiv:2202.05433 (2022)

    2022 arXiv

  27. [35]

    Zha, D. et al. Data-centric artificial intelligence: A survey. ACM Comput. Surv. 57, 1–42 (2025)

    2025

  28. [36]

    Promda: Prompt-based data augmentation for low-resource nlu tasks

    Wang, Y .et al. Promda: Prompt-based data augmentation for low-resource nlu tasks. arXiv preprint arXiv:2202.12499 (2022)

    2022 arXiv

  29. [37]

    Han, S. et al. Classifying social determinants of health from unstructured electronic health records using deep learning-based natural language processing. J. Biomed. Informatics 127, 103984 (2022). 17/27

    2022

  30. [38]

    Mitra, A. et al. Associations between natural language processing–enriched social determinants of health and suicide death among us veterans. JAMA Netw. Open 6, e233079–e233079 (2023)

    2023

  31. [39]

    & Wang, S

    Majid, I., Mishra, V ., Ravindranath, R. & Wang, S. Y . Evaluating the performance of large language models for named entity recognition in ophthalmology clinical free-text notes. In AMIA Annual Symposium Proceedings, vol. 2024, 778 (2025)

    2025

  32. [40]

    M., Schmerder, M

    Dennst¨adt, F., Hastings, J., Putora, P. M., Schmerder, M. & Cihoric, N. Implementing large language models in healthcare while balancing control, collaboration, costs and security. npj Digit. Medicine 8, 143 (2025)

    2025

  33. [41]

    & Roth, B

    Kougia, V ., Sedova, A., Stephan, A., Zaporojets, K. & Roth, B. Analysing zero-shot temporal relation extraction on clinical notes using temporal consistency. arXiv preprint arXiv:2406.11486 (2024)

    2024 arXiv

  34. [42]

    Styler IV , W. F.et al. Temporal annotation in the clinical domain. Transactions association for computational linguistics 2, 143–154 (2014)

    2014

  35. [43]

    Bethard, S. et al. SemEval-2016 task 12: Clinical TempEval. In Bethard, S. et al. (eds.) Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 1052–1062, DOI: 10.18653/v1/S16-1165 (Association for Computational Linguistics, San Diego, California, 2016)

    2016 doi

  36. [44]

    & You, J

    Liu, Z., Han, P., Yu, H., Li, H. & You, J. Time-r1: Towards comprehensive temporal reasoning in llms. arXiv preprint arXiv:2505.13508 (2025)

    2025 arXiv

  37. [45]

    Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. Adv. neural information processing systems 35, 24824–24837 (2022)

    2022

  38. [46]

    Mitra, A. et al. Orca 2: Teaching small language models how to reason. arXiv preprint arXiv:2311.11045 (2023)

    2023 arXiv

  39. [47]

    J., Satapathy, R., Goh, R

    Yeo, W. J., Satapathy, R., Goh, R. S. M. & Cambria, E. How interpretable are reasoning explanations from prompting large language models? arXiv preprint arXiv:2402.11863 (2024)

    2024 arXiv

  40. [48]

    & Lin, B

    Bilal, A., Ebert, D. & Lin, B. Llms for explainable ai: A comprehensive survey. arXiv preprint arXiv:2504.00125 (2025)

    2025 arXiv

  41. [49]

    Agarwal, C., Tanneru, S. H. & Lakkaraju, H. Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. arXiv preprint arXiv:2402.04614 (2024)

    2024 arXiv

  42. [50]

    Jin, Q. et al. Hidden flaws behind expert-level accuracy of multimodal gpt-4 vision in medicine. npj Digit. Medicine 7, 190 (2024)

    2024

  43. [51]

    Yang, Z. et al. Unveiling gpt-4v’s hidden challenges behind high accuracy on usmle questions: Observational study. J. Med. Internet Res. 27, e65146 (2025)

    2025

  44. [52]

    & Faltings, B

    Paul, D., West, R., Bosselut, A. & Faltings, B. Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, 15012–15032 (2024)

    2024

  45. [53]

    L., Chelly Dagdia, Z., Zeitouni, K

    Abgrall, G., Holder, A. L., Chelly Dagdia, Z., Zeitouni, K. & Monnet, X. Should ai models be explainable to clinicians? Critical Care 28, 301 (2024)

    2024

  46. [54]

    Medical hallucinations in foundation models and their impact on healthcare.arXiv preprint arXiv:2503.05777 (2025)

    Kim, Y .et al. Medical hallucinations in foundation models and their impact on healthcare.arXiv preprint arXiv:2503.05777 (2025)

    2025

  47. [55]

    Ji, Z. et al. Survey of hallucination in natural language generation. ACM computing surveys 55, 1–38 (2023)

    2023

  48. [56]

    Mishra, P. et al. Synfac-edit: Synthetic imitation edit feedback for factual alignment in clinical summarization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 20061–20083 (2024)

    2024

  49. [57]

    & Selvaraj, S

    Yao, Z., Schloss, B. & Selvaraj, S. Improving summarization with human edits. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2604–2620 (2023)

    2023

  50. [58]

    Yao, Z. & Yu, H. A survey on llm-based multi-agent ai hospital. OSF (2025)

    2025

  51. [59]

    Tsai, J. et al. Development of a surveillance system to identify incidence of evictions among patients in veterans affairs medical centers across the united states. J. Community Heal. 1–10 (2025)

    2025

  52. [60]

    Yao, Z., Jaafar, A., Wang, B., Yang, Z. & Yu, H. Do clinicians know how to prompt? the need for automatic prompt optimization help in clinical note generation. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, 182–201 (2024). 18/27 Appendix Step 2:...

    2024

  53. [61]

    This includes cases where the eviction event occurred in the current year (e.g., ”this year,” ”last month”, ”last week”, ”several months ago,” or ”a few months ago”, ”recently”...)

    Current refers to events related to eviction that happened currently or recently, with specificity about the timing. This includes cases where the eviction event occurred in the current year (e.g., ”this year,” ”last month”, ”last week”, ”several months ago,” or ”a few months ...

  54. [62]

    The tenant received an eviction notice recently, but negotiations with the landlord to pay overdue rent are still ongoing

    History refers to events related to eviction that happened in the past, but with less specificity about the timing. This includes cases where the eviction event occurred in the distant past (e.g., ”last year,” ”several years ago,” or ”a few years ago”), or where the mutual res...

  55. [63]

    Eviction mr history 1

    Current means that the agreement or action related to mutual rescission is mentioned within a recent period (e.g., ”a few months ago,” ”recently,” ”this year”). Eviction mr history 1. Mutual Rescission (mr) refers to a specific legal agreement in which both the landlord and te...

  56. [64]

    contact one of the homeless shelters

    History refers to events related to eviction that happened in the past, but with less specificity about the timing. This includes cases where the Mutual Rescission occurred in the distant past (e.g., ”last year,” ”several years ago,” or ”a few years ago”), or where the mutual ...

  57. [65]

    Do NOT include the definitions or examples provided in your rewrited content

  58. [66]

    Focus exclusively on the social history, disregarding sentences related to family history or other topics

  59. [67]

    The augmented notes should clearly reflect the label context, be contextually coherent, with varied and diverse expressions

  60. [68]

    Focus on the unique circumstances, events, and actions related to this individual case

    The augmented note should be a detailed description of a specific patient case that illustrates the application of the SDoH label. Focus on the unique circumstances, events, and actions related to this individual case. Avoid using general or broad descriptions of processes or ...

  61. [69]

    Augmented Notes: input variables=[”raw notes”, ”label”, ”definition”] Annotation Prompt in Step 1: Binary Classi- fication Go through each sentence of the patient note

    Your output should not exceed 100 words. Augmented Notes: input variables=[”raw notes”, ”label”, ”definition”] Annotation Prompt in Step 1: Binary Classi- fication Go through each sentence of the patient note. If a sentence reflects eviction-related social determinants of heal...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.