Pith. sign in

REVIEW 3 major objections 4 minor 109 references

Clinical Communication Processing with Models Trained on LLM-Generated Synthetic Data: A Structured Survey and Novel Application Case Studies

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that LLM-generated synthetic clinical communication can bootstrap clinical NLP systems in channels and languages lacking labeled real-world data, and supports this with thirteen application case studies.

desk verdict A genuinely useful survey with honest limitations; treat the 'can bootstrap' claim as conditional until someone runs a real transfer test. read the letter →

arxiv 2608.05993 v1 pith:MFVRSOPV submitted 2026-08-06 cs.CL

Alexander Apartsin , Yehudit Aperstein This is my paper
classification cs.CL
keywords syntheticclinicalcommunicationNLPLLM-generateddatamedicaldialoguepatienttriagehandofflow-resourcelanguagestrain-on-synthetictest-on-real
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that large language models can generate synthetic clinical communication — patient self-descriptions, portal messages, ambulance radio reports, nurse handoffs — and that models trained on that synthetic text can perform clinical NLP tasks in channels and languages where no labeled real-world data exists. It organizes the field around a source-to-communication-to-downstream-model pipeline and adds thirteen application case studies, several of which reach strong headline numbers (0.90 macro-F1 for medication-question risk on real patient text; 0.79 class macro-F1 for SBAR handoff completeness on synthetic evaluation). The recurring pattern is that small fine-tuned encoders trained on synthetic data beat zero-shot LLM baselines, and that deliberately degraded synthetic text improves robustness. The paper is candid that almost all evidence is train-on-synthetic/test-on-synthetic, so the decisive claim — that such models transfer to authentic clinical communication — is still largely unproven. A reader should care because successful transfer would make scarce, private clinical conversations a renewable training resource.

What carries the argument

The load-bearing mechanism is the source-to-communication generation pipeline, in which a structured or documentary clinical source (a diagnostic label, symptom list, discharge case, or casualty profile) is rendered by an LLM into communicative text, with labels either fixed at generation time and audited by a judge or added in a second labeling pass. Around that pipeline the paper builds a four-level evaluation scheme — text quality, clinical quality, dataset quality, and downstream utility — and labels each study by regime: S→S (train and test on synthetic), S→R (train synthetic, test real), R+S→R (mixed training, real test), and R→R (real baseline). The S→R protocol is the decisive yardstick, and the paper uses it to identify which of its thirteen studies actually measure transfer. Recurring design patterns — fine-tuned encoders over zero-shot LLMs, deliberate degradation for robustness, generation-time labels with judge auditing, synthetic augmentation of rare classes, and local on-premises generation for privacy — carry the practical argument.

What would settle it

Take one of the released case-study corpora, say the SBAR handoff set or the EMS pre-arrival reports, train the same fine-tuned encoder on it, and evaluate on a small set of authentic handoff notes or EMS transcripts collected from a hospital; if macro-F1 collapses relative to the published synthetic-test result (for instance, from 0.79 to near the zero-shot baseline), the claim that synthetic communication bootstraps clinical NLP is refuted for that channel.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM-generated synthetic clinical communication has matured from a privacy-preserving substitute for real records into a practical resource for building, evaluating, and benchmarking clinical NLP systems. Concretely, it claims that the pipeline clinical source to LLM-generated communication to downstream healthcare NLP model can bootstrap working systems for channels that lack labeled real-world data, including EMS pre-arrival reports, field-radio casualty documentation, nurse handoffs, patient-portal triage, and low-resource discharge communication. The evidence is thirteen case studies in which fine-tuned encoders consistently outperform zero-shot LLM baselines (for example, DistilBERT 0.81 versus 0.29 macro-F1 for portal triage, BioClinicalBERT with a safety cascade reaching 0.997 accuracy for longitudinal priority triage, and BioBERT and BlueBERT reaching 0.90 macro-F1 for medication-question risk on authentic patient text). The paper's own boundary condition is that most of these results are measured on held-out synthetic text, making train-on-synthetic/test-on-authentic transfer the field's central open validation step.

Load-bearing premise

The paper's practical claims rest on the assumption that held-out synthetic communication is a valid proxy for authentic clinical communication, so that train-and-test-on-synthetic results predict how a model will behave on real patient text.

Editorial extensions

If this is right

  • Fine-tuned encoder models trained on synthetic clinical communication can match or beat zero-shot LLMs on downstream tasks, giving a route to deployable triage, routing, and handoff systems in channels with no labeled real data.
  • Deliberately degraded synthetic text — ASR noise, radio dropouts, omissions — can be used to train models that hold up on noisy real-world input, as the EMS routing study shows (removing noisy training variants drops macro-F1 by 6.6 to 10.6 points).
  • Synthetic augmentation of rare, high-stakes classes can correct class imbalance and improve classification of critical cases on authentic patient text, as in medication-question risk (0.90 macro-F1).
  • Local on-premises generators can bootstrap usable datasets without sending protected health content to hosted LLMs, supporting privacy-preserving institutional use.
  • The decisive next step is train-on-synthetic/test-on-authentic evaluation: until that evidence accumulates, synthetic communication is a practical research resource rather than validated clinical infrastructure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A cheap way to close the S→R gap might be per-channel transfer-calibration sets: a few dozen authentic transcripts used only for evaluation, so that S→S development can be monitored against a real anchor without needing large labeled corpora — a design the paper does not propose but its evaluation taxonomy invites.
  • If S→R transfer holds for even a subset of channels, the same generation pipeline could be pointed at other under-resourced clinical languages and modalities, since the paper's Hebrew discharge and casualty-radio studies already demonstrate the pattern in a low-resource language.
  • The paper's text-only synthesis stops at the transcript; extending the pipeline to generate audio with matched acoustic noise would let transcription-robust models be trained without recording real emergency calls, a direction the paper flags as an open space but does not execute.
  • Synthetic communication with controlled omissions and noise could double as a counterfactual probe: rendering the same clinical case in different registers would reveal which surface cues a model actually relies on, supporting safety auditing beyond the paper's degradation-for-robustness results.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper surveys the emerging area of clinical communication processing using LLM-generated synthetic data, organizing the literature by source representation, communication form, generation method, and downstream task. It presents thirteen novel case studies spanning patient-clinician conversations, doctor-to-patient instructions, portal messaging, EMS/dispatch, and handoff channels, almost all trained and evaluated on synthetic data. The authors report that fine-tuned encoders outperform zero-shot baselines and that deliberately degraded synthetic text improves robustness. They explicitly acknowledge that most evidence is train-on-synthetic/test-on-synthetic (S→S) and that transfer to authentic text remains largely outstanding.

Significance. If the claims are read as within-generator feasibility results, the paper is a useful structured survey plus a reproducible suite of case studies, with public data and code. Its honest labeling of the S→S regime and its candid limitations in Section 4 are strengths. The channel taxonomy and the reusable design patterns in Table 19 are likely to be of value to the community. However, the headline claim that synthetic communication 'can bootstrap such systems' is not established for real clinical communication; the paper's own limitations undermine the unqualified abstract assertion. A major revision that reframes the claim and/or adds S→R evidence would make this a solid contribution.

major comments (3)
  1. [Abstract and §6] The claim that synthetic communication 'can bootstrap such systems' is not supported by the S→S evaluation regime. Section 4.1 states that a model trained on synthetic data underperforms on real text in ways that held-out synthetic evaluation cannot detect, and §6 concedes that the decisive S→R evidence is 'still largely outstanding.' Since all but one of the case studies in Table 6 evaluate on held-out synthetic communication, the unqualified abstract claim overstates what the data show. Please either temper the claim to 'can bootstrap systems evaluated on held-out synthetic communication' or add a fully train-on-synthetic/test-on-real validation study.
  2. [§3.3.5, Table 6] The medication-question study is the only one with real test data, but it is R+S→R, not S→R: GPT-4.1 generates only additional synthetic critical questions to rebalance a real training set. This design cannot validate the fully synthetic training claim that the abstract and conclusions make. The paper should state explicitly that this study tests augmentation, not full synthetic bootstrapping, and should not be cited as evidence for the central claim.
  3. [§2.3, §3, and §4.3] The S→S regime shares generator, prompt templates, writer profiles, and noise injection between training and test, so a fine-tuned encoder can exploit generator-specific regularities. For example, §3.3.1 emits each message carrying its own structured labels, §3.2.1 anchors marked spans to appear verbatim, and §3.3.2 fixes target severity in the prompt. The paper's own §4.3 acknowledges that a model assessed only on synthetic communication 'has been measured against a world its own family generated.' This means headline results such as 0.79 class macro-F1 on SBAR completeness (Table 18) and 0.38 macro-F1 on EMS specialty routing (Table 16) are best interpreted as within-generator consistency, not clinical utility. The manuscript should apply this caveat prominently wherever 'bootstrap' or 'utility' is claimed.
minor comments (4)
  1. [§3.3.3 and elsewhere] The notation 'Bio_ClinicalBERT' appears in §3.3.3 while 'BioClinicalBERT' is used elsewhere (e.g., Table 13, §3.4.1); please standardize the spelling.
  2. [§3.4.2] The acronym 'IDF Form 101' should be expanded to 'Israel Defense Forces Form 101' at first use for readers outside the military-medicine context.
  3. [§3.2.1 and Table 10] The term 'sklearn reranker' should be written as 'scikit-learn reranker' to be accessible to the clinical NLP audience.
  4. [§2.3] The evaluation regimes S→S, S→R, R+S→R, and R→R are defined only in prose; a small glossary table would improve readability and prevent ambiguity in later sections.

Circularity Check

4 steps flagged · score 6.0 of 10

Headline 'synthetic communication can bootstrap such systems' is supported almost entirely by S→S evaluations in which labels are emitted with the generated text; the paper's own caveats confirm that the decisive train-on-synthetic/test-on-real evidence is still outstanding.

  1. self definitional [Section 2.3 (Evaluating Synthetic Clinical Communication), reinforced in Section 3 introduction and Section 4.3]
    "The train-on-synthetic, test-on-real protocol is the most decisive evidence of utility. It helps to label each study by its evaluation regime: S→S (train and test on synthetic data), S→R (train on synthetic, test on real), R+S→R (train on real plus synthetic, test on real), and R→R (a real-data baseline), with cross-generator (S₁→S₂) and human-written (S→H) variants as stronger tests. As Section 3 makes explicit, almost all of the application studies surveyed here operate in the S→S regime, which is why transfer to authentic communication remains the field's central open question."

    The load-bearing claim that synthetic communication can bootstrap clinical NLP systems is evaluated almost entirely in the S→S regime: held-out synthetic test messages come from the same generator, prompt templates, writer profiles, and noise injection as the training messages, and their labels are assigned by the generation process rather than by independent annotation of authentic text. A fine-tuned encoder can therefore exploit generator-specific surface regularities that are correlated with labels in the synthetic data but absent or different in real communication.

  2. self definitional [Section 2.2.3 (Automatic Annotation), applied in Section 3.3.2 (Postpartum severity triage)]
    "In the strongest form, embedded ground truth, the generator emits the text and its labels jointly, as when a triage severity or a target action is fixed in the prompt and the message written to match it."

    Here the label is not a property discovered in the text by an independent annotator; it is the prompt condition that caused the text to be written. The postpartum study instantiates this: 'a writer profile and the target severity are fixed in the prompt, so the intended labels come from the source scenario.' When the held-out test set is generated with the same label-first protocol, the classifier's 0.975 macro-F1 measures recovery of the generator's own injected condition from surface form. That task is correlated with the label by construction and does not assess whether severity labels can be predicted from authentic postpartum messages, whose severity is not fixed in any prompt.

2 more flagged steps
  1. self definitional [Section 3.2.1 (Extracting Clinical Decisions from Hebrew Discharge Summaries)]
    "GPT-4o-mini (Batch API) generates 518 synthetic Hebrew summaries carrying 1,222 gold spans, anchored to MedDec- and MIMIC-III-derived clinical targets by marked-span anchoring, with every span validated to appear verbatim in its summary so that the labels are recoverable by construction."

    The gold spans are inserted into the text and validated to appear verbatim, so the extraction task is defined as recovering strings the generator was forced to include. Both training and test summaries are produced with the same marked-span anchoring, so the reported strict F1 of 0.427 and relaxed F1 of 0.926 measure reconstruction of verbatim inserted spans, not extraction of clinical decisions from natural Hebrew discharge prose. The study itself notes that 'Because both training and test data are synthetic, transfer to genuine Hebrew notes remains the open question,' confirming that the headline metric is a property of the construction procedure rather than of real-world decision extraction.

  2. self definitional [Section 3.4.2 (Reconstructing Casualty Records from Field Radio)]
    "GPT-4o, with fifty further samples from GPT-4o-mini, converts structured casualty profiles into 500 messy Hebrew transcripts, injecting static, communication cutouts, slang, and self-corrections at three reliability levels and emitting the ground-truth record alongside each transcript."

    The structured casualty profile is both the generation input and the evaluation target: the ground-truth record is emitted as a by-product of the same generation call that produces the transcript. The held-out test set of 100 transcripts is drawn from the same pipeline, so AlephBERT's 79.3% exact-match measures how well a model inverts the generator's own profile-to-transcript mapping under injected noise, rather than how well real, unlabeled field-radio traffic would be reconstructed. There is no authentic casualty record against which the prediction is tested, so the result is a within-generator consistency score.

full rationale

The paper's central claim is that synthetic clinical communication can bootstrap clinical NLP systems where no labeled real-world data exists. The evidence offered is almost entirely S→S: Section 2.3 defines the regime, Section 3 labels nearly every case study as S→S, and Section 4.3 concedes that a model assessed only on synthetic communication has been measured against a world its own family generated. The specific reduction is visible in the generation protocols: labels are fixed in the prompt or emitted alongside the transcript, and held-out synthetic test sets share the same generator, templates, writer profiles, and noise injection. Fine-tuned encoders can therefore learn generator-specific regularities correlated with labels, so headline metrics such as 0.79 class macro-F1 on SBAR completeness, 0.38 macro-F1 on EMS routing, and 0.975 macro-F1 on postpartum triage demonstrate within-generator consistency rather than transfer to authentic communication. The one real-data evaluation, Section 3.3.5, is explicitly R+S→R: real patient questions are in the training mix and synthetic generation only rebalances the critical class, so it does not test the fully-synthetic claim. The paper is unusually honest about this, stating in Section 4.1 that 'a model trained on synthetic data underperforms on real text in ways that held-out synthetic evaluation cannot detect' and in Section 6 that the decisive end-to-end S→R evidence 'is still largely outstanding.' That candor prevents a higher score, as does the presence of some independent comparative content (fine-tuned encoders versus zero-shot baselines, and the effect of deliberate degradation). Still, the abstract's conclusion that synthetic communication 'can bootstrap such systems' is supported only by self-referential evaluations, and the derivation chain from generation condition to predicted label is circular by construction in the label-first and verbatim-anchoring studies. Self-citations such as references [1] and [3] are pointers to fuller reports rather than load-bearing justifications, so they do not add to the circularity score. Overall, the central claim partially reduces to a fit to the generator's own distribution, yielding a score of 6.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

All thirteen case studies depend on free design choices in synthetic generation, and the evaluation loop is self-referential: synthetic data is both training and test distribution. The leading axioms are that LLM-generated text can stand in for real clinical communication and that labels fixed at generation time are trustworthy; the paper itself qualifies both in Section 4. No truly new entities are postulated.

free parameters (5)
  • Generator sampling temperature (postpartum study) = 1.3
    GPT-4o-mini message generation used temperature 1.3 'for diversity' (Section 3.3.2); this affects the synthetic triage message distribution and therefore downstream scores.
  • Noise levels in patient self-description rewrite = medium, heavy
    Llama-3.1-8B rewrote 1,200 clean descriptions into medium- and heavy-noise variants (Section 3.1.1); the chosen noise levels shape task difficulty and reported accuracy.
  • Information-reveal tiers in diagnostic questioning = 100%, 80%, 50%
    Three tiers define how much of the case the simulated patient reveals (Section 3.1.2, Table 8); tier choice determines measured questioning efficiency.
  • Reliability levels in casualty radio transcripts = three levels
    GPT-4o injected static, cutouts, slang, and self-corrections at three reliability levels (Section 3.4.2); the mix of levels is a hand-chosen design variable.
  • Quality-tier filtering threshold in oncology judge = not specified
    A dual-model judge assigned quality tiers (Section 3.3.4); the threshold for admitting generated examples is not reported, making label-quality control a hidden free parameter.
assumptions (5)
  • domain assumption Held-out synthetic communication is a valid proxy for authentic clinical communication (S-to-S evaluation is informative)
    All but one case study evaluate on synthetic test data; this is the load-bearing premise for the survey's feasibility claims. Section 2.3 defines regimes, and Section 4.1 admits distribution shift makes S-to-S insufficient.
  • domain assumption The LLM generator does not invent clinical truth; it realizes facts fixed by the source
    Section 1 states 'the model does not invent clinical truth.' This is not guaranteed; Section 4.1 discusses clinical hallucination as a central risk.
  • domain assumption Labels fixed at generation time are correct and recoverable from generated text
    Section 2.2.3 and the case studies rely on embedded ground truth; Section 3.3.2 notes agreement 'still requires validation.'
  • domain assumption Macro-averaged F1 is the right summary for imbalanced safety-critical classes
    Section 2.3 states this; it is a modeling choice rather than a mathematical theorem, and it determines the headline results.
  • domain assumption Synthetic text without a real patient behind it can be shared without consent and de-identification burdens
    Section 2.2.1 motivates synthetic data through privacy, but Section 4.2 qualifies that synthetic data may inherit real privacy risk from conditioning data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Clinical Communication Processing with Models Trained on LLM-Generated Synthetic Data: A Structured Survey and Novel Application Case Studies." pith.science (2026). https://pith.science/paper/MFVRSOPV

@misc{pith2026260805993,
  author       = {Pith},
  title        = {Pith review of: Clinical Communication Processing with Models Trained on LLM-Generated Synthetic Data: A Structured Survey and Novel Application Case Studies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MFVRSOPV}},
  note         = {Machine review of arXiv:2608.05993}
}
read the original abstract

Much clinical value is conveyed not through structured records but through communication: exchanges in which patients describe symptoms, clinicians reason and give instructions, ambulances hand over to emergency departments, and nurses pass on a shift. Such language differs from tabular data because meaning depends on speaker role, intent, causality, uncertainty, omission, and channel noise. Healthcare natural language processing must therefore interpret information as conveyed rather than coded. This requires well-annotated corpora, which are scarce because authentic exchanges are private, fragmented, and costly to annotate. Large language models offer a way forward by transforming clinical sources, such as records, diagnostic labels, symptom lists, or care plans, into written and transcribed communication for downstream models. We present a structured narrative survey organized by source representation, communication form and participants, generation method, and downstream task, complemented by thirteen novel case studies. These build clinical NLP systems for communication channels and languages without labeled real-world data, including EMS pre-arrival reports, field-radio casualty documentation, nurse handoffs, patient-portal triage, and low-resource discharge communication. They show that synthetic communication can bootstrap such systems. Findings include the competitiveness of fine-tuned encoder models over evaluated zero-shot baselines and the value of deliberately degraded communication for robustness. The main limitation is that most studies evaluate on held-out synthetic communication, while train-on-synthetic, test-on-authentic evidence remains limited. We conclude that syn-thetic clinical communication is becoming a practical research resource; establishing it as reusable clinical infrastructure will require authentic-data transfer, safety and external validation.

Figures

Figures reproduced from arXiv: 2608.05993 by the authors.

Figure 1
Figure 1. Clinical-decision extraction: strict versus relaxed F1 across models. 3.3 Telemedicine and Patient-Portal Messaging In telemedicine and portal messaging the patient writes asyn￾chronously to the care team; processing is required because staff cannot read every incoming message in time, so models must triage, route, and prioritize them. Portal messaging is now studied both as a generation target, synthesizing realist… view at source ↗
Figure 2
Figure 2. Portal-message triage: fine-tuned heads versus zero- and few-shot prompting. 3.3.2 Severity Triage of Postpartum Messages After a caesarean section, patients message their care team with concerns spanning routine soreness to hemorrhage, and safe after-care depends on sorting them quickly. This study frames the problem as four-class severity classification under the Manchester scheme, implemented as a two-stage casca… view at source ↗
Figure 3
Figure 3. Postpartum severity triage: GPT-4o-mini zero-shot confusion matrix. 3.3.3 Clinical-Priority Triage of Longitudinal Portal Mes￾sages Beyond a single message, a patient's portal history must be sorted by clinical risk while holding a difficult balance between missed emergencies and alert fatigue. This study frames that as multi-class risk classification over longitudinal message sequences, with an explicit safety obje… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Home-care status detection: per-class recall of the top fusion models. 3.4 Paramedic-to-Hospital, Dispatch, and Handoff These are handovers between a paramedic, caller, or outgoing clinician and a hospital, dispatcher, or incoming team; processing matters because the c…
Figure 5
Figure 5. Figure 5: EMS report routing: macro-F1 across training setups; noisy synthetic data helps. 3.4.2 Reconstructing Casualty Records from Field Radio On the battlefield only about half of casualty encounters are documented at all, and what reaches a hospital arrives as cha￾otic radi…
Figure 6
Figure 6. Figure 6: Casualty-record reconstruction: accuracy, F1, and hallucination counts. 3.4.3 Completeness Checking of SBAR Handovers Shift handovers are among the most error-prone moments in care, and the SBAR format exists to make them complete. This study turns completeness itself …
Figure 7
Figure 7. Figure 7: SBAR handover completeness: class macro-F1 across models. 4 Limitations and Caveats of Synthetic Data Synthetic clinical communication is useful precisely because it is not real, and that same fact is the source of its limitations. This section states them plainly, so …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

109 extracted references · 49 canonical work pages

  1. [1]

    Werthaim, M

    M. Werthaim, M. Kimhi, A. Apartsin, Y. Aperstein. A benchmark for evaluating diagnostic questioning efficiency of LLMs in patient conversations. Scientific Reports, 2026

    2026

  2. [2]

    E. Mama, L. Sheri, Y. Aperstein, A. Apartsin. From Fuzzy Speech to Medical Insight: Benchmarking LLMs on Noisy Patient Narratives. arXiv:2509.11803, 2025

    work page arXiv 2025

  3. [3]

    Goncharok, A

    D. Goncharok, A. Shifman, A. Apartsin, Y. Aperstein. When Cu- riosity Signals Danger: Predicting Health Crises Through Online Medication Inquiries. arXiv:2509.11802, 2025

    work page arXiv 2025

  4. [4]

    Reliable Extraction of Clinical Follow-Up Instructions: A Hybrid Neural-Symbolic Pipeline

    M. Laufer, Y. Aperstein, A. Apartsin. Reliable Extraction of Clin- ical Follow-Up Instructions: A Hybrid Neural-Symbolic Pipe- line. arXiv:2605.26560, 2026

    work page Pith review arXiv 2026

  5. [5]

    T. Atia, Y. Aperstein, A. Apartsin. SeaAlert: Critical Information Extraction From Maritime Distress Communications with Large Language Models. IEEE Access, 2026

    2026

  6. [6]

    Aperstein, A

    Y. Aperstein, A. Gottlib, G. Benita, A. Apartsin. Explainable Se- mantic Text Relations: A Question -Answering Framework for Comparing Document Content. Information, 16(12):1090, 2025

    2025

  7. [7]

    Do Large Language Models Need Intent? Revisiting Response Generation Strategies for Service Assistant

    I. Bolshinsky, S. Kupiec, A. Sasson, Y. Aperstein, A. Apartsin. Do Large Language Models Need Intent? Revisiting Re- sponse Generation Strategies for Service Assistant. arXiv:2509.05006, 2025

    work page Pith review arXiv 2025

  8. [8]

    CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks

    A. Apartsin, Y. Aperstein. CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Bench- marks. arXiv:2606.03650, 2026

    work page Pith review arXiv 2026

Show all 109 references
  1. [9]

    Shapira, A

    G. Shapira, A. Apartsin. SPINE: A Benchmark for Measuring the Epistemic Backbone of Large Language Models. Harvard Dataverse, 2026. doi:10.7910/DVN/CG9X6D

    2026 doi

  2. [10]

    Apartsin, O

    A. Apartsin, O. Sason, Y. Aperstein. Toward a Benchmark for Controllable Simulation of Imperfect Students with Large Language Models. arXiv:2605.25601, 2026

    2026 arXiv

  3. [11]

    Aperstein, A

    Y. Aperstein, A. Apartsin. A Controlled Synthetic Benchmark for Educational Aspect -Based Sentiment Analysis. arXiv:2605.25502, 2026

    2026 arXiv

  4. [12]

    Cohen, D

    Y. Cohen, D. Ohayon, R. Somkin, Y. Aperstein, A. Apartsin. Code Review Without Borders: Evaluating Synthetic vs. Real Data for Review Recommendation. arXiv:2509.04810, 2025

    2025 arXiv

  5. [13]

    Aperstein, L

    Y. Aperstein, L. Halifa, S. Bar, A. Apartsin. LLM -guided head- line rewriting for clickability enhancement without clickbait. arXiv:2603.22459, 2026

    2026

  6. [14]

    Aperstein, Y

    Y. Aperstein, Y. Cohen, A. Apartsin. Generative AI-Based Plat- form for Deliberate Teaching Practice: A Review and a Sug- gested Framework. Education Sciences, 2025. 18

    2025

  7. [15]

    Apartsin, Y

    A. Apartsin, Y. Aperstein. Framing, Judging, Steering: An As- sessable Competency Model for Teaching Students to Reason With Generative AI. arXiv:2606.05983, 2026

    2026 arXiv

  8. [16]

    Dahary, A

    S. Dahary, A. Edana, A. Apartsin, Y. Aperstein. From Joy to Fear: A Benchmark of Emotion Estimation in Pop Song Lyr- ics. arXiv:2509.05617, 2025

    2025 arXiv

  9. [17]

    Cohen, S

    M. Cohen, S. Shani, E. Menahem, Y. Aperstein, A. Apartsin. Reading Between the Lines: Classifying Resume Seniority with Large Language Models. arXiv:2509.09229, 2025

    2025 arXiv

  10. [18]

    Aperstein, E

    Y. Aperstein, E. Moran, A. Apartsin. IRC -Bench: Recognizing Entities from Contextual Cues in First-Person Reminiscences. Machine Learning and Knowledge Extraction, 2026

    2026

  11. [19]

    Alshaikhdeeb et al

    B. Alshaikhdeeb et al. Generation of Synthetic Clinical Text: A Systematic Review. arXiv:2507.18451, 2025

    2025 arXiv

  12. [20]

    A Scoping Review of Synthetic Data Generation by Language Models in Biomedical Research and Applica- tion

    Rao, Liu, et al. A Scoping Review of Synthetic Data Generation by Language Models in Biomedical Research and Applica- tion. Journal of Healthcare Informatics Research (arXiv:2506.16594), 2026

    2026

  13. [21]

    M. Loni, F. Poursalim, M. Asadi, et al. A review on generative AI models for synthetic medical text, time series, and longitu- dinal data. npj Digital Medicine, 2024

    2024

  14. [22]

    Ibrahim, Y

    M. Ibrahim, Y. Al Khalil, S. Amirrajab, et al. Generative AI for Synthetic Data Across Multiple Medical Modalities: A Sys- tematic Review. arXiv:2407.00116, 2024

    2024 arXiv

  15. [23]

    Zhou et al

    H. Zhou et al. A Survey on Medical Large Language Models. arXiv:2406.03712, 2024

    2024 arXiv

  16. [24]

    A. M. Alkalbani, A. S. Alrawahi, A. Salah, et al. A Systematic Review of Large Language Models in Medical Specialties: Applications, Challenges and Future Directions. Information, 16(6):489, 2025

    2025

  17. [25]

    X. Liu, J. Chang, J. J. Zhang. Knowledge -Grounded Dialogue Generation for Medical Conversations: A Survey. Interna- tional Conference on Information Visualisation (IV), 2023

    2023

  18. [26]

    M. Lyu, X. Li, Z. Chen, et al. Natural Language Generation in Healthcare: A Review of Methods and Applications. arXiv:2505.04073, 2025

    2025 arXiv

  19. [27]

    Zeng et al

    G. Zeng et al. MedDialog: Large -scale Medical Dialogue Datasets. EMNLP, 2020

    2020

  20. [28]

    Wang et al

    J. Wang et al. NoteChat: A Dataset of Synthetic Patient -Physi- cian Conversations Conditioned on Clinical Notes. Findings of ACL (arXiv:2310.15959), 2024

    2024 arXiv

  21. [29]

    Xu et al

    R. Xu et al. Knowledge -Infused Prompting: Assessing and Ad- vancing Clinical Text Data Generation with LLMs (ClinGen). arXiv:2311.00287, 2023

    2023 arXiv

  22. [30]

    K. Wang, J. Zhu, M. Ren, et al. A Survey on Data Synthesis and Augmentation for Large Language Models. arXiv:2410.12896, 2024

    2024 arXiv

  23. [31]

    A. R. Sarkar, Y.-S. Chuang, N. Mohammed, et al. De-identifica- tion is not enough: a comparison between de -identified and synthetic clinical notes. Scientific Reports (arXiv:2402.00179), 2024

    2024 arXiv

  24. [32]

    Kaabachi, J

    B. Kaabachi, J. Despraz, T. Meurers, et al. A scoping review of privacy and utility metrics in medical synthetic data. npj Dig- ital Medicine, 2025

    2025

  25. [33]

    Yim et al

    W. Yim et al. ACI -BENCH: a Novel Ambient Clinical Intelli- gence Dataset for Benchmarking Automatic Visit Note Gen- eration. Scientific Data (arXiv:2306.02022), 2023

    2023 arXiv

  26. [34]

    Ben Abacha et al

    A. Ben Abacha et al. An Empirical Study of Clinical Note Gen- eration from Doctor -Patient Encounters (MTS -Dialog). EACL, 2023

    2023

  27. [35]

    Papadopoulos Korfiatis, F

    A. Papadopoulos Korfiatis, F. Moramarco, et al. PriMock57: A Dataset of Primary Care Mock Consultations. ACL (arXiv:2204.00333), 2022

    2022 arXiv

  28. [36]

    Rujas, R

    M. Rujas, R. M. Gómez Del Moral Herranz, G. Fico, et al. Syn- thetic data generation in healthcare: A scoping review of re- views on domains, motivations, and future applications. International Journal of Medical Informatics, 2024

    2024

  29. [37]

    R. Li, X. Wang, H. Yu. Two Directions for Clinical Data Gener- ation with Large Language Models: Data-to-Label and Label- to-Data. Findings of EMNLP, 2023

    2023

  30. [38]

    Gormley, K

    K. Gormley, K. Lockhart, J. Isaac. Using natural language pro- cessing in facilitating pre -hospital telephone triage of emer- gency calls. British Paramedic Journal, 2022

    2022

  31. [39]

    Ritter, S

    A. Ritter, S. Clark, Mausam, O. Etzioni. Named Entity Recogni- tion in Tweets: An Experimental Study. EMNLP, 2011

    2011

  32. [40]

    Derczynski, E

    L. Derczynski, E. Nichols, M. van Erp, N. Limsopatham. Results of the WNUT2017 Shared Task on Novel and Emerging En- tity Recognition. W-NUT@EMNLP, 2017

    2017

  33. [41]

    A. Wang, K. Cho, M. Lewis. Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (QAGS). ACL, 2020

    2020

  34. [42]

    Scialom et al

    T. Scialom et al. QuestEval: Summarization Asks for Fact-based Evaluation. EMNLP, 2021

    2021

  35. [43]

    W. Yin, D. Radev, C. Xiong. DocNLI: A Large-scale Dataset for Document-level Natural Language Inference. Findings of ACL-IJCNLP, 2021

    2021

  36. [44]

    Zuluaga-Gomez et al

    J. Zuluaga-Gomez et al. ATCO2 corpus: A Large-Scale Dataset for Research on ASR and NLU of Air Traffic Control Com- munications. arXiv:2211.04054, 2023

    2023 arXiv

  37. [45]

    Sun et al

    K. Sun et al. Speech -based Slot Filling using Large Language Models. arXiv:2311.07418, 2023

    2023 arXiv

  38. [46]

    Kao, K.-F

    H.-C. Kao, K.-F. Tang, E. Y. Chang. Context -Aware Symptom Checking for Disease Diagnosis Using Hierarchical Rein- forcement Learning. AAAI, 2018

    2018

  39. [47]

    Wei et al

    Z. Wei et al. Task -oriented Dialogue System for Automatic Di- agnosis. ACL, 2018

    2018

  40. [48]

    Li et al

    S. Li et al. MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning. NeurIPS (arXiv:2406.00922), 2024

    2024 arXiv

  41. [49]

    Tu et al

    T. Tu et al. Towards conversational diagnostic artificial intelli- gence (AMIE). Nature, 2025

    2025

  42. [50]

    Markel, S

    J. Markel, S. Opferman, J. Landay, C. Piech. GPTeach: Interac- tive TA Training with GPT -based Students. ACM Learn- ing@Scale, 2023. 19

    2023

  43. [51]

    Shea et al

    R. Shea et al. ACE: A LLM-based Negotiation Coaching System. arXiv:2410.01555, 2024

    2024 arXiv

  44. [52]

    Zhang et al

    Z. Zhang et al. Simulating Classroom Education with LLM-Em- powered Agents. arXiv:2406.19226, 2024

    2024 arXiv

  45. [53]

    Holderried et al

    F. Holderried et al. A Generative Pretrained Transformer (GPT)- Powered Chatbot as a Simulated Patient to Practice History Taking. JMIR Medical Education, 2024

    2024

  46. [54]

    Johri et al

    S. Johri et al. An evaluation framework for clinical use of large language models in patient interaction tasks (CRAFT -MD). Nature Medicine, 2025

    2025

  47. [55]

    Jour- nal of Medical Internet Research, 2025

    Embracing the Future of Medical Education With Large Lan- guage Model-Based Virtual Patients: Scoping Review. Jour- nal of Medical Internet Research, 2025

    2025

  48. [56]

    JMIR Medical Informatics, 2026

    Large Language Model-Based Virtual Patient Systems for His- tory-Taking in Medical Education: A Comprehensive Sys- tematic Review. JMIR Medical Informatics, 2026

    2026

  49. [57]

    Das et al

    S. Das et al. Synthetic Patient -Physician Dialogue Generation from Clinical Notes Using LLM (SynDial). arXiv:2408.06285, 2024

    2024 arXiv

  50. [58]

    Ben Abacha et al

    A. Ben Abacha et al. Overview of the MEDIQA -Chat 2023 Shared Tasks on the Summarization and Generation of Doc- tor-Patient Conversations. ACL ClinicalNLP, 2023

    2023

  51. [59]

    Chintagunta et al

    B. Chintagunta et al. Can LLMs generate high -quality synthetic note-oriented doctor -patient conversations? arXiv:2306.16931, 2023

    2023 arXiv

  52. [60]

    Du et al

    Y. Du et al. LLMs Can Simulate Standardized Patients via Agent Coevolution (EvoPatient). ACL (arXiv:2412.11716), 2025

    2025 arXiv

  53. [61]

    Kang et al

    D. Kang et al. PatientSim: A Persona -Driven Simulator for Re- alistic Doctor-Patient Interactions. arXiv:2505.17818, 2025

    2025

  54. [62]

    Ge et al

    X. Ge et al. EMSDialog: Synthetic Multi-person EMS Dialogue Generation from Electronic Patient Care Reports via Multi - LLM Agents. ACL Findings (arXiv:2604.07549), 2026

    2026 arXiv

  55. [63]

    BMC Emergency Medicine (arXiv:2510.21228), 2026

    DispatchMAS: Fusing Taxonomy and AI Agents for Emergency Medical Services. BMC Emergency Medicine (arXiv:2510.21228), 2026

    2026

  56. [64]

    Prehospital and Disaster Medicine (PubMed 39675178), 2024

    Use of a large language model for ambulance dispatch and triage. Prehospital and Disaster Medicine (PubMed 39675178), 2024

    2024

  57. [65]

    Hartman et al

    V. Hartman et al. Developing and Evaluating Large Language Model-Generated Emergency Medicine Handoff Notes. JAMA Network Open, 2024

    2024

  58. [66]

    Gatto et al

    J. Gatto et al. In-Context Learning for Preserving Patient Privacy: Synthesizing Realistic Patient Portal Messages. ML4H (arXiv:2411.06549), 2024

    2024 arXiv

  59. [67]

    JAMIA, 32(6):1032, 2025

    Detecting emergencies in patient portal messages using LLMs and knowledge-graph-based RAG. JAMIA, 32(6):1032, 2025

    2025

  60. [68]

    Yao et al

    Z. Yao et al. DischargeSim: A Simulation Benchmark for Edu- cational Doctor -Patient Communication at Discharge. arXiv:2509.07188, 2025

    2025

  61. [69]

    Xu et al

    J. Xu et al. Overview of the First Shared Task on Clinical Text Generation: RRG24 and Discharge Me! BioNLP, ACL (arXiv:2409.16603), 2024

    2024 arXiv

  62. [70]

    A. Kang, J. Y. Chen, Z. Lee-Youngzie, et al. Synthetic Data Gen- eration with LLM for Improved Depression Prediction. arXiv:2411.17672, 2024

    2024 arXiv

  63. [71]

    Ramachandran et al

    P. Ramachandran et al. Synth-SBDH: A Synthetic Dataset of So- cial and Behavioral Determinants of Health for Clinical Text. arXiv:2406.06056, 2024

    2024 arXiv

  64. [72]

    Fan et al

    Z. Fan et al. AI Hospital: Benchmarking LLMs in a Multi-agent Medical Interaction Simulator. arXiv:2402.09742, 2024

    2024 arXiv

  65. [73]

    Louie et al

    R. Louie et al. Roleplay-doh: Enabling Domain-Experts to Cre- ate LLM -simulated Patients via Eliciting and Adhering to Principles. EMNLP, 2024

    2024

  66. [74]

    Keerthana, M

    G. Keerthana, M. Gupta. CLI -RAG: A Retrieval -Augmented Framework for Clinically Structured and Context-Aware Text Generation with LLMs. arXiv:2507.06715, 2025

    2025 arXiv

  67. [75]

    Zhang, M

    K. Zhang, M. Hu, H. A. D. Le, et al. A Survey on Evaluating Quality and Trustworthiness in LLM -Generated Data. arXiv:2601.17717, 2026

    2026 arXiv

  68. [76]

    Mishra et al

    P. Mishra et al. SYNFAC-EDIT: Synthetic Imitation Edit Feed- back for Factual Alignment in Clinical Summarization. arXiv:2402.13919, 2024

    2024 arXiv

  69. [77]

    arXiv:2502.14921, 2025

    The Canary's Echo: Auditing Privacy Risks of LLM -Generated Synthetic Text. arXiv:2502.14921, 2025

    2025

  70. [78]

    Y. Sun, V. Schlegel, S. Nandakumar, et al. Evaluating Differen- tially Private Generation of Domain -Specific Text. arXiv:2508.20452, 2025

    2025 arXiv

  71. [79]

    Nayak et al

    S. Nayak et al. Synthetic data distillation enables the extraction of clinical information at scale. npj Digital Medicine, 2025

    2025

  72. [80]

    Iglesias, G

    G. Iglesias, G. Bello-Orgaz, M. Navas-Loro, et al. Fidelity, Di- versity, and Privacy: A Multi -Dimensional LLM Evaluation for Clinical Data Augmentation. arXiv:2604.27014, 2026

    2026 arXiv

  73. [81]

    Räisä, B

    O. Räisä, B. van Breugel, M. van der Schaar. Position: All Cur- rent Generative Fidelity and Diversity Metrics are Flawed. arXiv:2505.22450, 2025

    2025 arXiv

  74. [82]

    Asgari, N

    E. Asgari, N. Montaña-Brown, M. Dubois, et al. A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. npj Digital Medicine, 2025

    2025

  75. [83]

    Bedrick, A

    S. Bedrick, A. S. Doğruöz, S. Nisioi. A Typology of Synthetic Datasets for Dialogue Processing in Clinical Contexts. LREC (arXiv:2505.03025), 2026

    2026

  76. [84]

    J. R. Hampton, M. J. G. Harrison, J. R. A. Mitchell, J. S. Prichard, C. Seymour. Relative contributions of history-taking, physical examination, and laboratory investigation to diagnosis and management of medical outpatients. British Medical Journal, 2(5969):486–489, 1975

    1975

  77. [85]

    Malpractice Risks in Communication Fail- ures: 2015 Annual Benchmarking Report

    CRICO Strategies. Malpractice Risks in Communication Fail- ures: 2015 Annual Benchmarking Report. Cambridge, MA,

    2015

  78. [86]

    Sentinel Event Data Summary (annual root-cause reports)

    The Joint Commission. Sentinel Event Data Summary (annual root-cause reports). Oakbrook Terrace, IL. jointcommis- sion.org/resources/sentinel-event/sentinel-event-data-sum- mary 20

  79. [87]

    Iedema et al

    R. Iedema et al. Clinical handover between paramedics and emergency department staff: the SBAR and IMIST -AMBO acronyms. International Journal of Paramedic Practice

  80. [88]

    T. M. Seinen, J. A. Kors, E. M. van Mulligen, et al. Using Struc- tured Codes and Free -Text Notes to Measure Information Complementarity in Electronic Health Records. Journal of Medical Internet Research, 2025

    2025

  81. [89]

    Nath et al

    B. Nath et al. Patient Portal Message Volume and Time Spent on the EHR: An Observational Study of Primary Care Clinicians. Journal of General Internal Medicine, 2024

    2024

  82. [90]

    Joshi, K

    I. Joshi, K. Kume, C. Drucker, et al. Impact of language barrier during emergency calls on out-of-hospital cardiac arrest treat- ment and outcomes. Resuscitation, 2025

    2025

  83. [91]

    S. N. Blomberg et al. Machine learning as a supportive tool to recognize cardiac arrest in emergency calls. Resuscitation, 138:322–329, 2019

    2019

  84. [92]

    S. N. Blomberg et al. Effect of Machine Learning on Dispatcher Recognition of Out -of-Hospital Cardiac Arrest During Calls to Emergency Medical Services: A Randomized Clinical Trial. JAMA Network Open, 2021

    2021

  85. [93]

    P. J. Lukac, W. Turner, S. Vangala, et al. Ambient AI Scribes in Clinical Practice: A Randomized Trial. NEJM AI, 2025

    2025

  86. [94]

    Wang et al

    H. Wang et al. An Emergency Medical Services Clinical Audit System driven by Named Entity Recognition from Deep Learning. arXiv:2007.03596, 2020

    2007 arXiv

  87. [95]

    Wang et al

    L. Wang et al. Using natural language processing in emergency medicine health service research: A systematic review and meta-analysis. Academic Emergency Medicine, 2024

    2024

  88. [96]

    N. W. Sterling et al. Applications of natural language processing at emergency department triage: A narrative review. PLOS ONE, 2023

    2023

  89. [97]

    R. M. Cronin et al. A comparison of rule -based and machine learning approaches for classifying patient portal messages. International Journal of Medical Informatics, 2017

    2017

  90. [98]

    Lim et al

    J. Lim et al. MATRIX: Multi -Agent Simulation Framework for Safe Interactions and Contextual Clinical Conversational Evaluation. arXiv:2508.19163, 2025

    2025 arXiv

  91. [99]

    Schmidgall et al

    S. Schmidgall et al. AgentClinic: A Multimodal Agent Bench- mark to Evaluate AI in Simulated Clinical Environments. arXiv:2405.07960, 2024

    2024 arXiv

  92. [100]

    Qin et al

    Z. Qin et al. Scaling Laws of Synthetic Data for Language Mod- els. COLM (arXiv:2503.19551), 2025

    2025

  93. [101]

    Shumailov et al

    I. Shumailov et al. The Curse of Recursion: Training on Gener- ated Data Makes Models Forget. arXiv:2305.17493, 2023

    2023 arXiv

  94. [102]

    Bao et al

    Z. Bao et al. A Survey on Recent Advances in Conversational Data Generation. arXiv:2405.13003, 2024

    2024 arXiv

  95. [103]

    Wang et al

    Y. Wang et al. M4: Multi-generator, Multi-domain, and Multi- lingual Black -Box Machine -Generated Text Detection. EACL, 2024

    2024

  96. [104]

    Yao et al

    Z. Yao et al. MedQA-CS: An OSCE-Style Benchmark for Eval- uating LLM Clinical Skills. arXiv:2410.01553, 2024

    2024

  97. [105]

    N. Barman. Symptom2Disease. Kaggle dataset, 2023. kaggle.com/datasets/niyarrbarman/symptom2disease

    2023

  98. [106]

    Elhalimi et al

    Y. Elhalimi et al. MedDec: A Dataset for Extracting Clinical Decisions from Discharge Summaries. Findings of ACL / PhysioNet, 2024

    2024

  99. [107]

    Johnson et al

    A. Johnson et al. MIMIC-IV-ED (v2.2). PhysioNet, 2023

    2023

  100. [108]

    arXiv:2601.13178, 2026

    Medical Triage as Pairwise Ranking (PMR -Bench). arXiv:2601.13178, 2026

    2026

  101. [2015]

    candello.com/Insights/Candello-Reports

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.