pith. sign in

arxiv: 2604.08555 · v1 · submitted 2026-03-16 · 💻 cs.CL

SynDocDis: A Metadata-Driven Framework for Generating Synthetic Physician Discussions Using Large Language Models

Beny Rubinstein , Sergio Matos This is my paper

Pith reviewed 2026-05-15 10:15 UTC · model grok-4.3

classification 💻 cs.CL
keywords synthetic physician dialogueslarge language modelsmedical data privacyoncologyhepatologyAI in medicinedialogue generationstructured prompting
0
0 comments X

The pith

SynDocDis generates synthetic physician-to-physician dialogues from de-identified metadata using LLMs and structured prompting.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents SynDocDis, a framework that creates realistic physician-to-physician discussions about patient cases by feeding de-identified metadata into large language models through structured prompts. This addresses the barrier of privacy rules that block access to real clinical conversations while filling the gap left by existing synthetic data methods focused on patient-doctor exchanges. The approach targets oncology and hepatology scenarios to produce dialogues that can train medical AI systems. When tested by five practicing physicians across nine cases, the outputs received mean ratings of 4.4 out of 5 for communication effectiveness and 4.1 out of 5 for medical content quality, with 91 percent rated clinically relevant and strong agreement among raters. The result is a privacy-safe way to build datasets for medical education and clinical decision support.

Core claim

SynDocDis is a metadata-driven framework that applies structured prompting techniques to large language models to generate clinically accurate physician-to-physician dialogues. In nine oncology and hepatology scenarios, evaluations by five practicing physicians produced mean scores of 4.4/5 for communication effectiveness and 4.1/5 for medical content quality, with interrater reliability of kappa 0.70 and 91 percent clinical relevance ratings, all while preserving the privacy of doctors and patients.

What carries the argument

Structured prompting with de-identified case metadata that guides LLMs to synthesize physician reasoning and dialogue.

If this is right

  • Supplies privacy-compliant training data for AI agents that need to understand or participate in physician-level clinical reasoning.
  • Creates accessible examples of case discussions for medical education and training programs.
  • Supports development of clinical decision support tools that rely on realistic multi-doctor interactions.
  • Extends synthetic data methods to physician-to-physician communication in specialized fields.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same metadata-driven prompting could be adapted to generate discussions in other medical specialties beyond oncology and hepatology.
  • Pairing the outputs with automated fact-checking tools could reduce reliance on subjective physician ratings alone.
  • The dialogues might serve as starting points for simulating team-based decision making in complex or multi-specialty cases.

Load-bearing premise

Structured prompting with de-identified metadata is sufficient for LLMs to produce clinically accurate physician reasoning and dialogue.

What would settle it

A study that compares the generated dialogues against real anonymized physician discussions using objective clinical accuracy checks or a much larger set of blinded physician reviews would test the claim.

Figures

Figures reproduced from arXiv: 2604.08555 by Beny Rubinstein, Sergio Matos.

Figure 1
Figure 1. Figure 1: Overview of study design and workflow. 3.2 Data Collection: Physician input through metadata We developed a data entry form and asked physicians to collect metadata de￾scribing real patient case discussions they were engaged in through their profes- [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Generation of synthetic physician discussions following the CIDI framework. User Prompting In addition to role-playing—asking the model to adopt a persona and act accordingly—we adopted a technique called emotion prompting, using capital letters to emphasize important aspects (see [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: User and System prompts used to generate synthetic patient case discussions [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Evaluation by medical experts (5=Excellent; 3=Adequate; 1=Criteria not met). To assess the agreement among physician evaluators, we employed the weighted Fleiss’ κ using quadratic weights to penalize disagreements between distant cat￾egories (e.g., 1 vs. 5), and calculated 95% confidence intervals using bootstrap resampling with 1000 iterations. The results show substantial agreement among the physician ev… view at source ↗
read the original abstract

Physician-physician discussions of patient cases represent a rich source of clinical knowledge and reasoning that could feed AI agents to enrich and even participate in subsequent interactions. However, privacy regulations and ethical considerations severely restrict access to such data. While synthetic data generation using Large Language Models offers a promising alternative, existing approaches primarily focus on patient-physician interactions or structured medical records, leaving a significant gap in physician-to-physician communication synthesis. We present SynDocDis, a novel framework that combines structured prompting techniques with privacy-preserving de-identified case metadata to generate clinically accurate physician-to-physician dialogues. Evaluation by five practicing physicians in nine oncology and hepatology scenarios demonstrated exceptional communication effectiveness (mean 4.4/5) and strong medical content quality (mean 4.1/5), with substantial interrater reliability (kappa = 0.70, 95% CI: 0.67-0.73). The framework achieved 91% clinical relevance ratings while maintaining doctors' and patients' privacy. These results place SynDocDis as a promising framework for advancing medical AI research ethically and responsibly through privacy-compliant synthetic physician dialogue generation with direct applications in medical education and clinical decision support.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces SynDocDis, a metadata-driven framework for generating synthetic physician-to-physician discussions using LLMs. It uses structured prompting with de-identified case metadata to create dialogues in oncology and hepatology scenarios. The evaluation involves five practicing physicians rating nine scenarios, reporting mean scores of 4.4/5 for communication effectiveness, 4.1/5 for medical content quality, 91% clinical relevance, and interrater reliability with kappa = 0.70.

Significance. If the central claims hold under more rigorous validation, this work could be significant for medical AI research by providing a privacy-preserving method to generate synthetic data for training clinical decision support systems and medical education tools, addressing the gap in physician-to-physician dialogue synthesis.

major comments (2)
  1. [Evaluation] Evaluation section: The claim of clinically accurate physician reasoning rests on subjective 5-point ratings from only 5 physicians across 9 scenarios, with no baselines, no detailed generation prompts, and no objective metrics such as fact verification of specific assertions or comparison to real de-identified cases. This makes the reported means (4.4/5 communication, 4.1/5 content) and 91% relevance insufficient to substantiate clinical fidelity, as interrater agreement (kappa=0.70) shows consistency but not correctness.
  2. [Methods] Methods section: The structured prompting techniques and the precise structure of the de-identified metadata are not described in sufficient detail to allow reproduction or independent assessment of whether the approach yields faithful medical reasoning chains.
minor comments (1)
  1. [Abstract] The abstract would benefit from briefly noting the small scale and subjective nature of the evaluation to balance the strong claims.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. The comments highlight important areas for improving reproducibility and the strength of our evaluation claims. We address each point below and commit to revisions that enhance the paper without overstating our current results.

read point-by-point responses

  1. Referee: [Evaluation] Evaluation section: The claim of clinically accurate physician reasoning rests on subjective 5-point ratings from only 5 physicians across 9 scenarios, with no baselines, no detailed generation prompts, and no objective metrics such as fact verification of specific assertions or comparison to real de-identified cases. This makes the reported means (4.4/5 communication, 4.1/5 content) and 91% relevance insufficient to substantiate clinical fidelity, as interrater agreement (kappa=0.70) shows consistency but not correctness.

    Authors: We agree that the evaluation is limited to subjective expert ratings and lacks baselines or objective verification metrics, which tempers the strength of claims about clinical fidelity. Expert ratings by practicing physicians remain a common and appropriate method for initial assessment of dialogue quality in medical AI research, and the reported kappa of 0.70 indicates good consistency. To strengthen the manuscript, we will add the detailed generation prompts to an appendix, expand the limitations section to explicitly discuss the absence of ground-truth comparisons due to privacy constraints, and include additional objective measures such as semantic similarity scores where feasible. We will also clarify that the current results represent a pilot evaluation of the framework rather than definitive proof of clinical accuracy. revision: partial

  2. Referee: [Methods] Methods section: The structured prompting techniques and the precise structure of the de-identified metadata are not described in sufficient detail to allow reproduction or independent assessment of whether the approach yields faithful medical reasoning chains.

    Authors: We agree that the current description of the prompting techniques and metadata structure is insufficient for full reproducibility. In the revised version, we will expand the Methods section to include the exact template structure for de-identified case metadata (listing all fields and their formats) and provide the complete structured prompting techniques, including verbatim example prompts used for oncology and hepatology scenarios. This will enable independent assessment and reproduction of the generation process. revision: yes

Circularity Check

0 steps flagged

No circularity detected in framework or evaluation chain

full rationale

The paper presents SynDocDis as a prompting-based framework that takes de-identified metadata as input to generate physician dialogues via LLMs, then evaluates outputs separately via external physician raters on independent scales for communication, content quality, relevance, and interrater agreement. No equations, fitted parameters renamed as predictions, self-citations supporting core claims, uniqueness theorems, or ansatzes appear in the text. The generation method and the human evaluation are distinct steps with no reduction of one to the other by construction, so the derivation chain remains self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that current LLMs can translate structured metadata into clinically faithful dialogues; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption Large language models guided by structured de-identified metadata can generate clinically accurate physician-to-physician dialogues
    This assumption underpins both the generation process and the claim of clinical relevance.

pith-pipeline@v0.9.0 · 5512 in / 1269 out tokens · 45718 ms · 2026-05-15T10:15:45.662918+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages

  1. [1]

    Ayers, Adam Poliak, Mark Dredze, Eric C

    Ayers, J.W., Poliak, A., Dredze, M., Leas, E.C., Zhu, Z., Kelley, J.B., Faix, D.J., Goodman, A.M., Longhurst, C.A., Hogarth, M., Smith, D.M.: Comparing physi- cian and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Internal Medicine183, 589 (6 2023). https: //doi.org/10.1001/jamainternmed.2023.18...

    work page doi:10.1001/jamainternmed.2023.1838 2023

  2. [2]

    Journal of Participa- tory Medicine (2025, forthcoming)

    Campos Jr., H., Wolfe, D., Luan, H., Sim, I.: Generative AI as third agent: LLMs and the transformation of the clinician-patient relationship. Journal of Participa- tory Medicine (2025, forthcoming). https://doi.org/10.2196/68146

    work page doi:10.2196/68146 2025

  3. [3]

    npj Digital Medicine6, 186 (10 2023)

    Giuffrè, M., Shung, D.L.: Harnessing the power of synthetic data in healthcare: innovation, application, and privacy. npj Digital Medicine6, 186 (10 2023). https://doi.org/10.1038/s41746-023-00927-3, https://www.nature. com/articles/s41746-023-00927-3

    work page doi:10.1038/s41746-023-00927-3 2023

  4. [4]

    JAMA Network Open7, e2448723 (12 2024)

    Hartman, V., Zhang, X., Poddar, R., McCarty, M., Fortenko, A., Sholle, E., Sharma, R., Campion, T., Steel, P.A.D.: Developing and evaluating large language model–generated emergency medicine handoff notes. JAMA Network Open7, e2448723 (12 2024). https://doi.org/10.1001/jamanetworkopen.2024.48723, https: //jamanetwork.com/journals/jamanetworkopen/fullartic...

    work page doi:10.1001/jamanetworkopen.2024.48723 2024

  5. [5]

    In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval

    Li, D., Ren, Z., Ren, P., Chen, Z., Fan, M., Ma, J., de Rijke, M.: Semi-supervised variational reasoning for medical dialogue generation. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval. pp. 544–554. ACM (7 2021). https://doi.org/10.1145/3404835. 3462921, https://dl.acm.org/doi/10.1145/...

    work page doi:10.1145/3404835 2021

  6. [6]

    In: Lu, W., Huang, S., Hong, Y., Zhou, X

    Liu, W., Tang, J., Cheng, Y., Li, W., Zheng, Y., Liang, X.: Meddg: An entity- centric medical consultation dataset for entity-aware medical dialogue generation. In: Lu, W., Huang, S., Hong, Y., Zhou, X. (eds.) Natural Language Processing and Chinese Computing. pp. 447–459. Springer International Publishing, Cham (2022)

    work page 2022

  7. [7]

    Bioethics37, 424–429 (6 2023)

    Lorenzini, G., Ossa, L.A., Shaw, D.M., Elger, B.S.: Artificial intelligence and the doctor–patient relationship expanding the paradigm of shared decision mak- ing. Bioethics37, 424–429 (6 2023). https://doi.org/10.1111/bioe.13158, https: //onlinelibrary.wiley.com/doi/10.1111/bioe.13158

    work page doi:10.1111/bioe.13158 2023

  8. [8]

    IOS Press (11 2024)

    Moser, D., Bender, M., Sariyar, M.: Generating Synthetic Healthcare Di- alogues in Emergency Medicine Using Large Language Models. IOS Press (11 2024). https://doi.org/10.3233/SHTI241099, https://ebooks.iospress.nl/doi/ 10.3233/SHTI241099

    work page doi:10.3233/shti241099 2024

  9. [9]

    Smith, Nima PourNejatian, Anthony B

    Peng, C., Yang, X., Chen, A., Smith, K.E., PourNejatian, N., Costa, A.B., Martin, C., Flores, M.G., Zhang, Y., Magoc, T., Lipori, G., Mitchell, D.A., Ospina, N.S., Ahmed, M.M., Hogan, W.R., Shenkman, E.A., Guo, Y., Bian, J., Wu, Y.: A study of generative large language model for medical research and healthcare. npj Digital Medicine6, 210 (11 2023). https:...

    work page doi:10.1038/s41746-023-00958-w 2023

  10. [10]

    Frontiers in Psychology15(8 2024)

    Riedl, R., Hogeterp, S.A., Reuter, M.: Do patients prefer a human doctor, ar- tificial intelligence, or a blend, and is this preference dependent on medical discipline? Empirical evidence and implications for medical practice. Frontiers in Psychology15(8 2024). https://doi.org/10.3389/fpsyg.2024.1422177, https: //www.frontiersin.org/articles/10.3389/fpsyg...

    work page doi:10.3389/fpsyg.2024.1422177 2024

  11. [11]

    2024 , journal =

    Sarkar, A.: AI should challenge, not obey. Communications of the ACM67, 18–21 (10 2024). https://doi.org/10.1145/3649404

    work page doi:10.1145/3649404 2024

  12. [12]

    npj Digital Medicine7, 20 (1 2024)

    Savage, T., Nayak, A., Gallo, R., Rangan, E., Chen, J.H.: Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine. npj Digital Medicine7, 20 (1 2024). https://doi.org/10.1038/s41746-024-01010-1

    work page doi:10.1038/s41746-024-01010-1 2024

  13. [13]

    JAMA Network Open7, e2422399 (7 2024)

    Small, W.R., Wiesenfeld, B., Brandfield-Harvey, B., Jonassen, Z., Mandal, S., Stevens, E.R., Major, V.J., Lostraglio, E., Szerencsy, A., Jones, S., Aphinyanaphongs, Y., Johnson, S.B., Nov, O., Mann, D.: Large language model–based responses to patients’ in-basket messages. JAMA Network Open7, e2422399 (7 2024). https://doi.org/10.1001/jamanetworkopen.2024....

    work page doi:10.1001/jamanetworkopen.2024.22399 2024

  14. [14]

    https://doi.org/10.1101/2025.03.04

    Spitzer, P., Hendriks, D., Rudolph, J., Schlaeger, S., Ricke, J., Kühl, N., Hoppe, B.F., Feuerriegel, S.: The effect of medical explanations from large language models on diagnostic decisions in radiology (3 2025). https://doi.org/10.1101/2025.03.04. 25323357, http://medrxiv.org/lookup/doi/10.1101/2025.03.04.25323357

    work page doi:10.1101/2025.03.04 2025

  15. [15]

    Information15, 264 (5 2024)

    Sufi, F.: Addressing data scarcity in the medical domain: A GPT-based approach for synthetic data generation and feature extraction. Information15, 264 (5 2024). https://doi.org/10.3390/info15050264, https://www.mdpi.com/2078-2489/ 15/5/264

    work page doi:10.3390/info15050264 2024

  16. [16]

    In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., ichter, b., Xia, F., Chi, E., Le, Q.V., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models. In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neural Information Processing Systems. vol. 35, pp. 24824– 24837.CurranAssociates,Inc.(2022),https:...

    work page 2022

  17. [17]

    JAMA Internal Medicine (5 2025)

    Williams, C.Y.K., Subramanian, C.R., Ali, S.S., Apolinario, M., Askin, E., Barish, P., Cheng, M., Deardorff, W.J., Donthi, N., Ganeshan, S., Huang, O., Kantor, Metadata-Driven Generation of Synthetic Physician Discussions 13 M.A., Lai, A.R., Manchanda, A., Moore, K.A., Muniyappa, A.N., Nair, G., Patel, P.P., Santhosh, L., Schneider, S., Torres, S., Yukawa...

    work page doi:10.1001/jamainternmed.2025 2025

  18. [18]

    In: Webber, B., Cohn, T., He, Y., Liu, Y

    Zeng, G., Yang, W., Ju, Z., Yang, Y., Wang, S., Zhang, R., Zhou, M., Zeng, J., Dong, X., Zhang, R., Fang, H., Zhu, P., Chen, S., Xie, P.: MedDialog: Large- scale medical dialogue datasets. In: Webber, B., Cohn, T., He, Y., Liu, Y. (eds.) Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP). pp. 9241–9250. Associ...

    work page doi:10.18653/v1/2020.emnlp-main.743 2020