Leveraging Generative AI Through Prompt Engineering and Rigorous Validation to Create Comprehensive Synthetic Datasets for AI Training in Healthcare

Polycarp Nalela

REVIEW 3 major objections 4 minor 23 references

Leveraging Generative AI Through Prompt Engineering and Rigorous Validation to Create Comprehensive Synthetic Datasets for AI Training in Healthcare

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GPT-4 with prompt engineering and rigorous validation can generate high-quality synthetic EHR data for AI training.

desk verdict A broad synthetic EHR pipeline whose validation never touches real data, so the central quality claim is unproven, though it is a useful workflow enumeration. read the letter →

arxiv 2504.20921 v1 pith:DIRREYJZ submitted 2025-04-29 cs.AI

Polycarp Nalela This is my paper · ORCID

classification cs.AI

keywords PromptEngineeringMedicalSyntheticDataElectronicHealthRecordsArtificialIntelligencevalidationGPT-4anomalydetectionPostgreSQL

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a generative AI pipeline—crafting table-specific prompts for GPT-4 and then screening every record with multiple language models—can produce synthetic hospital data good enough to train healthcare AI. The motivation is concrete: privacy regulations make real patient records hard to obtain, so a scalable source of realistic records would unblock EHR applications. To back this up, the authors generated 445,500 records spanning 22 tables, ran coherence, plausibility, consistency, anomaly, and diversity checks, and integrated the survivors into a PostgreSQL EHR database. The paper argues that the resulting dataset is high-quality, privacy-compliant, and ready for AI training. A sympathetic reader would take the central hypothesis as: rigorous validation is what turns raw LLM output into trustworthy synthetic medical data.

What carries the argument

The carrying mechanism is the validation pipeline itself, not any single generative or evaluative model. BERT's Next Sentence Prediction checks that consecutive sentences in a clinical narrative follow logically; GPT-2 perplexity scores how closely a record resembles natural medical text; RoBERTa's NLI determines whether field pairs (for example, a penicillin allergy and a prescribed antibiotic) entail, contradict, or stay neutral; an autoencoder flags rows whose reconstruction error exceeds the mean plus two standard deviations. Diversity analysis via the Shannon Diversity Index and category coverage completes the loop, and only records that clear the bar are loaded into PostgreSQL.

What would settle it

A concrete check would be to train a standard predictive model on the synthetic corpus and evaluate it on a held-out set of real electronic health records; if performance is substantially worse than a model trained on real data—or if clinical reviewers find records that pass all validation thresholds but are medically impossible—the central claim would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that a GPT-4-based generation pipeline, driven by per-table prompts, produces comprehensive EHR-scale synthetic data that a five-part validation framework certifies as coherent, plausible, consistent, minimally anomalous, and diverse. The paper's own reported numbers carry the claim: BERT NSP probabilities are essentially 1.0, the minimum consistency score from RoBERTa is 0.9750, GPT-2 perplexity clusters between 20 and 50, and autoencoder reconstruction errors concentrate below 0.08. By these criteria, the generated dataset—42 patients, 445,500 records, 49 features across 22 tables—is judged ready for AI training, and the validated records are loaded into a PostgreSQL database as the EHR system's data layer.

Load-bearing premise

The load-bearing premise is that the automated checks—coherence, perplexity, consistency, and anomaly scores—actually capture whether synthetic records are high-quality enough for medical AI, because the paper never compares the generated data to real EHR records or to clinician judgment.

Editorial extensions

If this is right

If the claim is right, AI systems for EHR applications can be developed and benchmarked on synthetic records without obtaining real patient data, lowering the privacy barrier.
The 22-table relational structure means the pipeline can populate a full hospital database, not just isolated notes, so downstream AI tasks can use linked admissions, labs, medications, and discharge summaries.
The thresholded validation scores give a concrete release criterion: data that passes NSP, perplexity, NLI, and anomaly checks is deemed trainable, making the process repeatable.
Because generation is prompt-driven, the same approach can be re-targeted to new specialties, patient populations, or data schemas by rewriting prompts and re-running validation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not compare its synthetic records against real EHR data or test a downstream model, so the claim that validation scores equal clinical utility remains an assumption; a direct comparison would settle it.
A natural extension would be to train a diagnostic or prediction model on the synthetic corpus and evaluate it on held-out real records, translating internal quality scores into an external, task-level test.
The validation metrics are text-focused, so subtle numerical or temporal dependencies across tables—for example, lab timing relative to medication orders—may evade detection; a clinician-review study of flagged edge cases would probe this.
If validated here, the same pipeline could generalize to other regulated domains where synthetic training data must satisfy both realism and privacy constraints.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper describes a broad pipeline for generating synthetic EHR data with GPT-4 and a stack of language-model and autoencoder validators, but the validation never touches real data, so “high-quality” is not shown. That said, it is a coherent workflow description and worth engaging with as a checklist of pitfalls in synthetic-data validation.

What is actually new: the scope. 22 interrelated tables, 445,500 records, and a multi-layer validation stack (BERT NSP, GPT-2 perplexity, RoBERTa NLI, autoencoder reconstruction, Shannon diversity) is more than most synthetic EHR papers attempt. The PostgreSQL integration is a practical detail that makes the pipeline concrete. The writing is clear and the methods are sufficiently specified that, if prompts and code were shared, the pipeline could be reproduced. That counts for something.

Where it falls down: the evidence is self-referential. Every validation metric is computed on the generated data alone. The GPT-2 perplexity cutoff is the 95th percentile of the same data; the autoencoder threshold is mean plus two standard deviations of its own reconstruction errors. So by construction a fixed slice of records gets flagged, and the scores say nothing about whether the data are clinically plausible or useful. No comparison to real EHRs, no downstream task (train on synthetic, test on real). The claim that this “facilitates AI training” is not established. There is also an internal numerical inconsistency: Section 3.2 reports 22,950 records “fell below” the 95th-percentile perplexity threshold, but with 445,500 records that number is about 5%, which would be the count above the threshold. The anomaly count of 40,490 (about 9%) is not reconciled with the “high-quality” verdict. And the prompts, code, and data are not released, so the framework is not reproducible as presented.

Credit where due: the authors are not claiming a new algorithm; they claim a rigorous validation workflow. The idea of combining multiple validation lenses is reasonable, and the paper honestly describes each component. The soft spot is not the idea, it is the lack of any external anchor.

Who this is for: someone thinking about building a synthetic EHR pipeline will find a useful enumeration of validation options and schema design. It should not be cited as evidence that LM-based validation produces high-quality medical data.

Recommendation: I would not desk-reject it without a second thought, but I would not accept it either. It deserves a serious referee only if the editor asks the authors to add external validation (or explicitly reframe the contribution as a workflow description). As it stands, the central claim is unsupported. Send it to review with the expectation of major revision, or reject with an invitation to resubmit with downstream evaluation.

Referee Report

3 major / 4 minor

Summary. The manuscript proposes a pipeline for generating synthetic electronic health record (EHR) data using prompt engineering with GPT-4, targeting 22 database tables with customized prompts. The generated 445,500-record dataset is then subjected to a validation framework comprising BERT Next Sentence Prediction for coherence, GPT-2 perplexity for plausibility, RoBERTa Natural Language Inference for consistency, autoencoder reconstruction error for anomaly detection, and diversity analysis. The authors claim that this approach demonstrates that generative AI with rigorous validation can produce high-quality synthetic medical data that facilitates AI training while addressing privacy concerns. The validated data is integrated into a PostgreSQL database for an EHR system.

Significance. If the central claim were established, the pipeline would be a practical contribution to privacy-preserving healthcare AI, offering a scalable way to generate large, interconnected synthetic EHR datasets. The manuscript describes a multi-stage validation architecture and integrates the result into a relational database, which is a useful workflow sketch. However, the evidence presented does not support the claim of 'high-quality' data: every validation metric is computed on the synthetic data itself, thresholds are derived from the same data, and no external baseline, real-EHR comparison, or downstream task evaluation is provided. The paper thus remains a workflow description rather than a validated demonstration of synthetic data utility.

major comments (3)

[§2.3, §3.2, §3.3] The validation framework is self-referential in a way that makes the reported quality metrics uninformative. The perplexity threshold is defined as the 95th percentile of the generated data's own perplexity scores, and the autoencoder anomaly threshold is defined as the mean reconstruction error plus two standard deviations of the same data. Consequently, approximately 5% of records are flagged by the perplexity criterion and a fixed proportion by the reconstruction criterion by construction, regardless of whether the data are clinically realistic. The claim that these thresholds distinguish 'realistic' from 'implausible' records therefore has no external anchor. The manuscript provides no comparison with real EHR data distributions, no medical expert review, and no downstream task evaluation, so the central claim of producing 'high-quality synthetic medical data' is not supported.
[§3.2] There is an internal inconsistency in the plausibility assessment. The text states: 'From this analysis, 22,950 records fell below the predefined perplexity threshold', but the threshold is defined as the 95th percentile of the entire dataset of 445,500 records, and 22,950 is approximately 5.15% of that total. This number must refer to records that exceed the threshold, not those that fall below it. The same section then says 'Only a small fraction of records surpassed the threshold', which contradicts the earlier phrase. This inconsistency obscures the actual fraction of flagged records and must be corrected.
[§2.3, §4] The validation metrics measure linguistic fluency and internal logical consistency, not clinical or medical correctness. For example, BERT NSP coherence and RoBERTa NLI consistency assess whether sentences follow each other or whether premises and hypotheses align, but they do not verify that laboratory values fall within physiologically plausible ranges, that diagnoses correspond to symptoms, or that medication prescriptions follow clinical guidelines. The paper itself cites examples such as a potassium level of 15 mmol/L as an anomaly (Section 1.4), yet the autoencoder anomaly detection is based purely on reconstruction error of the generated data, and no clinical reference ranges are applied. Without an external medical standard, the assertion that the synthetic data are 'high-quality' and 'medically reasonable' is not established.

minor comments (4)

[§1.5, contributions] The sentence 'The study incorporate the validated synthetic data into a PostgreSQL database' should read 'incorporates' for subject-verb agreement.
[§2.2] The phrase 'Iterative refining these prompts' is grammatically incomplete; it should be 'Through iterative refinement of these prompts' or similar.
[§3.4] The combined anomaly scores are used in the results and discussion (Figures 5, text) but their computation is never defined in Section 2.3. The reader cannot interpret the range '10 to 120' or the claimed peaks without knowing how coherence, plausibility, and anomaly detection outcomes are combined.
[§2.3] The statement 'After integrating data from 42 patients across the 22 tables, we obtained a robust dataset comprising 445,500 records' is unexplained: a factor of roughly 10,600 records per patient is surprising and the relationship between patients and records is not described.

Circularity Check

2 steps flagged · score 6.0 of 10

Validation 'quality' is defined by thresholds fitted to the generated data itself, so the central claim of high-quality, training-ready synthetic data is partially circular.

fitted input called prediction [Section 2.3, plausibility assessment; Section 3.2.]
"The distribution of perplexity scores across records was visualized using histograms, with a predefined threshold (95th percentile) demarcating realistic records from implausible ones."

The 95th percentile is computed from the perplexity scores of the generated records themselves, so the split into 'realistic' versus 'implausible' records is tied by construction to the internal distribution of GPT-2 scores on the synthetic data. No independent anchor to real EHR text or to clinical validity is used. Consequently, the reported result that most records are plausible follows from the choice of a distributional cutoff, not from evidence that the data would support AI training.
fitted input called prediction [Section 2.3, anomaly detection; Section 3.3.]
"A threshold for anomalies was defined dynamically as the mean reconstruction error plus two standard deviations, capturing significant deviations from the dataset's normal patterns."

The anomaly threshold is estimated from the reconstruction errors on the very data being validated. 'Normal patterns' are the autoencoder's own reconstruction behavior on the synthetic records, so flagging records above mean plus two standard deviations is a property of the fitted error distribution, not a clinical or real-data benchmark. The conclusion that the dataset is robust because most errors cluster below this self-derived threshold therefore reduces to an internal consistency check rather than an independent measure of data quality.

full rationale

The paper does not rely on a self-citation chain, and no uniqueness theorem or prior-work ansatz is load-bearing. Its circularity is instead in the validation logic: the properties that are used to certify the synthetic data as 'high-quality' are computed on the generated data and their thresholds are fitted to the same data. The 95th-percentile perplexity cutoff and the autoencoder mean-plus-two-standard-deviations cutoff are distributional statistics of the generated records, so the statement that most records pass validation is true by construction. The central claim that the pipeline 'can effectively produce high-quality synthetic medical data, facilitating the training of AI algorithms' is then supported only by these self-referential metrics: coherence, plausibility, consistency, and anomaly status are all defined with respect to language-model fluency or autoencoder reconstruction on the synthetic data, with no comparison to real EHRs and no downstream training-and-evaluation experiment on real medical records. This makes the quality conclusion partially circular and the training-utility claim unsupported. The paper is a plausible workflow description, but its headline empirical claim is not independently established. Score 6 reflects that the central validation outcome is partly forced by construction, while the dataset generation and integration components are non-circular contributions.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. It relies on standard ML models and statistical measures, but the assumptions that these internal metrics establish real-world data quality are load-bearing and untested.

free parameters (2)

Perplexity threshold = 95th percentile of the data's perplexity scores
Set based on the generated data distribution, not an external standard. Used in Section 2.3 to flag implausible records.
Anomaly threshold = Mean reconstruction error + 2 standard deviations
Computed from the autoencoder's reconstruction errors on the same data being evaluated. Used in Section 2.3.

assumptions (4)

domain assumption GPT-2 perplexity is a valid measure of medical narrative plausibility.
Invoked in Section 2.3 and 3.2 without evidence that GPT-2's language modeling reflects clinical correctness.
domain assumption BERT Next Sentence Prediction is a valid measure of clinical coherence.
Used in Section 2.3 to judge logical flow in medical narratives, though NSP is a general language task, not a clinical metric.
domain assumption RoBERTa Natural Language Inference is a valid measure of logical consistency between medical fields.
Applied in Section 2.3 to relationships like allergy and medication, but NLI is not trained on medical domain rules.
domain assumption Autoencoder reconstruction error is a valid indicator of data quality in healthcare records.
Used in Section 2.3 to flag anomalies, but reconstruction error only captures statistical rarity, not clinical validity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Leveraging Generative AI Through Prompt Engineering and Rigorous Validation to Create Comprehensive Synthetic Datasets for AI Training in Healthcare." pith.science (2026). https://pith.science/paper/DIRREYJZ

@misc{pith2026250420921,
  author       = {Pith},
  title        = {Pith review of: Leveraging Generative AI Through Prompt Engineering and Rigorous Validation to Create Comprehensive Synthetic Datasets for AI Training in Healthcare},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DIRREYJZ}},
  note         = {Machine review of arXiv:2504.20921}
}

read the original abstract

Access to high-quality medical data is often restricted due to privacy concerns, posing significant challenges for training artificial intelligence (AI) algorithms within Electronic Health Record (EHR) applications. In this study, prompt engineering with the GPT-4 API was employed to generate high-quality synthetic datasets aimed at overcoming this limitation. The generated data encompassed a comprehensive array of patient admission information, including healthcare provider details, hospital departments, wards, bed assignments, patient demographics, emergency contacts, vital signs, immunizations, allergies, medical histories, appointments, hospital visits, laboratory tests, diagnoses, treatment plans, medications, clinical notes, visit logs, discharge summaries, and referrals. To ensure data quality and integrity, advanced validation techniques were implemented utilizing models such as BERT's Next Sentence Prediction for sentence coherence, GPT-2 for overall plausibility, RoBERTa for logical consistency, autoencoders for anomaly detection, and conducted diversity analysis. Synthetic data that met all validation criteria were integrated into a comprehensive PostgreSQL database, serving as the data management system for the EHR application. This approach demonstrates that leveraging generative AI models with rigorous validation can effectively produce high-quality synthetic medical data, facilitating the training of AI algorithms while addressing privacy concerns associated with real patient data.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 19 canonical work pages

[1]

Esteva, A. et al. A guide to deep learning in healthcare. Nature medicine 25, 24–29 (2019)

work page 2019
[2]

Jiang, F. et al. Artificial intelligence in healthcare: past, present and future. Stroke and vascular neurology 2 (2017)

work page 2017
[3]

Department of Health Human Services

U.S. Department of Health Human Services. Health information privacy (2015). Retrieved from https://www.hhs.gov/hipaa/index.html

work page 2015
[4]

Regulation (eu) 2016/679 of the european parliament and of the council

Regulation, P. Regulation (eu) 2016/679 of the european parliament and of the council. Regulation (eu) 679, 2016 (2016). 14

work page 2016
[5]

J., Lu, M

Chen, R. J., Lu, M. Y., Chen, T. Y., Williamson, D. F. & Mahmood, F. Syn- thetic data in machine learning for medicine and healthcare. Nature Biomedical Engineering 5, 493–497 (2021)

work page 2021
[6]

The rise of synthetic data will boost ai performance

Gonfalonieri, A. The rise of synthetic data will boost ai performance. Harvard Business Review (2020). Retrieved from https://hbr.org/2020/09/ the- rise-of-synthetic-data-will-boost-ai-performance

work page 2020
[7]

K., Lin, C

Baowaly, M. K., Lin, C. -C., Liu, C. -L. & Chen, K. -T. Synthesizing electronic health records using improved generative adversarial networks. Journal of the American Medical Informatics Association 26, 228–241 (2019)

work page 2019
[8]

& Van der Schaar, M

Yoon, J., Jarrett, D. & Van der Schaar, M. Time-series generative adversarial networks. Advances in neural information processing systems 32 (2019)

work page 2019

Show all 23 references

[9]

Choi, E. et al. Generating multi -label discrete patient records using generative adversarial networks , 286 –305 (PMLR, 2017)

2017
[10]

Brown, T. B. Language models are few -shot learners. arXiv preprint arXiv:2005.14165 (2020)

2020 arXiv
[11]

& McDonell, K

Reynolds, L. & McDonell, K. Prompt programming for large language models: Beyond the few -shot paradigm , 1–7 (2021)

2021
[12]

Wang, H. et al. Knowledge -aware graph neural networks with label smoothness regularization for recommender systems , 968–977 (2019)

2019
[13]

Kenton, J. D. M. -W. C. & Toutanova, L. K. Bert: Pre -training of deep bidirectional transformers for language understanding , Vol. 1, 2 (Minneapolis, Minnesota, 2019)

2019
[14]

Radford, A. et al. Language models are unsupervised multitask learners. OpenAI blog 1, 9 (2019)

2019
[15]

Roberta: A robustly optimized bert pretraining approach

Liu, Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 364 (2019)

2019 arXiv
[16]

& Cho, S

An, J. & Cho, S. Variational autoencoder based anomaly detection using reconstruction probability. Special lecture on IE 2, 1–18 (2015)

2015
[17]

Eastell, R. et al. Postmenopausal osteoporosis. Nature reviews Disease primers 2, 1–16 (2016)

2016
[18]

Lehmann, C. U. et al. Pediatric aspects of inpatient health information technology systems. Pediatrics 135, e756 –e768 (2015)

2015
[19]

Gennari, F. J. Hypokalemia. New England Journal of Medicine 339, 451 –458 (1998)

1998
[20]

C., Booth, S., Matari´c, M

Chang, A., Fontaine, M. C., Booth, S., Matari´c, M. J. & Nikolaidis, S. Quality- diversity generative sampling for learning with synthetic data , Vol. 38, 19805 – 19812 (2024)

2024
[21]

D., Martani, A., Shaw, D., Elger, B

Genevi`eve, L. D., Martani, A., Shaw, D., Elger, B. S. & Wangmo, T. Structural racism in precision medicine: leaving no one behind. BMC Medical Ethics 21, 1–13 (2020)

2020
[22]

& Kadambi, A

McDuff, D., Curran, T. & Kadambi, A. Synthetic data in healthcare. arXiv preprint arXiv:2304.03243 (2023)

2023 arXiv
[23]

& Cook, D

Dahmen, J. & Cook, D. Synsys: A synthetic data generation system for healthcare applications. Sensors 19, 1181 (2019)

2019

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Esteva, A. et al. A guide to deep learning in healthcare. Nature medicine 25, 24–29 (2019)

work page 2019

[2] [2]

Jiang, F. et al. Artificial intelligence in healthcare: past, present and future. Stroke and vascular neurology 2 (2017)

work page 2017

[3] [3]

Department of Health Human Services

U.S. Department of Health Human Services. Health information privacy (2015). Retrieved from https://www.hhs.gov/hipaa/index.html

work page 2015

[4] [4]

Regulation (eu) 2016/679 of the european parliament and of the council

Regulation, P. Regulation (eu) 2016/679 of the european parliament and of the council. Regulation (eu) 679, 2016 (2016). 14

work page 2016

[5] [5]

J., Lu, M

Chen, R. J., Lu, M. Y., Chen, T. Y., Williamson, D. F. & Mahmood, F. Syn- thetic data in machine learning for medicine and healthcare. Nature Biomedical Engineering 5, 493–497 (2021)

work page 2021

[6] [6]

The rise of synthetic data will boost ai performance

Gonfalonieri, A. The rise of synthetic data will boost ai performance. Harvard Business Review (2020). Retrieved from https://hbr.org/2020/09/ the- rise-of-synthetic-data-will-boost-ai-performance

work page 2020

[7] [7]

K., Lin, C

Baowaly, M. K., Lin, C. -C., Liu, C. -L. & Chen, K. -T. Synthesizing electronic health records using improved generative adversarial networks. Journal of the American Medical Informatics Association 26, 228–241 (2019)

work page 2019

[8] [8]

& Van der Schaar, M

Yoon, J., Jarrett, D. & Van der Schaar, M. Time-series generative adversarial networks. Advances in neural information processing systems 32 (2019)

work page 2019