Differentiate ChatGPT-generated and Human-written Medical Texts

Dajiang Zhu; Haixing Dai; Hongmin Cai; Shaochen Xu; Tianming Liu; Wenxiong Liao; Xiang Li; Xiaoke Huang; Yiyang Zhang; Zhengliang Liu

Differentiate ChatGPT-generated and Human-written Medical Texts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.11567 v1 pith:Q6S2YL2U submitted 2023-04-23 cs.CL cs.AI

Wenxiong Liao , Zhengliang Liu , Haixing Dai , Shaochen Xu , Zihao Wu , Yiyang Zhang , Xiaoke Huang , Dajiang Zhu

show 3 more authors

Hongmin Cai Tianming Liu Xiang Li

This is my paper · ORCID

classification cs.CLcs.AI

keywords medicaltextschatgptgeneratedcontentdetectwrittenchatgpt-generated

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

Background: Large language models such as ChatGPT are capable of generating grammatically perfect and human-like text content, and a large number of ChatGPT-generated texts have appeared on the Internet. However, medical texts such as clinical notes and diagnoses require rigorous validation, and erroneous medical content generated by ChatGPT could potentially lead to disinformation that poses significant harm to healthcare and the general public. Objective: This research is among the first studies on responsible and ethical AIGC (Artificial Intelligence Generated Content) in medicine. We focus on analyzing the differences between medical texts written by human experts and generated by ChatGPT, and designing machine learning workflows to effectively detect and differentiate medical texts generated by ChatGPT. Methods: We first construct a suite of datasets containing medical texts written by human experts and generated by ChatGPT. In the next step, we analyze the linguistic features of these two types of content and uncover differences in vocabulary, part-of-speech, dependency, sentiment, perplexity, etc. Finally, we design and implement machine learning methods to detect medical text generated by ChatGPT. Results: Medical texts written by humans are more concrete, more diverse, and typically contain more useful information, while medical texts generated by ChatGPT pay more attention to fluency and logic, and usually express general terminologies rather than effective information specific to the context of the problem. A BERT-based model can effectively detect medical texts generated by ChatGPT, and the F1 exceeds 95%.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Detecting AI-Generated Text in Educational Content: Leveraging Machine Learning and Explainable AI for Academic Integrity
cs.CL 2025-01 conditional novelty 4.0 of 10

XGBoost and Random Forest can distinguish ChatGPT-written cybersecurity paragraphs from human Wikipedia paragraphs with 81 to 83% accuracy, and a narrow XGBoost model beat GPTZero in a three-class test, yet the benchm...
Using Machine Learning to Distinguish Human-written from Machine-generated Creative Fiction
cs.CL 2024-12 conditional novelty 4.0 of 10

Naive Bayes and MLP classifiers distinguish about 100-word excerpts of human-written detective fiction from ChatGPT-3.5 output with roughly 96% accuracy on in-domain tests.

Pith tools