Pith. sign in

REVIEW 4 major objections 5 minor 68 references

Dr. GPT Will See You Now, but Should It? Exploring the Benefits and Harms of Large Language Models in Medical Diagnosis using Crowdsourced Clinical Cases

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Physicians who scored 212 everyday health prompts to four public LLMs judged roughly 76% of the responses medically valid, with the best model still failing about one in five cases.

desk verdict The 76% accuracy headline is a fragile measurement that collapses to ~56% under a stricter cutoff, but the paper's cautious conclusion about LLM health advice survives, and the crowdsourced dataset is a real contribution. read the letter →

arxiv 2506.13805 v1 pith:ZTTAAND7 submitted 2025-06-13 cs.CY cs.AI

Bonam Mingole , Aditya Majumdar , Firdaus Ahmed Choudhury , Jennifer L. Kraschnewski , Shyam S. Sundar , Amulya Yadav This is my paper
classification cs.CYcs.AI
keywords largelanguagemodelsmedicaldiagnosiscrowdsourcedevaluationphysicianratingsretrieval-augmentedgenerationeverydayhealthqueriespatientsafetyin-the-wild
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper evaluates how well publicly available large language models answer the informal, everyday health questions ordinary people actually ask. Using 212 crowdsourced prompts to four LLMs, scored by nine board-certified physicians on a four-dimension rating instrument, the authors find that about 76% of responses are judged 'accurate' (a Validity score of 3 or higher on a 0–5 scale). The best model, GPT-4o, was valid in 84.6% of cases, while Llama3-8b was valid in only 50%. The paper's warning conclusion is that even the strongest model produced invalid answers in roughly one out of five cases, a failure rate the authors say is unacceptable for health decisions. A companion comparison found that retrieval-augmented (RAG) versions of the models did not reliably beat the baselines, and interviews with clinicians surfaced both benefits and harms of using LLMs for self-triage.

What carries the argument

The load-bearing instrument is a physician rating protocol adapted from the QUEST framework. For each of 212 responses, one board-certified physician assigns scores from 0 to 5 on Validity, Quality of Information, Understanding and Reasoning, and Harm; the paper's headline accuracy statistic is the share of responses with Validity ≥ 3. This single number carries the argument, so the threshold choice and the one-rater-per-entry design determine how the claim reads. The secondary mechanism is a RAG pipeline that chunks a medical-school curriculum, embeds chunks with a sentence-transformer model, retrieves the top five by cosine similarity, and appends them as context; RAG outputs were compared with baselines through blinded pairwise clinician preference with binomial tests.

What would settle it

Score the same 212 responses with at least two independent board-certified physicians per entry and compute inter-rater agreement. If agreement is low (e.g., Cohen's kappa below 0.6) or the consensus-based validity share differs substantially from 76.2%, the central accuracy claim would not be reproducible under stricter measurement. Alternatively, re-analyzing the existing data with a stricter validity threshold (e.g., Validity ≥ 4) would show how sensitive the claim is to the authors' chosen cutoff.

Watch

Extended reading notes

Core claim

The central claim is that, in the wild, LLM responses to everyday health queries are mostly—but not safely—accurate. On 212 competition entries, physicians' Validity ratings averaged 3.0 or higher in 76.2% of responses; accuracy by model was 84.6% (GPT-4o), 82.5% (GPT-3.5), 73.6% (Gemini-1.5 Pro), and 50.0% (Llama3-8b). Harm ratings were low overall, but the authors treat the remaining error rate as the decisive result: even the best model generates invalid responses in roughly one of every five cases, and acting on such errors could lead to harmful clinical outcomes. The paper also reports that stratified analyses found lower validity in Internal Medicine, Neurology, and Dermatology; that mid-length prompts (60–250 characters) yield the most valid responses; and that clinicians in a blinded pairwise test preferred baseline responses over RAG-enhanced ones for Gemini and Llama, with no significant difference for ChatGPT models. Qualitative interviews frame LLMs as useful for health literacy and preliminary triage but risky when users over-rely on them.

Load-bearing premise

The claim that 76% of LLM responses are accurate rests on the assumption that a single physician's 0–5 Validity rating, with no inter-rater reliability check and no gold-standard diagnosis, is a valid measure of accuracy; a different rater or a different cutoff could materially change the headline number.

Editorial extensions

If this is right

  • If the 76% validity estimate holds, the typical user asking a public LLM about an everyday symptom receives a medically valid response roughly three times out of four.
  • Even the top model fails about once in five queries, so using LLM answers as the sole basis for self-diagnosis or triage decisions carries material risk.
  • Model choice matters: smaller open models like Llama3-8b may be far less reliable (50% valid), so performance claims from one model should not be generalized.
  • Prompt format matters: queries of 60–250 characters produced the most valid responses; both very short and very long prompts reduced validity.
  • RAG enhancement with a curated medical knowledge base did not reliably improve clinician-rated response quality; for Gemini and Llama, baseline responses were preferred.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A user who makes repeated LLM health queries accumulates risk: if each query has a one-in-five failure chance at the best model, the probability of at least one invalid answer grows quickly with N queries, which matters for chronic or recurring symptom tracking.
  • The single-physician-per-response design leaves the headline figure unmeasured for inter-rater reliability; a multi-rater consensus study could shift the 76% estimate materially in either direction.
  • The RAG negative result may be specific to the single university curriculum and retrieval parameters; a broader or more current knowledge base could produce different relative preferences.
  • The prompt-length sweet spot suggests that lightweight user-facing guidance on how to phrase health queries could reduce invalid responses at no model cost—a testable intervention the paper does not itself run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports an in-the-wild evaluation of four publicly accessible LLMs for everyday health queries, using a university competition in which 34 participants submitted 212 prompts. Nine board-certified physicians rated each LLM response on validity, quality of information, understanding/reasoning, and harm using 0-5 Likert scales, and the paper's headline result is that ~76% of responses were deemed accurate. The authors also compare baseline LLMs against RAG-enhanced versions via blinded pairwise clinician preferences and conduct semi-structured interviews with seven medical professionals to derive qualitative themes about benefits and harms. The paper concludes that even the best model, GPT-4o, produces invalid responses in roughly one out of five cases and that LLMs should not be relied on for self-diagnosis.

Significance. The crowdsourced prompt design is a genuine strength: it captures real user-generated health queries rather than exam questions or expert-written vignettes, and the physician panel adds clinical grounding to the evaluation. The RAG comparison is also valuable because it tests a popular mitigation strategy on the same user-prompt corpus. The cautionary conclusion is robust in direction: even under the most favorable reading, no model is error-free, and the estimated failure rates are non-negligible. However, the headline quantitative claim of 76.2% accuracy is fragile because it depends on an unvalidated threshold, on single-physician ratings without inter-rater reliability or confidence intervals, and on an unbalanced model mix; the sample-size inconsistencies also need correction before the numbers can be used as published results.

major comments (4)
  1. [Section 4] The definition of an 'accurate' response as Validity >= 3, where 3 is labeled 'Average,' is introduced without justification, and the headline 76.2% figure is not robust to this cutoff. Figure 1 shows that 34% of entries scored Validity 4 and 22% scored 5, so raising the threshold to >=4 would reduce the aggregate accuracy from 76.2% to approximately 56%, a material change to the paper's central claim. The authors should either justify the chosen threshold on clinical or measurement grounds or report a sensitivity analysis across thresholds.
  2. [Section 3.2] Each of the 212 entries was assessed by exactly one physician, with no inter-rater reliability check, no gold-standard diagnosis, and no confidence intervals reported around the 76.2% figure. A single physician's Likert rating is thus treated as ground truth for 'accuracy,' and different raters or a stricter interpretation of the scale could shift the headline number materially. The authors should report rater agreement on a subset of responses (e.g., Cohen's kappa) and provide uncertainty intervals for the reported percentages.
  3. [Section 3.1 and Table 2] The paper reports inconsistent sample sizes: Section 3.1 states 212 total responses with 26 GPT-4o entries, while Table 2 lists n=213 with n=27 for GPT-4o, and Table 3 repeats n=27 for GPT-4o. Because the percentages in Table 2 cannot be reproduced from the counts given in the text, the data reporting needs to be corrected so that all tables and the text rely on the same underlying counts.
  4. [Section 4 and Table 2] The aggregate 76.2% figure is dominated by Gemini-1.5 Pro, which contributed 140 of 212 entries according to Section 3.1, so the aggregate is a participant-usage-weighted statistic rather than an average over models. Per-model validities in Table 2 range from 50.0% (Llama3-8b) to 84.6% (GPT-4o), and these per-model values, with confidence intervals and ideally a model-balanced aggregate, should be the primary comparative quantities; the current presentation conflates usage patterns with model capability.
minor comments (5)
  1. [Section 4.2] The sentence 'this fundamental information deficit can creates challenges for the LLM' contains a subject-verb agreement error and should read 'can create challenges.'
  2. [Ethical Considerations Statement] There is a typo: 'in reponse to their health concerns' should be 'in response to their health concerns.'
  3. [Table 3 and Appendix 16] The Gemini-1.5 Pro p-value is reported as 0.0000; it should be reported as p < 0.001 to avoid implying an exact zero probability.
  4. [Section 5.2] The pairwise comparison section states that response pairs were 'randomly and evenly distributed' but does not state whether the medical professionals were blinded to which response came from the baseline versus the RAG-enhanced model; if blinding was not used, this should be disclosed and discussed as a possible source of bias.
  5. [Section 4.1] The hypothesis that Internal Medicine reviewers applied stricter standards because they were overrepresented on the panel is speculative and should be flagged as such or tested with rater-specialty data; as written, it is presented as a possible explanation without supporting evidence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an empirical measurement with physician ratings as external ground truth.

full rationale

This paper contains no derivation chain that could be circular. Its central claims are direct empirical measurements: LLM outputs were generated by participants, then rated by board-certified physicians, and percentages were tabulated from those ratings. The definition of an 'accurate' response as Validity >= 3 (Section 4) is an explicit operationalization of the measured rating, not a fitted parameter or a predicted quantity derived from another input; it is a potential robustness or cutoff concern, but not a circular step. No parameters are fitted to a subset of the data and then announced as predictions. The comparison of baseline versus RAG LLM responses rests on independent blinded physician preference, and the qualitative insights come from interviews; neither reduces to the authors' own inputs. The paper's methodological citations (QUEST framework, LangChain, binomial test) are external tools, and no load-bearing argument is justified solely by a self-citation. The internal count inconsistency (212 in Section 3.1 versus n=213 in Table 2) and the single-rater design are correctness or measurement limitations, but they do not make any claim true by construction. I therefore find no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. Its quantitative claims rest on hand-chosen cutoffs for accuracy, harm, and quality, on the assumption that a single physician's rating is ground truth, on the representativeness of a volunteer sample, and on the assumption that pairwise preferences by seven medical professionals measure clinical appropriateness.

free parameters (4)
  • accuracy_threshold = Validity >= 3 on a 0-5 Likert scale
    Defines an LLM response as 'accurate' in Table 2 and the 76.2% headline; this cutoff is hand-chosen and not justified by clinical criteria.
  • harm_threshold = Harm <= 2 on a 0-5 Likert scale
    Defines 'not harmful' in Table 2; arbitrary cutoff.
  • quality_thresholds = QoI and UaR >= 3
    Defines 'strong response quality' for QoI and UaR in Table 2.
  • specialty_inclusion_cutoff = at least 10 entries per specialty
    Specialty subgroup analysis restricted to n >= 10 (Section 4.1), a post-hoc choice that shapes reported specialty patterns.
assumptions (4)
  • domain assumption A single board-certified physician's Likert rating is a valid and reliable measure of response accuracy and harm.
    Section 3.2 assigns each entry to exactly one physician with no inter-rater reliability check; the entire 76% accuracy claim and harm percentages rest on this.
  • domain assumption Self-selected participants' prompts are representative of everyday health queries by the general public.
    Section 3.1 recruits volunteers from one university, 65% frequent GenAI users and 22 of 34 from CS-related departments, yet the paper frames this as 'in-the-wild' evaluation.
  • domain assumption The adapted QUEST criteria are interpreted consistently across raters and specialties.
    The evaluation form (Appendix 12) lists criteria, but without training or IRR evidence, consistent interpretation is assumed.
  • domain assumption Physicians' preferences in the RAG pairwise comparison reflect clinical appropriateness rather than style or wording.
    Section 5.2 asks 7 medical professionals to pick the 'more clinically appropriate' response; no rubric or calibration is described, and preferences may be driven by readability or length.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dr. GPT Will See You Now, but Should It? Exploring the Benefits and Harms of Large Language Models in Medical Diagnosis using Crowdsourced Clinical Cases." pith.science (2026). https://pith.science/paper/ZTTAAND7

@misc{pith2026250613805,
  author       = {Pith},
  title        = {Pith review of: Dr. GPT Will See You Now, but Should It? Exploring the Benefits and Harms of Large Language Models in Medical Diagnosis using Crowdsourced Clinical Cases},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZTTAAND7}},
  note         = {Machine review of arXiv:2506.13805}
}
read the original abstract

The proliferation of Large Language Models (LLMs) in high-stakes applications such as medical (self-)diagnosis and preliminary triage raises significant ethical and practical concerns about the effectiveness, appropriateness, and possible harmfulness of the use of these technologies for health-related concerns and queries. Some prior work has considered the effectiveness of LLMs in answering expert-written health queries/prompts, questions from medical examination banks, or queries based on pre-existing clinical cases. Unfortunately, these existing studies completely ignore an in-the-wild evaluation of the effectiveness of LLMs in answering everyday health concerns and queries typically asked by general users, which corresponds to the more prevalent use case for LLMs. To address this research gap, this paper presents the findings from a university-level competition that leveraged a novel, crowdsourced approach for evaluating the effectiveness of LLMs in answering everyday health queries. Over the course of a week, a total of 34 participants prompted four publicly accessible LLMs with 212 real (or imagined) health concerns, and the LLM generated responses were evaluated by a team of nine board-certified physicians. At a high level, our findings indicate that on average, 76% of the 212 LLM responses were deemed to be accurate by physicians. Further, with the help of medical professionals, we investigated whether RAG versions of these LLMs (powered with a comprehensive medical knowledge base) can improve the quality of responses generated by LLMs. Finally, we also derive qualitative insights to explain our quantitative findings by conducting interviews with seven medical professionals who were shown all the prompts in our competition. This paper aims to provide a more grounded understanding of how LLMs perform in real-world everyday health communication.

Figures

Figures reproduced from arXiv: 2506.13805 by the authors.

Figure 1
Figure 1. Grouped bar plot showing rating distributions [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Box plots of Validity rating per strata. n= number [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Examples of responses from Gemini-1.5 Pro [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Our RAG pipeline implementation and evaluation methods. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Distribution of total rating excluding Harm for the [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: LLM diagnoses Evaluation form [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Box plot of all four ratings per medical specialty (with n [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Box plot of validity ratings for all medical specialties. Median values are labeled inside or over the boxes. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Box plot of all four ratings per prompt length. Median values are labeled inside or over the boxes. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Box plot of all four ratings per competition track. Median values are labeled inside the boxes. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 55 canonical work pages

  1. [1]

    R.; Mahajan, S.; Chaurasia, A.; et al

    Aboueid, S.; Meyer, S.; Wallace, J. R.; Mahajan, S.; Chaurasia, A.; et al. 2021. Young adults’ perspectives on the use of symptom checkers for self-triage and self-diagnosis: Qualitative study. JMIR public health and surveillance, 7(1): e22637

    work page 2021

  2. [2]

    M.; and Moussa, N

    AlGhamdi, K. M.; and Moussa, N. A. 2012. Internet use by the public to search for health-related information. International journal of medical informatics, 81(6): 363--373

    work page 2012

  3. [3]

    T.; Lotfinia, M.; Bressem, K.; Siepmann, R.; Ferber, D.; Kuhl, C.; Kather, J

    Arasteh, S. T.; Lotfinia, M.; Bressem, K.; Siepmann, R.; Ferber, D.; Kuhl, C.; Kather, J. N.; Nebelung, S.; and Truhn, D. 2024. RadioRAG: Factual Large Language Models for Enhanced Diagnostics in Radiology Using Dynamic Retrieval Augmented Generation. arXiv preprint arXiv:2407.15621

    arXiv 2024

  4. [4]

    Aydin, S.; Karabacak, M.; Vlachos, V.; and Margetis, K. 2025. Navigating the potential and pitfalls of large language models in patient-centered medication guidance and self-decision support. Frontiers in Medicine, 12: 1527864

    work page 2025

  5. [5]

    Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; DasSarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  6. [6]

    Balasubramanian, N. S. P.; and Dakshit, S. 2024. Can Public LLMs be used for Self-Diagnosis of Medical Conditions? arXiv preprint arXiv:2405.11407

    work page Pith review arXiv 2024

  7. [7]

    H.; and Bazzari, F

    Bazzari, A. H.; and Bazzari, F. H. 2024. Assessing the ability of GPT-4o to visually recognize medications and provide patient education. Scientific Reports, 14(1): 26749

    work page 2024

  8. [8]

    L.; and Childress, J

    Beauchamp, T. L.; and Childress, J. F. 1994. Principles of biomedical ethics. Edicoes Loyola

    work page 1994

Show all 68 references
  1. [9]

    B \'e chard, P.; and Ayala, O. M. 2024. Reducing hallucination in structured outputs via Retrieval-Augmented Generation. arXiv preprint arXiv:2404.08189

    2024 arXiv

  2. [10]

    A.; Wornow, M.; Swaminathan, A.; Lehmann, L

    Bedi, S.; Liu, Y.; Orr-Ewing, L.; Dash, D.; Koyejo, S.; Callahan, A.; Fries, J. A.; Wornow, M.; Swaminathan, A.; Lehmann, L. S.; et al. 2024. Testing and evaluation of health care applications of large language models: a systematic review. JAMA

    2024

  3. [11]

    Braun, V.; and Clarke, V. 2006. Using thematic analysis in psychology. Qualitative research in psychology, 3(2): 77--101

    2006

  4. [12]

    Cascella, M.; Montomoli, J.; Bellini, V.; and Bignami, E. 2023. Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. Journal of medical systems, 47(1): 33

    2023

  5. [13]

    Castagnari, E.; Muyama, L.; and Coulet, A. 2024. Prompting Large Language Models for Supporting the Differential Diagnosis of Anemia. arXiv preprint arXiv:2409.15377

    2024 arXiv

  6. [14]

    P.; Ferragina, P.; Tozzi, A

    De Angelis, L.; Baglivo, F.; Arzilli, G.; Privitera, G. P.; Ferragina, P.; Tozzi, A. E.; and Rizzo, C. 2023. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Frontiers in public health, 11: 1166120

    2023

  7. [15]

    Dras, M. 2015. Evaluating human pairwise preference judgments. Computational Linguistics, 41(2): 337--345

    2015

  8. [16]

    Eneva, Y.; and Dogan, B. 2025. Evaluation of Medical Diagnosis Capabilities of Three Artificial Intelligence Models--ChatGPT-3.5, Google Gemini, Microsoft Copilot: Sustainable Development Goals (SDGs). Journal of Lifestyle and SDGs Review, 5(2): e03545--e03545

    2025

  9. [17]

    Farnood, A.; Johnston, B.; and Mair, F. S. 2020. A mixed methods systematic review of the effects of patient online self-diagnosing in the ‘smart-phone society’on the healthcare professional-patient relationship and medical authority. BMC Medical Informatics and Decision Makin...

    2020

  10. [18]

    Gao, Y.; Xiong, Y.; Gao, X.; Jia, K.; Pan, J.; Bi, Y.; Dai, Y.; Sun, J.; Wang, H.; and Wang, H. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2: 1

    2023 arXiv

  11. [19]

    A.; Salah, K.; Jayaraman, R.; ur Rehman, M

    Gebreab, S. A.; Salah, K.; Jayaraman, R.; ur Rehman, M. H.; and Ellaham, S. 2024. Llm-based framework for administrative task automation in healthcare. In 2024 12th International Symposium on Digital Forensics and Security (ISDFS), 1--7. IEEE

    2024

  12. [20]

    Grote, T.; and Berens, P. 2024. A paradigm shift?—On the ethics of medical large language models. Bioethics, 38(5): 383--390

    2024

  13. [21]

    M.; Jhaveri, M.; Patel, N.; and Suthar, P

    Gupta, R.; Hamid, A. M.; Jhaveri, M.; Patel, N.; and Suthar, P. P. 2024. Comparative evaluation of AI models such as ChatGPT 3.5, ChatGPT 4.0, and Google Gemini in neuroradiology diagnostics. Cureus, 16(8): e67766

    2024

  14. [22]

    Hirosawa, T.; Harada, Y.; Mizuta, K.; Sakamoto, T.; Tokumasu, K.; and Shimizu, T. 2024. Evaluating ChatGPT-4’s accuracy in identifying final diagnoses within differential diagnoses compared with those of physicians: experimental study for diagnostic cases. JMIR Formative Resea...

    2024

  15. [23]

    L.; Takita, H.; Matsushita, S.; Mitsuyama, Y.; Shimono, T.; Miki, Y.; et al

    Horiuchi, D.; Tatekawa, H.; Oura, T.; Oue, S.; Walston, S. L.; Takita, H.; Matsushita, S.; Mitsuyama, Y.; Shimono, T.; Miki, Y.; et al. 2024. Comparing the Diagnostic Performance of GPT-4-based ChatGPT, GPT-4V-based ChatGPT, and Radiologists in Challenging Neuroradiology Cases...

    2024

  16. [24]

    Jeong, C. 2023. A Study on the Implementation of Generative AI Services Using an Enterprise Data-Based LLM Application Architecture. Advances in Artificial Intelligence and Machine Learning, 03(04): 1588–1618

    2023

  17. [25]

    Jin, M.; Yu, Q.; Zhang, C.; Shu, D.; Zhu, S.; Du, M.; Zhang, Y.; and Meng, Y. 2024. Health-LLM: Personalized retrieval-augmented disease prediction model. arXiv preprint arXiv: 2402.00746

    2024 arXiv

  18. [26]

    Jin, Y.; and Zhang, Y. 2024. OrthoDoc: Multimodal Large Language Model for Assisting Diagnosis in Computed Tomography. arXiv preprint arXiv:2409.09052

    2024 arXiv

  19. [27]

    P.; and O’Sullivan, E

    Khan, M. P.; and O’Sullivan, E. D. 2024. A comparison of the diagnostic ability of large language models in challenging clinical cases. Frontiers in Artificial Intelligence, 7: 1379297

    2024

  20. [28]

    G.; Sullivan, K

    Kleinbaum, D. G.; Sullivan, K. M.; Barker, N. D.; Kleinbaum, D. G.; Sullivan, K. M.; and Barker, N. D. 2013. Stratified analysis. ActivEpi Companion Textbook: A supplement for use with the ActivEpi CD-ROM, 419--476

    2013

  21. [29]

    Lafferty, G.; and Wyatt, T. 1995. Where to stick your data points: the treatment of measurements within wide bins. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 355(2-3): 541--547

    1995

  22. [30]

    LangChain. 2024. LangChain Framework. https://python.langchain.com/docs/introduction/. Accessed: 2024-11-06

    2024

  23. [31]

    M.; Tuwani, R.; Kompa, B.; Varma, A.; Finlayson, S

    Levine, D. M.; Tuwani, R.; Kompa, B.; Varma, A.; Finlayson, S. G.; Mehrotra, A.; and Beam, A. 2023. The diagnostic and triage accuracy of the GPT-3 artificial intelligence model. MedRxiv

    2023

  24. [32]

    T.; Purkayastha, S.; Celi, L

    Li, H.; Moon, J. T.; Purkayastha, S.; Celi, L. A.; Trivedi, H.; and Gichoya, J. W. 2023. Ethics of large language models in medicine and medical research. The Lancet Digital Health, 5(6): e333--e335

    2023

  25. [33]

    B.; and Wright, A

    Liu, S.; McCoy, A. B.; and Wright, A. 2025. Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines. Journal of the American Medical Informatics Association, ocaf008

    2025

  26. [34]

    Liu, Y.; Zhou, H.; Guo, Z.; Shareghi, E.; Vuli \'c , I.; Korhonen, A.; and Collier, N. 2024. Aligning with human judgement: The role of pairwise preference in large language model evaluators. arXiv preprint arXiv:2403.16950

    2024 arXiv

  27. [35]

    Marks, M.; and Haupt, C. E. 2023. AI chatbots, health privacy, and challenges to HIPAA compliance. Jama, 330(4): 309--310

    2023

  28. [36]

    McDuff, D.; Schaekermann, M.; Tu, T.; Palepu, A.; Wang, A.; Garrison, J.; Singhal, K.; Sharma, Y.; Azizi, S.; Kulkarni, K.; et al. 2023. Towards accurate differential diagnosis with large language models. arXiv preprint arXiv:2312.00164

    2023 arXiv

  29. [37]

    T.; Michaeli, J

    Michaeli, D. T.; Michaeli, J. C.; Albers, S.; and Michaeli, T. 2024. The healthcare workforce shortage of nurses and physicians: Practice, theory, evidence, and ways forward. Policy, Politics, & Nursing Practice, 25(4): 216--227

    2024

  30. [38]

    A.; Lester, J

    Omiye, J. A.; Lester, J. C.; Spichak, S.; Rotemberg, V.; and Daneshjou, R. 2023. Large language models propagate race-based medicine. NPJ Digital Medicine, 6(1): 195

    2023

  31. [39]

    A.; Darvell, M.; and Gray, J

    Powell, J. A.; Darvell, M.; and Gray, J. 2003. The doctor, the patient and the world-wide web: how the internet is changing healthcare. Journal of the royal society of medicine, 96(2): 74--76

    2003

  32. [40]

    Accessed: 2025-04-24

    Presiado, M.; Montero, A.; Lopes, L.; and Hamel, L. 2024. KFF Health Misinformation Tracking Poll: Artificial Intelligence and Health Information. https://www.kff.org/health-information-and-trust/poll-finding/kff-health-misinformation-tracking-poll-artificial-intelligence-and-...

    2024

  33. [41]

    Radwan, N. 2022. The internet’s role in undermining the credibility of the healthcare industry. International Journal of Computations, Information and Manufacturing (IJCIM), 2(1)

    2022

  34. [42]

    Raja, M.; Yuvaraajan, E.; et al. 2024. A RAG-based Medical Assistant Especially for Infectious Diseases. In 2024 International Conference on Inventive Computation Technologies (ICICT), 1128--1133. IEEE

    2024

  35. [43]

    L.; Li, A.; Pearson, A

    R \' os-Hoyo, A.; Shan, N. L.; Li, A.; Pearson, A. T.; Pusztai, L.; and Howard, F. M. 2024. Evaluation of large language models as a diagnostic aid for complex medical cases. Frontiers in Medicine, 11: 1380148

    2024

  36. [44]

    I.; Martin, R.; and Kaczmarczyk, R

    Roos, J.; Wilhelm, T. I.; Martin, R.; and Kaczmarczyk, R. 2024. From Language Models to Medical Diagnoses: Assessing the Potential of GPT-4 and GPT-3.5-Turbo in Digital Health. AI, 5(4): 2680--2692

    2024

  37. [45]

    A.; Hunger, J.; Boss, A.; and Richter, J

    Saenger, J. A.; Hunger, J.; Boss, A.; and Richter, J. 2024. Delayed diagnosis of a transient ischemic attack caused by ChatGPT. Wiener klinische Wochenschrift, 136(7): 236--238

    2024

  38. [46]

    Sarvari, P.; and Al-fagih, Z. 2024. Towards evaluating the diagnostic ability of LLMs

    2024

  39. [47]

    Shah-Mohammadi, F.; and Finkelstein, J. 2024. Accuracy evaluation of GPT-assisted differential diagnosis in emergency department. Diagnostics, 14(16): 1779

    2024

  40. [48]

    Shuster, K.; Poff, S.; Chen, M.; Kiela, D.; and Weston, J. 2021. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567

    2021 arXiv

  41. [49]

    diagnosis please

    Sonoda, Y.; Kurokawa, R.; Nakamura, Y.; Kanzawa, J.; Kurokawa, M.; Ohizumi, Y.; Gonoi, W.; and Abe, O. 2024. Diagnostic performances of GPT-4o, Claude 3 Opus, and Gemini 1.5 pro in “diagnosis please” cases. Japanese journal of radiology, 42(11): 1231--1235

    2024

  42. [50]

    Staab, R.; Vero, M.; Balunovi \'c , M.; and Vechev, M. 2023. Beyond memorization: Violating privacy via inference with large language models. arXiv preprint arXiv:2310.07298

    2023 arXiv

  43. [51]

    ecological validity

    Suchy, Y.; DesRuisseaux, L. A.; Mora, M. G.; Brothers, S. L.; and Niermeyer, M. A. 2024. Conceptualization of the term “ecological validity” in neuropsychological research on executive function assessment: a systematic review and call to action. Journal of the International Ne...

    2024

  44. [52]

    S.; Shim, W

    Suh, P. S.; Shim, W. H.; Suh, C. H.; Heo, H.; Park, C. R.; Eom, H. J.; Park, K. J.; Choe, J.; Kim, P. H.; Park, H. J.; et al. 2024. Comparing diagnostic accuracy of radiologists versus GPT-4V and Gemini Pro Vision using image inputs from diagnosis please cases. Radiology, 312(...

    2024

  45. [53]

    Tam, T. Y. C.; Sivarajkumar, S.; Kapoor, S.; Stolyar, A. V.; Polanska, K.; McCarthy, K. R.; Osterhoudt, H.; Wu, X.; Visweswaran, S.; Fu, S.; et al. 2024. A framework for human evaluation of large language models in healthcare derived from literature review. NPJ Digital Medicin...

    2024

  46. [54]

    Tural, B.; \"O rpek, Z.; and Destan, Z. 2024. Retrieval-Augmented Generation (RAG) and LLM Integration. In 2024 8th International Symposium on Innovative Approaches in Smart Technologies (ISAS), 1--5. IEEE

    2024

  47. [55]

    Van Bulck, L.; and Moons, P. 2024. What if your patient switches from Dr. Google to Dr. ChatGPT? A vignette-based survey of the trustworthiness, value, and danger of ChatGPT-generated responses to health questions. European Journal of Cardiovascular Nursing, 23(1): 95--98

    2024

  48. [56]

    E.; Haberland, M.; Reddy, T.; Cournapeau, D.; Burovski, E.; Peterson, P.; Weckesser, W.; Bright, J.; et al

    Virtanen, P.; Gommers, R.; Oliphant, T. E.; Haberland, M.; Reddy, T.; Cournapeau, D.; Burovski, E.; Peterson, P.; Weckesser, W.; Bright, J.; et al. 2020. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature methods, 17(3): 261--272

    2020

  49. [57]

    Wallis, S. 2013. Binomial confidence intervals and contingency tests: mathematical fundamentals and the evaluation of alternative methods. Journal of quantitative linguistics, 20(3): 178--208

    2013

  50. [58]

    Wang, C.; Liu, S.; Yang, H.; Guo, J.; Wu, Y.; and Liu, J. 2023. Ethical considerations of using ChatGPT in health care. Journal of Medical Internet Research, 25: e48009

    2023

  51. [59]

    Wang, X.; and Cohen, R. A. 2023. Health information technology use among adults: United States, July-December 2022

    2023

  52. [60]

    Wei, Q.; Yao, Z.; Cui, Y.; Wei, B.; Jin, Z.; and Xu, X. 2024. Evaluation of ChatGPT-generated medical responses: a systematic review and meta-analysis. Journal of Biomedical Informatics, 104620

    2024

  53. [61]

    G.; and Wang, D

    Yang, Z.; Xu, X.; Yao, B.; Rogers, E.; Zhang, S.; Intille, S.; Shara, N.; Gao, G. G.; and Wang, D. 2024. Talk2care: An llm-based voice assistant for communication between healthcare providers and older adults. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquit...

    2024

  54. [62]

    C.; Enichen, E.; Rivera, C.; Auger, C

    Young, C. C.; Enichen, E.; Rivera, C.; Auger, C. A.; Grant, N.; Rao, A.; and Succi, M. D. 2024. Diagnostic accuracy of a custom large language model on rare pediatric disease case reports. American Journal of Medical Genetics Part A, e63878

    2024

  55. [63]

    A.; Celi, L

    Zack, T.; Lehman, E.; Suzgun, M.; Rodriguez, J. A.; Celi, L. A.; Gichoya, J.; Jurafsky, D.; Szolovits, P.; Bates, D. W.; Abdulnour, R.-E. E.; et al. 2024. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. The Lanc...

    2024

  56. [64]

    Zhang, S.; and Song, J. 2024. A chatbot based question and answer system for the auxiliary diagnosis of chronic diseases based on large language model. Scientific reports, 14(1): 17118

    2024

  57. [65]

    Zhang, X.; Lin, D.; Pforsich, H.; and Lin, V. W. 2020. Physician workforce in the United States of America: forecasting nationwide shortages. Human resources for health, 18: 1--9

    2020

  58. [66]

    Zuccon, G.; Koopman, B.; and Palotti, J. 2015. Diagnose this if you can: On the effectiveness of search engines in finding medical self-diagnosis information. In Advances in Information Retrieval: 37th European Conference on IR Research, ECIR 2015, Vienna, Austria, March 29-Ap...

    2015

  59. [67]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  60. [68]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.