Pith. sign in

REVIEW 4 major objections 6 minor 51 references

Zero-shot Performance of Generative AI in Brazilian Portuguese Medical Exam

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read General-purpose AI models reach applicant-level scores on a Brazilian Portuguese medical residency exam, with image questions as the remaining bottleneck.

desk verdict A useful, well-designed Brazilian Portuguese medical exam benchmark with a load-bearing zero-shot claim that needs verification and numeric inconsistencies that need cleanup. read the letter →

arxiv 2507.19885 v1 pith:LTDIR2AJ submitted 2025-07-26 cs.CL

Cesar Augusto Madid Truyts , Amanda Gomes Rabelo , Gabriel Mesquita de Souza , Daniel Scaldaferri Lages , Adriano Jose Pereira , Uri Adrian Prync Flato , Eduardo Pontes dos Reis , Joaquim Edson Vieira
show 2 more authors
Paulo Sergio Panse Silveira Edson Amaro Junior
This is my paper · ORCID
classification cs.CL
keywords generativeAIlargelanguagemodelsmultimodalmedicalresidencyexamBrazilianPortuguesezero-shotevaluationimageinterpretationeducation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Using 117 real questions from the 2024 HCFMUSP medical residency entrance exam, this study asks whether general-purpose language models can answer high-stakes clinical questions written in Brazilian Portuguese without prior exposure to those items. Six text-only and four multimodal (image-capable) models were prompted once in a standardized zero-shot protocol, and their accuracy was positioned against the actual distribution of human applicant scores. The paper's central finding is that the strongest models, Claude-3.5-Sonnet and Claude-3-Opus, perform within the main range of human candidates, while smaller and open-weights models fall well below. Accuracy drops systematically when questions include radiological or non-radiological images, and a physician review of one model's explanations shows that wrong answers usually come with unsound, sometimes unsafe, reasoning. The result matters because it tests AI on a real, linguistically authentic non-English medical exam rather than a translated benchmark, and it locates the remaining weakness in multimodal understanding rather than in Portuguese itself.

What carries the argument

The central object is the HCFMUSP residency exam itself: a publicly released, standardized 117-question multiple-choice test written in authentic Brazilian Portuguese, split into 74 text-only and 43 image-based items, with a known distribution of real applicant scores. The evaluation machinery is a zero-shot prompting protocol in which each model receives the same standardized instruction, answers five shuffled trials per question set, and has its answers extracted by regular expressions; accuracy is compared with the human score density, and image questions are further stratified into radiological and non-radiological categories. For explanation quality, one model's answers are reviewed by three physicians under concordance and safety criteria, with agreement quantified by Gwet's AC1 coefficient. This design lets the authors separate language effects from image-understanding effects without building a new benchmark or translating an existing one.

What would settle it

A contamination check would settle it: if any tested model can reproduce verbatim or near-verbatim text of any of the 117 exam questions when probed, or if its answer pattern matches the official key on items that appear in public training corpora, the zero-shot interpretation fails. The cleanest test is to rerun the same protocol on a newly written, never-published Portuguese residency exam and see whether the accuracy gap to human candidates persists.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a zero-shot commercial model can already perform like a typical medical residency applicant in Brazilian Portuguese, and that the main performance gap is modality-specific. On 74 text-only questions, Claude-3-Sonnet reached 72.97% accuracy, Claude-3-Opus 70.54%, Claude-3.5-Sonnet 70.27%, and GPT-4 Turbo 66.22%, placing the best models at or above the 65–70% peak of the human applicant distribution. On all 117 questions including images, Claude-3.5-Sonnet sustained 69.57%, while Claude-3-Opus fell to 63.59%, Claude-3-Sonnet to 54.70%, and Claude-3-Haiku to 44.44%, with radiological images producing the lowest scores. The authors further report that a three-physician review of Claude-3-Opus explanations judged roughly 94% of correct answers as coherently explained, while most incorrect answers were accompanied by flawed rationales that reviewers often deemed unsafe. These results are used to argue that Portuguese itself is not the binding constraint for top models, and that improved multimodal reasoning and language-specific fine-tuning are the needed next steps.

Load-bearing premise

The comparison is only meaningful as a zero-shot test if none of the tested models encountered these exam questions during training; that claim is asserted but not independently verifiable for proprietary models.

Editorial extensions

If this is right

  • Top-tier general-purpose models can serve as passable first-pass answerers on Portuguese clinical text, at roughly the level of the median residency applicant.
  • Image-based clinical questions, especially radiology, are the current reliability ceiling; any deployment in specialties where imaging is central should treat model output as draft rather than decision.
  • Because wrong answers almost always came with flawed explanations, an answer-only accuracy score overstates clinical usefulness; explanation safety needs separate evaluation.
  • The strongest models' comparable performance in Portuguese, Spanish, and English suggests language alone is not the dominant barrier, so investing in multimodal training data may pay off more than further English-to-Portuguese translation.
  • Annual public release of such exams creates a repeatable, contamination-aware evaluation loop: each new exam can serve as a fresh held-out test for the next generation of models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that a controlled Portuguese-versus-English translation study of the same 117 questions would sharply test the language-disparity reading; the current data support it only indirectly.
  • Because the exam and candidate scores are released publicly by law, the same protocol could be rerun yearly as a rolling benchmark, with each new exam serving as genuinely unseen data.
  • The pattern that experts disagreed most about the model's wrong answers suggests some of those items are genuinely ambiguous, so part of the accuracy gap may reflect question quality rather than model deficiency.
  • A testable prediction of the paper's image-bottleneck reading is that fine-tuning on Portuguese radiology and clinical image-text pairs should substantially raise image-question accuracy; if it does not, the limitation is architectural rather than data-driven.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript reports a zero-shot evaluation of ten generative AI models (six text-only LLMs and four multimodal Claude models) on the Brazilian Portuguese HCFMUSP medical residency entrance exam. The authors use 117 valid multiple-choice questions, 74 of which are text-only, run five shuffled trials per model, and compare model accuracy with the distribution of human candidate scores. They also conduct a secondary experiment in which three physicians evaluate the explanations produced by Claude-3-Opus for concordance with the chosen answer and potential patient harm, using Gwet's AC1 agreement coefficient. The central claim is that some proprietary models, especially Claude-3.5-Sonnet and Claude-3-Opus, reach accuracy comparable to typical human applicants on text questions, while performance drops on image-based questions, and that this reveals language and modality gaps for non-English medical AI.

Significance. If the core claims hold, this is a useful contribution to multilingual medical LLM evaluation. The study uses a real, high-stakes, publicly disclosed exam rather than a translated or synthetic benchmark, includes both text-only and multimodal conditions under identical prompting, applies repeated-measures statistics over five trials, and provides code, data, and model explanations in a public repository. The expert-evaluation component of Claude-3-Opus explanations is a valuable addition, as is the explicit comparison with the human candidate score distribution. The external and non-circular nature of the benchmark is a strength. However, the central zero-shot interpretation depends on an unverifiable training-data assumption, and several quantitative inconsistencies across the text and tables must be resolved before the main comparisons can be accepted.

major comments (4)
  1. [§2 Methods and Title/Abstract] The zero-shot framing is load-bearing but rests on the assertion in §2 that 'None of the models were exposed to the questions of this test during their training phase.' For proprietary models like Claude-3.5-Sonnet, GPT-4 Turbo, and Command R+, this cannot be independently audited, and no model version identifiers, API access dates, or training-cutoff dates are provided. The HCFMUSP exam questions were publicly disclosed, and the timeline relative to each model's training window is not established. The authors should either provide concrete cutoff-date and release-date evidence, reframe the study as a closed-book or post-publication assessment, or add a clear limitation paragraph explaining that the zero-shot characterization is conditional on an unverifiable assumption.
  2. [§3.1, §3.2, §3.3, Tables 1 and 2] Accuracy values are inconsistent between the narrative and the tables. For example, §3.1 reports Claude-3-sonnet at 72.97% and GPT-4 Turbo at 66.22%, while §3.3 reports 75% and 68.3% for the same models; §3.2 reports Claude-3-haiku at 44.44% and Claude-3-opus at 63.59%, while §3.3 reports 61.1% and 63.7% and refers to a nonexistent 'Claude-3-instant.' Table 1 also lists LLaMA-3-70b at 58.11%, which §3.1 does not mention in its summary of moderate performers. These discrepancies directly affect the headline claims and must be reconciled with a single verified set of numbers across text, tables, and figures.
  3. [§3.3 and Figure 5] The comparison between Experiment 1 model accuracy and the human candidate distribution is not apples-to-apples. Experiment 1 uses only the 74 text-only questions, but the human candidate scores are for the full 117-question exam, which includes image-based questions. If humans perform systematically better or worse on image questions, the statement that top models are 'comparable to human candidates' in the text-only condition could be misleading. The authors should either restrict the human comparison to Experiment 2, compute a human text-only score if the underlying data allow it, or explicitly discuss the direction and magnitude of this mismatch as a limitation.
  4. [§3.3 and Statistical Analysis] The claim that several models 'achieved accuracy levels comparable to human candidates' is based solely on visual overlap between model point estimates and the smoothed human score density in Figure 5. No statistical test quantifies this comparison, and the human density is estimated from aggregated candidate scores without uncertainty or per-question type information. An equivalence test or a formal interval-based comparison would provide a much stronger basis for the central claim; otherwise, the language should be softened to 'within the observed human score range.'
minor comments (6)
  1. [Abstract, §1] The phrase 'Brazilian spoken portuguese' should be 'Brazilian Portuguese'; the manuscript also contains repeated typos such as 'LLhama' instead of 'LLaMA.'
  2. [Tables 1 and 2] The 'Similarity' column entries are concatenated with model names (for example, 'aLLhama-3-8b' and 'gClaude-3.5-sonnet'), making the tables difficult to read. Use a separate compact-letter column or line breaks.
  3. [Figures] In the provided manuscript, Figures 1-5 are represented only by captions; the actual images are missing. Ensure the figures are included in the submission, as the panels are referenced extensively in the results.
  4. [References and Methods] Reference [26] is cited as the 'Haus Lin package' but the package is called 'hausekeep'; also, the Wikipedia citations for language statistics could be replaced with more authoritative sources, although this is not central to the findings.
  5. [§3.2] The sentence 'mean processing time per question increases were observed with the addiction of questions containing images' should read 'addition of questions containing images.'
  6. [§4 Discussion] The phrase 'this in congruent with the fact' should be 'this is incongruent with the fact'; also, clarify whether the exam was administered in 2023 or 2024, since §2 says the exam 'applied in 2024' but earlier refers to the 'HCFMUSP 2023' exam.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an external benchmark against a public exam; the only self-citation is a statistical method reference and is not load-bearing.

full rationale

This paper is an external evaluation of ten LLMs/MLLMs against a real, public medical residency exam. The central quantities are accuracy (Eq. 1), processing time, and expert agreement (Gwet AC1), all measured directly against the published answer key and the distribution of human candidate scores. No parameter is fitted to the outcome and then renamed as a prediction; no derived quantity is defined in terms of another measured quantity; and no claim is justified by an equation that reduces to its own input. The only author self-citation is reference [30] (Silveira & Siqueira) for the Gwet AC1 coefficient used in Experiment 3; that citation supplies an externally published statistical method and does not support any of the paper's empirical conclusions about model accuracy or language disparities. The zero-shot framing depends on the assertion that the models were trained before the exam questions were released, which cannot be independently audited for proprietary models. However, an unverifiable training-data precondition is a data-validity and contamination-risk concern, not a circular derivation: it does not make any computed quantity equal to an input by construction. The paper makes no self-referential uniqueness argument, does not smuggle an ansatz through citation, and does not rename a known result. Therefore the appropriate circularity finding is no significant circularity, score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities. The evaluation depends on unverifiable assumptions about model training data and API versioning, plus standard statistical assumptions.

assumptions (3)
  • domain assumption The exam questions were not present in the training data of the tested models.
    Stated in Methods without external verification; the entire zero-shot framing depends on this.
  • domain assumption The named API model versions correspond to the models being evaluated and their behavior is stable across the five trials.
    Models were accessed via Bedrock and an internal Azure GPT endpoint; version pinning is not documented in the manuscript.
  • standard math Standard statistical assumptions for repeated-measures ANOVA and linear mixed models are satisfied.
    Used for the omnibus tests and post-hoc comparisons; no diagnostics are shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zero-shot Performance of Generative AI in Brazilian Portuguese Medical Exam." pith.science (2026). https://pith.science/paper/LTDIR2AJ

@misc{pith2026250719885,
  author       = {Pith},
  title        = {Pith review of: Zero-shot Performance of Generative AI in Brazilian Portuguese Medical Exam},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LTDIR2AJ}},
  note         = {Machine review of arXiv:2507.19885}
}
read the original abstract

Artificial intelligence (AI) has shown the potential to revolutionize healthcare by improving diagnostic accuracy, optimizing workflows, and personalizing treatment plans. Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have achieved notable advancements in natural language processing and medical applications. However, the evaluation of these models has focused predominantly on the English language, leading to potential biases in their performance across different languages. This study investigates the capability of six LLMs (GPT-4.0 Turbo, LLaMA-3-8B, LLaMA-3-70B, Mixtral 8x7B Instruct, Titan Text G1-Express, and Command R+) and four MLLMs (Claude-3.5-Sonnet, Claude-3-Opus, Claude-3-Sonnet, and Claude-3-Haiku) to answer questions written in Brazilian spoken portuguese from the medical residency entrance exam of the Hospital das Cl\'inicas da Faculdade de Medicina da Universidade de S\~ao Paulo (HCFMUSP) - the largest health complex in South America. The performance of the models was benchmarked against human candidates, analyzing accuracy, processing time, and coherence of the generated explanations. The results show that while some models, particularly Claude-3.5-Sonnet and Claude-3-Opus, achieved accuracy levels comparable to human candidates, performance gaps persist, particularly in multimodal questions requiring image interpretation. Furthermore, the study highlights language disparities, emphasizing the need for further fine-tuning and data set augmentation for non-English medical AI applications. Our findings reinforce the importance of evaluating generative AI in various linguistic and clinical settings to ensure a fair and reliable deployment in healthcare. Future research should explore improved training methodologies, improved multimodal reasoning, and real-world clinical integration of AI-driven medical assistance.

Figures

Figures reproduced from arXiv: 2507.19885 by the authors.

Figure 1
Figure 1. Example of correct answer from the model and justification validated as [PITH_FULL_IMAGE:figures/full_fig_p023_1.png] view at source ↗
Figure 2
Figure 2. Example of incorrect answer from the model. However, the clinician [PITH_FULL_IMAGE:figures/full_fig_p023_2.png] view at source ↗
Figure 3
Figure 3. Comparison of Accuracy (median accuracy, percentage of correct answers) [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of model accuracy (five trials for each model) based on the type [PITH_FULL_IMAGE:figures/full_fig_p025_4.png]
Figure 5
Figure 5. Figure 5: Distribution of accuracy (solid line) and median (dashed line) of residency [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 35 canonical work pages

  1. [1]

    Artificial Intelligence and Machine Learning Technology Driven Modern Drug Discovery and Development

    Sarkar C, Das B, Rawat VS, Wahlang JB, Nongpiur A, Tiewsoh I, et al. Artificial Intelligence and Machine Learning Technology Driven Modern Drug Discovery and Development. Int J Mol Sci. 2023;24. Available from: https://doi.org/10.3390/ijms24032026

    work page doi:10.3390/ijms24032026 2023

  2. [2]

    Creation and Adoption of Large Language Models in Medicine

    Shah NH, Entwistle D, Pfeffer MA. Creation and Adoption of Large Language Models in Medicine. JAMA. 2023;330. Available from: https: //doi.org/10.1001/jama.2023.14217

    arXiv 2023

  3. [3]

    Artificial Intelligence and Machine Learning in Clinical Medicine, 2023

    Haug CJ, Drazen JM. Artificial Intelligence and Machine Learning in Clinical Medicine, 2023. N Engl J Med. 2023;388. Available from: https://doi.org/10.1056/nejmra2302038

    work page doi:10.1056/nejmra2302038 2023

  4. [4]

    Validation of Prediction Models for Critical Care Outcomes Using Natural Language Processing of Electronic Health Record Data

    Marafino BJ, Park M, Davies JM, Thombley R, Luft HS, Sing DC, et al. Validation of Prediction Models for Critical Care Outcomes Using Natural Language Processing of Electronic Health Record Data. JAMA Netw Open. 2018;1. Available from: https://doi.org/10.1001/jamanetworkopen. 2018.5097

    arXiv 2018

  5. [5]

    Procedure prediction from symbolic Electronic Health Records via time intervals analytics

    Moskovitch R, Polubriaginof F, Weiss A, Ryan P, Tatonetti N. Procedure prediction from symbolic Electronic Health Records via time intervals analytics. J Biomed Inform. 2017;75. Available from: https://doi.org/ 10.1016/j.jbi.2017.07.018

    work page doi:10.1016/j.jbi.2017.07.018 2017

  6. [6]

    Multitask learning and benchmarking with clinical time series data

    Harutyunyan H, Khachatrian H, Kale DC, Steeg GV, Galstyan A. Multitask learning and benchmarking with clinical time series data. Sci Data. 2019;6. Available from: https://doi.org/10.1038/s41597-019- 0103-9

    doi:10.1038/s41597-019- 2019

  7. [7]

    Holistic Evaluation of Language Models

    Bommasani R, Liang P, Lee T. Holistic Evaluation of Language Models. Ann N Y Acad Sci. 2023;1525. Available from: https://doi.org/10. 1111/nyas.15007

    work page 2023

  8. [8]

    Large language models encode clinical knowledge

    Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, et al. Large language models encode clinical knowledge. Nature. 2023;620. Available from: https://doi.org/10.1038/s41586-023-06291-2

    doi:10.1038/s41586-023-06291-2 2023

Show all 51 references
  1. [9]

    Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models; 2023

    Srivastava A, Rastogi A, Rao A, Shoeb AAM, Abid A, Fisch A, et al.. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models; 2023. Available from: https://arxiv. org/abs/2206.04615

    2023 arXiv

  2. [10]

    Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine; 2023

    Nori H, Lee YT, Zhang S, Carignan D, Edgar R, Fusi N, et al.. Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine; 2023. Available from: https://arxiv.org/abs/2311. 16452. REFERENCES 15

    2023

  3. [11]

    Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP World; 2022

    Ranathunga S, de Silva N. Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP World; 2022. Available from: https://arxiv.org/abs/2210.08523

    2022 arXiv

  4. [12]

    Portuguese-speaking world — Wikipedia, The Free Encyclopedia; 2024

    Wikipedia. Portuguese-speaking world — Wikipedia, The Free Encyclopedia; 2024. Available from: https://en.wikipedia.org/ wiki/Portuguese-speaking_world

    2024

  5. [13]

    List of languages by number of native speakers; 2025

    Wikipedia. List of languages by number of native speakers; 2025. Accessed: 2025-01-11. Available from: https://en.wikipedia.org/wiki/List_of_ languages_by_number_of_native_speakers

    2025

  6. [14]

    Usage Statistics of Content Languages for Websites; 2025

    W3Techs. Usage Statistics of Content Languages for Websites; 2025. Accessed: 2025-01-11. Available from: https://w3techs.com/ technologies/overview/content_language

    2025

  7. [15]

    Use of a Large Language Model to Identify and Classify Injuries With Free- Text Emergency Department Data

    Lorenzoni G, Gregori D, Bressan S, Ocagli H, Azzolina D, Dalt LD, et al. Use of a Large Language Model to Identify and Classify Injuries With Free- Text Emergency Department Data. JAMA Netw Open. 2024 5;7:e2413208-

    2024

  8. [16]

    Available from: https://doi.org/10.1001/jamanetworkopen.2024. 13208

    2024 doi

  9. [17]

    Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis

    Liu M, Okuhara T, Chang X, Shirabe R, Nishiie Y, Okada H, et al. Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis. J Med Internet Res. 2024;26:e60807. Available from: https://doi.org/10.2196/ 60807

    2024

  10. [18]

    Annotated dataset creation through large language models for non-english medical NLP

    Frei J, Kramer F. Annotated dataset creation through large language models for non-english medical NLP. J Biomed Inform. 2023;145. Available from: https://doi.org/10.1016/j.jbi.2023.104478

    2023

  11. [19]

    Introducing Bode: A Fine-Tuned Large Language Model for Portuguese Prompt-Based Task; 2024

    Garcia GL, Paiola PH, Morelli LH, Candido G, J´ unior AC, Jodas DS, et al.. Introducing Bode: A Fine-Tuned Large Language Model for Portuguese Prompt-Based Task; 2024. Available from: https://arxiv.org/abs/ 2401.02909

    2024 arXiv

  12. [20]

    Sabi´ a-2: A New Generation of Portuguese Large Language Models; 2024

    Almeida TS, Abonizio H, Nogueira R, Pires R. Sabi´ a-2: A New Generation of Portuguese Large Language Models; 2024. Available from: https:// arxiv.org/abs/2403.09887

    2024 arXiv

  13. [21]

    Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine

    Guillen-Grima F, Guillen-Aguinaga S, Guillen-Aguinaga L, Alas-Brun R, Onambele L, Ortega W, et al. Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine. Clin Pract. 2023;13. Av...

    2023 doi

  14. [22]

    Residˆ encia M´ edica 2024 – FUVEST divulga quest˜ oes de prova objetiva de concurso para residˆ encia m´ edica da FMUSP – Fuvest; 2024

    FUVEST. Residˆ encia M´ edica 2024 – FUVEST divulga quest˜ oes de prova objetiva de concurso para residˆ encia m´ edica da FMUSP – Fuvest; 2024. REFERENCES 16 Available from: https://www.fuvest.br/residencia-medica-2024- fuvest-divulga-questoes-de-prova-objetiva-de-concurso-pa...

    2024

  15. [23]

    Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models

    Kung TH, Cheatham M, Medenilla A, Sillos C, Leon LD, Elepa˜ no C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PloS Digit Health. 2023;2. Available from: https://doi.org/10.1371/journal.pdig.0000198

    2023 doi

  16. [24]

    Performance of ChatGPT, GPT-4, and Google Bard on a Neurosurgery Oral Boards Preparation Question Bank

    Ali R, Tang OY, Connolly ID, Fridley JS, Shin JH, Sullivan PLZ, et al. Performance of ChatGPT, GPT-4, and Google Bard on a Neurosurgery Oral Boards Preparation Question Bank. Neurosurgery. 2023;93(5). Available from: https://doi.org/10.1227/neu.0000000000002551

    2023 doi

  17. [25]

    ChatGPT-4 Performance on USMLE Step 1 Style Questions and Its Implications for Medical Education: A Comparative Study Across Systems and Disciplines

    Garabet R, Mackey BP, Cross J, Weingarten M. ChatGPT-4 Performance on USMLE Step 1 Style Questions and Its Implications for Medical Education: A Comparative Study Across Systems and Disciplines. Medical Science Educator. 2024;34. Available from: https://doi.org/10.1007/ s40670...

    2024

  18. [26]

    ChatGPT-4: An assessment of an upgraded artificial intelligence chatbot in the United States Medical Licensing Examination

    Mihalache A, Huang RS, Popovic MM, Muni RH. ChatGPT-4: An assessment of an upgraded artificial intelligence chatbot in the United States Medical Licensing Examination. Medical Teacher. 2024;46. Available from: https://doi.org/10.1080/0142159X.2023.2249588

    2024

  19. [27]

    hausekeep: A Collection of Utility Functions for Data Science and Statistics - Compute standard errors (within-subjects)

    Lin H. hausekeep: A Collection of Utility Functions for Data Science and Statistics - Compute standard errors (within-subjects);. Available from: https://hauselin.github.io/hausekeep/reference/seWithin.html

  20. [28]

    emmeans: Estimated marginal means, aka least-squares means

    Lenth RV. emmeans: Estimated marginal means, aka least-squares means. R package (version 1.7.1). R Foundation for Statistical Computing. 2021;34

    2021

  21. [29]

    R: A Language and Environment for Statistical Computing

    Team RC. R: A Language and Environment for Statistical Computing

  22. [30]

    Better to be in agreement than in bad company: A critical analysis of many kappa-like tests

    Silveira PSP, Siqueira JO. Better to be in agreement than in bad company: A critical analysis of many kappa-like tests. Behav Res Methods. 2022. Available from: https://doi.org/10.3758/s13428-022-01950-0

    2022 doi

  23. [31]

    Simultaneous Inference in General Parametric Models

    Hothorn T, Bretz F, Westfall P. Simultaneous Inference in General Parametric Models. Biometrical Journal. 2008 6;50:346-63. Available from: https://doi.org/10.1002/bimj.200810425

    2008 doi

  24. [32]

    Online Assessment of Applied Anatomy Knowledge: The Effect of Images on Medical Students’ Performance

    Sagoo MG, Vorstenbosch MATM, Bazira PJ, Ellis H, Kambouri M, Owen C. Online Assessment of Applied Anatomy Knowledge: The Effect of Images on Medical Students’ Performance. Anat Sci Educ. 2021;14. REFERENCES 17

    2021

  25. [33]

    The effects of images on multiple-choice questions in computer-based formative assessment

    Mart´ ın-Sanjos´ e JF, Juan MC, Viv´ o R, Abad F. The effects of images on multiple-choice questions in computer-based formative assessment. Digit Educ Rev. 2015

    2015

  26. [34]

    The Cognitive Basis for the Split- Attention Effect

    Pouw W, Rop G, de Koning B, Paas F. The Cognitive Basis for the Split- Attention Effect. J Exp Psychol Gen. 2019;148. Available from: https: //doi.org/10.1037/xge0000578

    2019 doi

  27. [35]

    Testing the seductive details effect: Does the format or the amount of seductive details matter? Appl Cogn Psychol

    Wang Z, Ardasheva Y, Carbonneau K, Liu Q. Testing the seductive details effect: Does the format or the amount of seductive details matter? Appl Cogn Psychol. 2021;35. Available from: https://doi.org/10.1002/acp. 3801

    2021 doi

  28. [36]

    Machine Learning in Dermatology: Current Applications, Opportunities, and Limitations

    Chan S, Reddy V, Myers B, Thibodeaux Q, Brownstone N, Liao W. Machine Learning in Dermatology: Current Applications, Opportunities, and Limitations. Dermatol Ther. 2020;10. Available from: https: //doi.org/10.1007/s13555-020-00372-0

    2020 doi

  29. [37]

    Can a picture ruin a thousand words? The effects of visual resources in exam questions

    Crisp V, Sweiry E. Can a picture ruin a thousand words? The effects of visual resources in exam questions. Educational Research. 2006;48. Available from: https://doi.org/10.1080/00131880600732249

    2006 doi

  30. [38]

    Artificial Intelligence in Dermatology—Where We Are and the Way to the Future: A Review

    Hogarty DT, Su JC, Phan K, Attia M, Hossny M, Nahavandi S, et al. Artificial Intelligence in Dermatology—Where We Are and the Way to the Future: A Review. Am J Clin Dermatol. 2020;21. Available from: https://doi.org/10.1007/s40257-019-00462-6

    2020 doi

  31. [39]

    Review of Machine Learning in Predicting Dermatological Outcomes

    Du AX, Emam S, Gniadecki R. Review of Machine Learning in Predicting Dermatological Outcomes. Front Med. 2020;7. Available from: https: //doi.org/10.3389/fmed.2020.00266

    2020

  32. [40]

    Radiology artificial intelligence: a systematic review and evaluation of methods (RAISE)

    Kelly BS, Judge C, Bollard SM, Clifford SM, Healy GM, Aziz A, et al. Radiology artificial intelligence: a systematic review and evaluation of methods (RAISE). Eur Radiol. 2022;32. Available from: https://doi. org/10.1007/s00330-022-08784-6

    2022 doi

  33. [41]

    Dermacen Analytica: A Novel Methodology Integrating Multi-Modal Large Language Models with Machine Learning in tele-dermatology; 2024

    Panagoulias DP, Tsoureli-Nikita E, Virvou M, Tsihrintzis GA. Dermacen Analytica: A Novel Methodology Integrating Multi-Modal Large Language Models with Machine Learning in tele-dermatology; 2024. Available from: https://arxiv.org/abs/2403.14243

    2024 arXiv

  34. [42]

    Hallucination of Multimodal Large Language Models: A Survey; 2024

    Bai Z, Wang P, Xiao T, He T, Han Z, Zhang Z, et al.. Hallucination of Multimodal Large Language Models: A Survey; 2024. Available from: https://arxiv.org/abs/2404.18930

    2024 arXiv

  35. [43]

    AI in radiology: From promise to practice - A guide to effective integration

    Katal S, York B, Gholamrezanezhad A. AI in radiology: From promise to practice - A guide to effective integration. Eur J Radiol. 2024 12;181. Available from: https://doi.org/10.1016/j.ejrad.2024.111798

    2024

  36. [44]

    MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context

    Gu Z, Yin C, Liu F, Zhang P. MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context

  37. [45]

    Mitigating Hallucinations in Large Language Models: A Comparative Study of RAG-enhanced vs

    Li A, Shrestha R, Jegatheeswaran T, Chan HO, Hong C, Joshi R. Mitigating Hallucinations in Large Language Models: A Comparative Study of RAG-enhanced vs. Human-Generated Medical Templates. medRxiv. 2024. Available from: https://doi.org/10.1101/2024.09.27. 24314506. REFERENCES 18

    2024 doi

  38. [46]

    Explainability does not mitigate the negative impact of incorrect AI advice in a personnel selection task

    Cecil J, Lermer E, Hudecek MFC, Sauer J, Gaube S. Explainability does not mitigate the negative impact of incorrect AI advice in a personnel selection task. Sci Rep. 2024;14:9736. Available from: https://doi.org/ 10.1038/s41598-024-60220-5

    2024 doi

  39. [47]

    Available from: https://arxiv.org/abs/2407.02730

    arXiv

  40. [48]

    ChatGPT 3.5 fails to write appropriate multiple choice practice exam questions

    Ngo A, Gupta S, Perrine O, Reddy R, Ershadi S, Remick D. ChatGPT 3.5 fails to write appropriate multiple choice practice exam questions. Acad Pathol. 2024;11. Available from: https://doi.org/10.1016/j.acpath. 2023.100099

    2024

  41. [50]

    ChatGPT: Transcending Language Limitations in Scientific Research Using Artificial Intelligence

    Osama M, Afridi S, Maaz M. ChatGPT: Transcending Language Limitations in Scientific Research Using Artificial Intelligence. J Coll Physicians Surg Pak. 2023;33. Available from: https://doi.org/10. 29271/jcpsp.2023.10.1198

    2023

  42. [51]

    Similarity

    Bedi S, Cui H, Fuentes M, Unell A, Wornow M, Banda JM, et al.. MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks; 2025. Available from: https://arxiv.org/abs/2505.23802. 5 TABLES 19 5 Tables Table 1: Summary statistics for Experiment 1. Model Accuracy St....

    2025 arXiv

  43. [2024]

    Available from: https://www.R-project.org/

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.