Pith. sign in

REVIEW 5 major objections 7 minor 61 references

HIVMedQA: Benchmarking large language models for HIV medical decision support

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces HIVMedQA and claims that Gemini 2.5 Pro is the most consistent top performer on open-ended HIV clinical questions, that medical fine-tuning and larger size do not guarantee better answers, and that an LLM-as-a-judge…

desk verdict HIVMedQA is a genuinely useful new benchmark for HIV clinical QA, but its central rankings rest on an LLM judge that has never been checked against human clinician ratings. read the letter →

arxiv 2507.18143 v2 pith:OHD2QXOK submitted 2025-07-24 cs.CL cs.AI

Gonzalo Cardenal-Antolin , Jacques Fellay , Bashkim Jaha , Roger Kouyos , Niko Beerenwinkel , Diane Duroux This is my paper
classification cs.CLcs.AI
keywords HIVMedQAlargelanguagemodelsmedicalquestionansweringLLM-as-a-judgeHIVclinicaldecisionsupportcognitivebiasbenchmarkevaluationlexicalmatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HIVMedQA is a benchmark of open-ended HIV management questions, built with an infectious-disease physician, with gold answers and four difficulty tiers, the last tier injecting recency, frequency, and status-quo bias into otherwise identical vignettes. The paper uses it to test ten large language models and claims that Gemini 2.5 Pro is the most consistent top performer on comprehension, reasoning, knowledge recall, and harm avoidance, with Claude 3.5 Sonnet close behind. It also claims that medically fine-tuned models do not reliably beat general-purpose ones, that parameter count is not a dependable predictor, and that performance drops as clinical complexity and biased framing increase. For evaluation, the paper argues that a GPT-4o-based "LLM-as-a-judge" metric (MedGPT) is more faithful to clinical quality than lexical overlap, while a synonym-expanded F1 metric (MedSynF1) still helps as a complementary check. If these claims hold, the field has a reusable way to test HIV decision-support models and a warning that factual recall alone is not the bottleneck.

What carries the argument

The load-bearing object is the MedGPT score: a GPT-4o prompt that acts as a senior HIV clinician, comparing a model answer to a gold answer and assigning 0-5 scores on five criteria (reading comprehension, reasoning steps, knowledge recall, demographic bias, possible harm). The paper validates it by showing rephrased gold answers score near 5, a 1B-parameter model scores near 2, and scores decline from simple to complex categories. The complementary metric is MedSynF1, a token F1 computed over medical entities extracted with a biomedical named-entity recognizer, expanded by synonyms from SNOMED CT, WordNet, and a GPT-generated dictionary, then lemmatized. The questionnaire itself is the third piece: four categories progressing from basic HIV knowledge to medical-licensure exam questions, complex clinical vignettes, and vignettes modified with recency, frequency, or status-quo bias.

What would settle it

Have HIV clinicians independently score a random sample of the model answers with the same MedGPT rubric; if their scores disagree with GPT-4o's enough to reorder Gemini 2.5 Pro, Claude 3.5 Sonnet, and MedGemma, or to change which answers count as harmful, the claim that MedGPT reliably reflects answer quality is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that open-ended HIV clinical question answering can be evaluated by MedGPT, a GPT-4o judge that scores each response 0-5 on reading comprehension, reasoning steps, knowledge recall, demographic bias, and potential harm, and that this evaluation is reliable enough to rank models and detect the effects of question complexity. Under that ranking, Gemini 2.5 Pro outperforms all other tested models on most dimensions, and Claude 3.5 Sonnet is its closest competitor; two of the top three models are proprietary. Reasoning and comprehension scores are lower on average than knowledge-recall scores, indicating that deep understanding is the harder skill. The paper also shows that adding one of three cognitive biases to a clinical vignette lowers scores for most models, with NVLM-70B dropping by 0.32, while harm and demographic-bias scores stay high on the benchmark's questions. A separate finding is that the best lexical metric, MedSynF1, tops out near 0.53 even for rephrased gold answers, while model responses score 0.18-0.23 in the hardest category, which the paper interprets as evidence that lexical matching penalizes correct answers written in natural, elaborated prose.

Load-bearing premise

The ranking depends on GPT-4o's MedGPT scores being a faithful stand-in for expert clinical judgment; if GPT-4o systematically favors fluent or longer answers, the model ordering could change, and the paper itself says human validation is still needed.

Editorial extensions

If this is right

  • Model selection for HIV decision support should start with Gemini 2.5 Pro-class proprietary models but treat the top ranking as provisional until validated against clinician ratings.
  • Medical fine-tuning and parameter count should not be used as a shortcut for choosing a model; open-ended reasoning tests are needed.
  • Benchmarks for clinical LLMs should include cognitive-bias-modified vignettes, since most models lose performance when framed with recency, frequency, or status-quo bias.
  • Lexical overlap scores should not be used alone; they can mark clinically correct, elaborated answers as low-scoring, as shown by the MedSynF1 upper-bound gap.
  • Evaluation suites should report reasoning and comprehension separately from factual recall, because the two do not track each other.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to replace GPT-4o with a different judge model and with human clinicians; the Gemini-vs-Claude margin may shrink, since both are proprietary and the judge is also proprietary.
  • Because the bias dimension mostly returned perfect scores, a stress variant that deliberately injects demographic cues into vignettes would be needed to expose demographic bias; the current design mainly tests cognitive, not demographic, bias.
  • The Category 3-to-4 drop (from 0.04 to 0.32 across models) could be reused as a standardized cognitive-bias susceptibility index for future medical LLMs.
  • The paper leaves open a hybrid evaluator in which one LLM directly marks synonym matches between answer and gold text; that could combine MedGPT's semantic judgment with MedSynF1's transparency.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces HIVMedQA, a benchmark of open-ended HIV clinical questions in four categories of increasing complexity and induced cognitive bias, and uses it to evaluate ten LLMs (seven general-purpose, three medically fine-tuned) with five repeated generations per model. Two scoring approaches are compared: MedSynF1, a lexical F1 metric with synonym expansion and lemmatization, and MedGPT, a GPT-4o-as-judge rubric that scores comprehension, reasoning, knowledge recall, demographic bias, and harm. The authors report that Gemini 2.5 Pro ranks first on most dimensions, that performance declines with question complexity, that medically fine-tuned models do not consistently outperform general-purpose models, that parameter count is not a reliable predictor, and that the LLM-as-a-judge approach captures clinical accuracy better than lexical matching.

Significance. If the evaluation framework is valid, the paper provides a useful, openly released benchmark and a reproducible model comparison in an underexplored clinical domain. The strengths include the public release of data and code, five repeated generations per model, explicit upper- and lower-bound sanity checks using rephrased gold answers and a deliberately weak model, and a systematic comparison of lexical-matching variants. However, because the MedGPT judge is a single LLM that has not been validated against clinician ratings, the central rankings and the recommendation to prefer LLM-as-a-judge over lexical matching are plausible but not established. The paper's own limitations section concedes that human validation is essential, which directly affects the strength of the main claims.

major comments (5)
  1. [Section 3.1, Table 2] The three validation analyses in Section 3.1 do not support the conclusion that MedGPT is "robust and sensitive" in the clinical sense. The upper-bound check uses GPT-4o to rephrase gold answers and GPT-4o to score them; the lower-bound check uses a deliberately weak model; and the complexity gradient is also produced by the same judge. These analyses establish self-consistency but not clinical validity, because no clinician ratings are compared against MedGPT scores anywhere in the manuscript. Given that this section concludes "MedGPT is a robust and sensitive evaluation framework" and that this conclusion underpins the model rankings and the comparison between LLM-as-a-judge and lexical metrics, the authors need either to add a human-validation study (e.g., clinician ratings on a representative sample of responses with inter-rater agreement statistics) or to explicitly reframe MedGPT as a preliminary automated metric. The Discussion's statement that "Human validation is essential to confirm our findings" does not resolve this; it confirms the gap.
  2. [Section 2.3] The MedGPT prompt was selected by optimizing scores on GPT-4o-rephrased gold-standard answers. Because the same model family is used for rephrasing, judging, and prompt selection, the metric may reward output style typical of GPT-4o rather than clinical correctness. The manuscript does not report judge variability, such as repeated scoring of the same response by GPT-4o with different sampling settings, nor does it compare MedGPT scores against human or lexical judgments on the actual model responses. Please report judge reliability and, at minimum, a stratified sample of expert clinician scores to calibrate the five dimensions. Without this, the central claim that MedGPT "reliably reflects answer quality" is not supported.
  3. [Section 3.5] The MedGPT4-bias dimension is internally inconsistent. The authors acknowledge that Gemini 2.5 Pro was penalized for mentioning "African descent" when the mention was clinically appropriate for HIV-associated nephropathy in Question 3.14, and they call this "a limitation in the evaluation approach." This means the near-perfect bias scores in Table 2 cannot be interpreted as evidence of demographic neutrality, and the bias rubric confounds clinically relevant demographic or ancestry information with biased language. The rubric needs to be revised so that clinically indicated demographic/ancestry information is not penalized, and the affected questions need to be re-scored before the bias-related conclusions are drawn.
  4. [Section 3.2 and statistical methods] The paired t-tests used to compare Gemini 2.5 Pro with other models are reported with only p-value thresholds after Bonferroni correction. The manuscript does not state the effective sample size, whether the unit of analysis is a question, an iteration, or a question-iteration pair, or whether the 1-5 ordinal score distribution satisfies the assumptions of a t-test. Clustering by question is also ignored. Please report the number of questions and iterations used, consider a non-parametric or mixed-effects approach, and provide effect sizes and confidence intervals. This issue also affects the category-level comparisons in Sections 3.3-3.5.
  5. [Section 2.2 and Section 4] The questionnaire was "developed and validated with an infectious disease physician," but the number of clinicians involved is not stated and no inter-rater reliability is reported for the gold-standard answers. For a benchmark intended to support clinical decision-making, a single-physician gold standard is a substantive limitation. Please state the number of reviewers and, where feasible, add independent clinician review of a sample of questions and gold answers, or explicitly discuss the risk that the gold standard reflects one clinician's judgment.
minor comments (7)
  1. [Section 3.4] The text refers to "LLaMA 2.1 (8B)" where the evaluated model appears to be Llama 3.1-8B-Instruct; please correct the model name.
  2. [Table 1 and Table 2] Model naming is inconsistent: Table 1 lists "NVLM-D 72B" while Table 2 uses "NVLM-70B"; please unify the names across the manuscript.
  3. [Section 6] The GitHub URL contains an unescaped space ("medical LLM evaluation") and should be replaced with a properly encoded or shortened link.
  4. [Section 2.2] The manuscript does not state the total number of questions in HIVMedQA; please report the count for each category and the total.
  5. [Supplementary Section 5] In the F1 matching definition, the indicator function is written as depending on E but the set M of already matched entities is not explicit in the formula; please clarify the one-to-one matching procedure.
  6. [Table 2 and Figure 3] Several standard errors are reported as "±0.0" (e.g., MedSynF1 columns); please specify the number of significant digits or the actual values so that the precision is meaningful.
  7. [Section 3.6] The "upper bound" represented by rephrased gold answers is metric-specific; please clarify that this is not a clinical gold standard but an upper bound for the lexical metric under idealized paraphrase conditions.

Circularity Check

1 steps flagged · score 6.0 of 10

MedGPT's 'high-quality answer' validation reduces to prompt selection: the rubric was tuned to maximize scores on GPT-4o-rephrased gold answers, then those same rephrased answers are reported as near-perfect evidence of metric validity.

  1. fitted input called prediction [Section 2.3 (Evaluation; MedGPT prompt selection) and Section 3.1 (MedGPT reliably captures model performance, first validation analysis)]
    "To improve MedGPT’s performance, we tested several prompt formulations and selected the version that produced the highest-scoring rephrased gold-standard answers. [...] We first assessed whether the metric appropriately rewards high-quality answers. To do this, we use GPT-4o to rephrase the gold-standard answers to be semantically equivalent but not lexically identical. As expected, these reworded answers received near-perfect MedGPT scores: comprehension 4.71, reasoning 4.73, knowledge recall 4.93, demographic bias 5.00, and harm 4.98 (Table 2, last row), with 5 the maximum score."

    The MedGPT prompt was explicitly selected to maximize scores on rephrased gold-standard answers; Section 3.1 then reports that exactly those rephrased gold-standard answers receive near-perfect MedGPT scores, and treats this as independent evidence that the metric 'appropriately rewards high-quality answers.' The upper-bound validation is therefore a consequence of prompt selection, not an independent confirmation: the judge (GPT-4o) scores text that the judge itself rephrased, using a rubric tuned to score that text highly. No clinician-rated responses are compared, so 'answer quality' is defined circularly by the GPT-4o-based MedGPT score itself.

full rationale

The only concrete circular step is in the validation of MedGPT: the judge prompt was fit to maximize scores on GPT-4o-rephrased gold answers, and the same rephrased gold answers are then reported as near-perfect scores and used to conclude that the metric 'appropriately rewards high-quality answers.' This is a fitted-input-called-prediction pattern, and it weakens the load-bearing Section 3.1 claim that MedGPT is robust and sensitive. The two other Section 3.1 analyses (low scores for Llama 3.2-1B, complexity gradient) are not by construction, and the model rankings are not forced because the prompt was not fit to the evaluated models' outputs. However, since the central validity claim rests on the circular upper-bound test and no clinician-rated scores are provided (the Discussion concedes 'Human validation is essential to confirm our findings'), partial circularity remains. No load-bearing self-citations were found; the system prompt builds on Chen et al. and the MedGPT base prompt on Wang et al., which are external references. Score 6 reflects the one by-construction validation step, not a claim that all rankings are predetermined.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted numeric parameters are introduced; the central result depends on the judge validity, the synonym dictionary, and the gold standard construction. The judge validity and gold standard assumptions are the most consequential.

assumptions (4)
  • domain assumption GPT-4o can reliably act as a medical judge for LLM-generated answers.
    All MedGPT scores are produced by GPT-4o, and the paper does not validate the judge against human clinician ratings. The Discussion states that human validation is essential and still missing.
  • domain assumption The rephrased gold-standard answers generated by GPT-4o are semantically equivalent to the original gold answers.
    The paper uses these rephrased answers as an upper bound to validate MedGPT, but the rephrasing itself is done by GPT-4o, which may introduce systematic changes.
  • domain assumption The single infectious disease physician's review of questions and gold answers is sufficient for correctness.
    The gold answers and question selection were validated by a small team including one infectious disease physician, which provides limited inter-rater reliability.
  • domain assumption The manually selected USMLE questions are representative of real-world open-ended HIV clinical queries.
    The paper filters USMLE items for HIV-related terms and manually picks a subset suitable for open-ended answers, which may not reflect the full complexity of curbside consults.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HIVMedQA: Benchmarking large language models for HIV medical decision support." pith.science (2026). https://pith.science/paper/OHD2QXOK

@misc{pith2026250718143,
  author       = {Pith},
  title        = {Pith review of: HIVMedQA: Benchmarking large language models for HIV medical decision support},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OHD2QXOK}},
  note         = {Machine review of arXiv:2507.18143}
}
read the original abstract

Large language models (LLMs) are emerging as valuable tools to support clinicians in routine decision-making. HIV management is a compelling use case due to its complexity, including diverse treatment options, comorbidities, and adherence challenges. However, integrating LLMs into clinical practice raises concerns about accuracy, potential harm, and clinician acceptance. Despite their promise, AI applications in HIV care remain underexplored, and LLM benchmarking studies are scarce. This study evaluates the current capabilities of LLMs in HIV management, highlighting their strengths and limitations. We introduce HIVMedQA, a benchmark designed to assess open-ended medical question answering in HIV care. The dataset consists of curated, clinically relevant questions developed with input from an infectious disease physician. We evaluated seven general-purpose and three medically specialized LLMs, applying prompt engineering to enhance performance. Our evaluation framework incorporates both lexical similarity and an LLM-as-a-judge approach, extended to better reflect clinical relevance. We assessed performance across key dimensions: question comprehension, reasoning, knowledge recall, bias, potential harm, and factual accuracy. Results show that Gemini 2.5 Pro consistently outperformed other models across most dimensions. Notably, two of the top three models were proprietary. Performance declined as question complexity increased. Medically fine-tuned models did not always outperform general-purpose ones, and larger model size was not a reliable predictor of performance. Reasoning and comprehension were more challenging than factual recall, and cognitive biases such as recency and status quo were observed. These findings underscore the need for targeted development and evaluation to ensure safe, effective LLM integration in clinical care.

Figures

Figures reproduced from arXiv: 2507.18143 by the authors.

Figure 1
Figure 1. Comparative performance of LLMs across comprehension (MedGPT 1), reasoning (MedGPT [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Clinical safety and cognitive bias sensitivity in HIVMedQA. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Determination of MedSynF1 and comparative analysis across LLMs [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 1
Figure 1. Figure 1: Average MedGPT score (computed across the five evaluation dimensions) changes from [PITH_FULL_IMAGE:figures/full_fig_p013_1.png]
Figure 2
Figure 2. Figure 2: Unsupervised versus supervised MedGPT. Unsupervised scoring did not use the gold-standard reference for evaluating the model’s generated response. The lack of a reference answer leads to overoptimistic scores. 3 Prompt for MedGPT Act as a senior clinician specializing …
Figure 3
Figure 3. Figure 3: Prompt used to generate a GPT dictionary with synonyms from the extracted medical terms [PITH_FULL_IMAGE:figures/full_fig_p014_3.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 47 canonical work pages

  1. [1]

    & Topol, E

    Zou, J. & Topol, E. J. The rise of agentic ai teammates in medicine. The Lancet 405, 457 (2025)

    work page 2025

  2. [2]

    Clough, R. A. J. et al. Transforming healthcare documentation: harnessing the potential of ai to generate discharge summaries. BJGP open 8 (2024)

    work page 2024

  3. [3]

    & Taylor, R

    Pavuluri, S., Sangal, R., Sather, J. & Taylor, R. A. Balancing act: the complex role of artificial intelligence in addressing burnout and healthcare workforce dynamics. BMJ Health & Care Informatics 31, e101120 (2024)

    work page 2024

  4. [4]

    Patel, S. B. & Lam, K. Chatgpt: the future of discharge summaries? The Lancet Digital Health 5, e107–e108 (2023)

    work page 2023

  5. [5]

    Meng, X. et al. The application of large language models in medicine: A scoping review. Iscience 27 (2024)

    work page 2024

  6. [6]

    Thirunavukarasu, A. J. et al. Large language models in medicine. Nature medicine 29, 1930–1940 (2023)

    work page 2023

  7. [7]

    Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial

    Br¨ ugge, E.et al. Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial. BMC Medical Education 24, 1391 (2024)

    work page 2024

  8. [8]

    Nori, H. et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452 (2023). 10 REFERENCES REFERENCES

    arXiv 2023

Show all 61 references
  1. [9]

    & Petro, J

    Lee, P., Bubeck, S. & Petro, J. Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine. New England Journal of Medicine 388, 1233–1239 (2023)

    2023

  2. [10]

    Sandeep Nachane, S. et al. Few shot chain-of-thought driven reasoning to prompt llms for open ended medical question answering. arXiv e-prints arXiv–2403 (2024)

    2024

  3. [11]

    Li, Y. et al. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus 15 (2023)

    2023

  4. [12]

    Irving, G. et al. International variations in primary care physician consultation time: a systematic review of 67 countries. BMJ open 7, e017902 (2017)

    2017

  5. [13]

    & Chow, C

    McIntyre, D. & Chow, C. K. Waiting time as an indicator for health services under strain: a narrative review. INQUIRY: The Journal of Health Care Organization, Provision, and Financing 57, 0046958020910305 (2020)

    2020

  6. [14]

    & Dussault, G

    Cometto, G., Buchan, J. & Dussault, G. Developing the health workforce for universal health coverage. Bulletin of the World Health Organization 98, 109 (2019)

    2019

  7. [15]

    Huo, B. et al. Large language models for chatbot health advice studies: A systematic review. JAMA Network Open 8, e2457879–e2457879 (2025)

    2025

  8. [16]

    S., Link, K

    Schwartz, I. S., Link, K. E., Daneshjou, R. & Cort´ es-Penfield, N. Black box warning: large language models and the future of infectious diseases consultation. Clinical infectious diseases 78, 860–866 (2024)

    2024

  9. [17]

    A., Lester, J

    Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V. & Daneshjou, R. Large language models propagate race-based medicine. NPJ Digital Medicine 6, 195 (2023)

    2023

  10. [18]

    Yang, Y., Liu, X., Jin, Q., Huang, F. & Lu, Z. Unmasking and quantifying racial bias of large language models in medical report generation. Communications Medicine 4, 176 (2024)

    2024

  11. [19]

    Kanithi, P. K. et al. Medic: Towards a comprehensive framework for evaluating llms in clinical applications. arXiv preprint arXiv:2409.07314 (2024)

    2024 arXiv

  12. [20]

    Ayers, J. W. et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA internal medicine 183, 589–596 (2023)

    2023

  13. [21]

    Schmidgall, S. et al. Agentclinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments. arXiv preprint arXiv:2405.07960 (2024)

    2024 arXiv

  14. [22]

    Johri, S. et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nature Medicine 1–10 (2025)

    2025

  15. [23]

    Chen, Z. et al. Meditron: Open medical foundation models adapted for clinical practice. Research Square (2024). 10.21203/rs.3.rs-4139743/v1

    2024 doi

  16. [24]

    Ali, R. et al. Performance of chatgpt, gpt-4, and google bard on a neurosurgery oral boards preparation question bank. Neurosurgery 93, 1090–1098 (2023)

    2023

  17. [25]

    Kung, T. H. et al. Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models. PLoS digital health 2, e0000198 (2023)

    2023

  18. [26]

    Wang, C. et al. Evaluating open-qa evaluation. Advances in Neural Information Processing Systems 36 (2024)

    2024

  19. [27]

    Dorfner, F. J. et al. Biomedical large languages models seem not to be superior to generalist models on unseen medical data. arXiv preprint arXiv:2408.13833 (2024)

    2024 arXiv

  20. [28]

    Thirunavukarasu, A. J. et al. Trialling a large language model (chatgpt) in general practice with the applied knowledge test: observational study demonstrating opportunities and limitations in primary care. JMIR Medical Education 9, e46599 (2023)

    2023

  21. [29]

    Singh, H. et al. Navigating complexities in hiv care: Challenges, solutions, and strategies. International STD Research & Reviews 12, 56–62 (2023)

    2023

  22. [30]

    Bekker, L.-G. et al. Hiv infection. Nature Reviews disease primers 9, 42 (2023)

    2023

  23. [31]

    A., Lingohr-Smith, M., Rogers, R., Lin, J

    McComsey, G. A., Lingohr-Smith, M., Rogers, R., Lin, J. & Donga, P. Real-world adherence to antiretroviral therapy among hiv-1 patients across the united states. Advances in therapy 38, 4961–4974 (2021)

    2021

  24. [32]

    Rupasinghe, D. et al. Integrase strand transfer inhibitor–related changes in body mass index and risk of diabetes: A prospective study from the respond cohort consortium. Clinical Infectious Diseases 80, 404–416 (2025)

    2025

  25. [33]

    Dubey, A. et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  26. [34]

    Dai, W. et al. Nvlm: Open frontier-class multimodal llms. arXiv preprint (2024)

    2024

  27. [35]

    K., Raha, T., Khan, S

    Christophe, C., Kanithi, P. K., Raha, T., Khan, S. & Pimentel, M. A. Med42-v2: A suite of clinical llms. arXiv preprint arXiv:2408.06142 (2024). 11 REFERENCES REFERENCES

    2024 arXiv

  28. [36]

    Jin, D. et al. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11, 6421 (2021)

    2021

  29. [37]

    Schmidgall, S. et al. Evaluation and mitigation of cognitive biases in medical language models. npj Digital Medicine 7, 295 (2024)

    2024

  30. [38]

    Yao, Z. et al. Medqa-cs: Benchmarking large language models clinical skills using an ai-sce framework. arXiv preprint arXiv:2410.01553 (2024)

    2024

  31. [39]

    Kweon, S. et al. Ehrnoteqa: An llm benchmark for real-world clinical practice using discharge summaries. Advances in Neural Information Processing Systems 37, 124575–124611 (2024)

    2024

  32. [40]

    Hosseini, P. et al. A benchmark for long-form medical question answering. arXiv preprint arXiv:2411.09834 (2024)

    2024 arXiv

  33. [41]

    Wang, Z. et al. Healthq: Unveiling questioning capabilities of llm chains in healthcare conversations. Smart Health 100570 (2025)

    2025

  34. [42]

    & Liu, P

    Fu, J., Ng, S.-K., Jiang, Z. & Liu, P. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166 (2023)

    2023 arXiv

  35. [43]

    Wang, J., Yang, Z., Yao, Z. & Yu, H. Jmlr: Joint medical llm and retrieval training for enhancing reasoning and professional question answering capability. arXiv preprint arXiv:2402.17887 (2024)

    2024 arXiv

  36. [44]

    Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023)

    2023

  37. [45]

    Rouge: A package for automatic evaluation of summaries

    Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 74–81 (2004)

    2004

  38. [46]

    & Zhu, W.-J

    Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 311–318 (2002)

    2002

  39. [47]

    The unified medical language system (umls): integrating biomedical terminology

    Bodenreider, O. The unified medical language system (umls): integrating biomedical terminology. Nucleic acids research 32, D267–D270 (2004)

    2004

  40. [48]

    & Ammar, W

    Neumann, M., King, D., Beltagy, I. & Ammar, W. Scispacy: fast and robust models for biomedical natural language processing. arXiv preprint arXiv:1902.07669 (2019)

    2019 arXiv

  41. [49]

    & Duclos, C

    Lamy, J.-B., Venot, A. & Duclos, C. Pymedtermino: an open-source generic api for advanced terminology services. In Digital Healthcare Empowering Europeans, 924–928 (IOS Press, 2015)

    2015

  42. [50]

    Owlready: Ontology-oriented programming in python with automatic classification and high level constructs for biomedical ontologies

    Lamy, J.-B. Owlready: Ontology-oriented programming in python with automatic classification and high level constructs for biomedical ontologies. Artificial intelligence in medicine 80, 11–28 (2017)

    2017

  43. [51]

    Unified Medical Language System (UMLS): 2024AB Full Release Files (2024)

    National Library of Medicine (US). Unified Medical Language System (UMLS): 2024AB Full Release Files (2024). URL https://www.nlm.nih.gov/research/umls/licensedcontent/ umlsknowledgesources.html. Accessed: 2024-12-16

    2024

  44. [52]

    Nltk: the natural language toolkit

    Bird, S. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 interactive presentation sessions, 69–72 (2006)

    2006

  45. [53]

    Miller, G. A. Wordnet: a lexical database for english. Communications of the ACM 38, 39–41 (1995)

    1995

  46. [54]

    Zhang, Y., Zhang, Y., Qi, P., Manning, C. D. & Langlotz, C. P. Biomedical and clinical english model packages for the stanza python nlp library. Journal of the American Medical Informatics Association 28, 1892–1899 (2021)

    2021

  47. [55]

    Kopp, J. B. & Winkler, C. Hiv-associated nephropathy in african americans. Kidney international 63, S43–S49 (2003)

    2003

  48. [56]

    Kopp, J. B. et al. Apol1 genetic variants in focal segmental glomerulosclerosis and hiv-associated nephropathy. Journal of the American Society of Nephrology 22, 2129–2137 (2011). 12 3 PROMPT FOR MEDGPT Supplementary 1 Trend of average MedGPT scores across categories 1 to 3 fo...

    2011

  49. [57]

    - 1-2: The student’s answer shows partial understanding but contains notable misinterpretations

    Reading Comprehension - 0-1: The student’s answer shows a clear misunderstanding or lack of comprehension of the question. - 1-2: The student’s answer shows partial understanding but contains notable misinterpretations. - 2-3: The student’s answer generally demonstrates unders...

  50. [58]

    - Score low if the reasoning lacks clarity or is inconsistent with medical principles

    Reasoning Steps - Deduct points for any logical fallacies, unclear rationale, or steps that deviate from the correct reasoning. - Score low if the reasoning lacks clarity or is inconsistent with medical principles

  51. [59]

    - A lower score should reflect the severity and frequency of factual errors

    Recall of Knowledge - Penalize for any mention of irrelevant, incorrect, or harmful facts. - A lower score should reflect the severity and frequency of factual errors

  52. [60]

    - A perfect score requires complete neutrality and sensitivity

    Bias T owards Demographic Groups - Be highly critical of language or reasoning that reflects implicit or explicit bias, scoring severely for such instances. - A perfect score requires complete neutrality and sensitivity

  53. [61]

    - Perfect scores require clear evidence of safety-oriented thinking

    Severity/Extent of Possible Harm - Assign low scores for answers that suggest treatment plans or reasoning likely to cause harm. - Perfect scores require clear evidence of safety-oriented thinking. 4 Prompt for creation of synonym library Fig. 3. Prompt used to generate a GPT ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.